Try to simplify our bot setup. (#3986)

This moves all the bot-related files to `dev/bots`, hiding it from our
home page in github. Also, simplifies the travis setup, though that
doesn't do any difference to the performance sadly.

dev/bots/.gitignore

@@ -0,0 +1,3 @@
+.recipe_deps
+android_tools
+*.pyc

dev/bots/README.md

@@ -0,0 +1,89 @@
+# Flutter's Build Infrastructure
+
+This directory exists to support building Flutter on our build infrastructure.
+
+The results of such builds are viewable at:
+* https://build.chromium.org/p/client.flutter/waterfall
+* https://travis-ci.org/flutter/flutter/builds (limited checking used just for PRs on github)
+
+The external master pages for the chromium infra bots do not allow
+forcing new builds. Contact @eseidelGoogle or another member of
+Google's Flutter team if you need to do that.
+
+The Travis-based bots are trivial, and just run a couple of shell
+scripts. The rest of this document discusses only the chromium infra
+bots.
+
+This infrastructure is broken into two parts. A buildbot master specified by our
+[builders.pyl](https://chromium.googlesource.com/chromium/tools/build.git/+/master/masters/master.client.flutter/builders.pyl)
+file, and a [set of
+recipes](https://chromium.googlesource.com/chromium/tools/build.git/+/master/scripts/slave/recipes/flutter)
+which we run on that master.  Both of these technologies are highly specific to
+Google's Chromium project. We're just borrowing some of their infrastructure.
+
+## Prerequisites
+
+- [install depot_tools](http://www.chromium.org/developers/how-tos/install-depot-tools)
+- Python package installer: `sudo apt-get install python-pip`
+- Python coverage package (only needed for `training_simulation`): `sudo pip install coverage`
+
+## Getting the code
+
+The following will get way more than just recipe code, but it _will_ get the recipe code:
+
+```bash
+mkdir chrome_infra
+cd chrome_infra
+fetch infra
+```
+
+More detailed instructions can be found [here](https://chromium.googlesource.com/infra/infra/+/master/doc/source.md).
+
+Most of the functionality for recipes comes from `recipe_modules`, which are
+unfortunately spread to many separate repositories.  After checking out the code
+search for files named `api.py` or `example.py` under `infra/build`.
+
+## Editing a recipe
+
+Flutter has one recipe per repository. Currently
+[flutter/flutter](https://chromium.googlesource.com/chromium/tools/build.git/+/master/scripts/slave/recipes/flutter/flutter.py)
+and
+[flutter/engine](https://chromium.googlesource.com/chromium/tools/build.git/+/master/scripts/slave/recipes/flutter/engine.py):
+
+- build/scripts/slave/recipes/flutter/flutter.py
+- build/scripts/slave/recipes/flutter/engine.py
+
+Recipes are just Python.  They are
+[documented](https://github.com/luci/recipes-py/blob/master/doc/user_guide.md)
+by the [luci/recipes-py github project](https://github.com/luci/recipes-py).
+
+The typical cycle for editing a recipe is:
+
+1. Make your edits (probably to files in `//chrome_infra/build/scripts/slave/recipes/flutter`).
+2. Run `build/scripts/slave/recipes.py simulation_test train flutter` to update expected files  (remove the flutter if you need to do a global update).
+3. Run `build/scripts/tools/run_recipe.py flutter/flutter` (or flutter/engine) if something was strange during training and you need to run it locally.
+4. Upload the patch (`git commit`, `git cl upload`) and send it to someone in the `recipes/flutter/OWNERS` file for review.
+
+## Editing the client.flutter buildbot master
+
+Flutter uses Chromium's fancy
+[builders.pyl](https://chromium.googlesource.com/infra/infra/+/master/doc/users/services/buildbot/builders.pyl.md)
+master generation system.  Chromium hosts 100s (if not 1000s) of buildbot
+masters and thus has lots of infrastructure for turning them up and down.
+Eventually all of buildbot is planned to be replaced by other infrastructure,
+but for now flutter has its own client.flutter master.
+
+You would need to edit client.flutter's master in order to add slaves (talk to
+@eseidelGoogle), add builder groups, or to change the html layout of
+https://build.chromium.org/p/client.flutter.  Carefully follow the [builders.pyl
+docs](https://chromium.googlesource.com/infra/infra/+/master/doc/users/services/buildbot/builders.pyl.md)
+to do so.
+
+## Future Directions
+
+We would like to host our own recipes instead of storing them in
+[build](https://chromium.googlesource.com/chromium/tools/build.git/+/master/scripts/slave/recipes/flutter).
+Support for [cross-repository
+recipes](https://github.com/luci/recipes-py/blob/master/doc/cross_repo.md) is
+in-progress.  If you view the git log of this directory, you'll see we initially
+tried, but it's not quite ready.

dev/bots/VERSION_LINUX_NDK

@@ -0,0 +1 @@
+7b727f0d4c853c9848847839a317300cb83f4ece

dev/bots/VERSION_LINUX_SDK

@@ -0,0 +1 @@
+0d320c50b0ed188c7e1182388e2beb623a1d307d

dev/bots/VERSION_MACOSX_NDK

@@ -0,0 +1 @@
+e21479f1ae8d2fac385c3965672c912d08dff280

dev/bots/VERSION_MACOSX_SDK

@@ -0,0 +1 @@
+fa5ea0ca1e0c7c2e40914f3202c7545de4dbca9c

dev/bots/docs.sh

@@ -0,0 +1,20 @@
+#!/bin/bash
+set -ex
+
+# Install dartdoc.
+pub global activate dartdoc
+
+# Generate flutter docs into dev/docs/doc/api/.
+(cd dev/tools; pub get)
+
+# This script generates a unified doc set, and creates
+# a custom index.html, placing everything into dev/docs/doc
+dart dev/tools/dartdoc.dart
+
+# Ensure google webmaster tools can verify our site.
+cp dev/docs/google2ed1af765c529f57.html dev/docs/doc
+
+# Upload the docs.
+if [ "$1" = "--upload" ]; then
+  gsutil -m rsync -d -r dev/docs/doc/ gs://docs.flutter.io/
+fi

dev/bots/download_android_tools.py

@@ -0,0 +1,97 @@
+#!/usr/bin/python
+# Copyright 2015 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""
+Downloads trimmed-down Android Tools from Google Cloud Storage and extracts
+them to INSTALL_DIR, updating INSTALL_DIR/VERSION_* stamp files with current
+version. Does nothing if INSTALL_DIR/VERSION_* are already up to date.
+"""
+
+import os
+import shutil
+import subprocess
+import sys
+import tarfile
+
+# Path constants. (All of these should be absolute paths.)
+THIS_DIR = os.path.abspath(os.path.dirname(__file__))
+INSTALL_DIR = os.path.join(THIS_DIR, 'android_tools')
+
+import find_depot_tools
+
+DEPOT_PATH = find_depot_tools.add_depot_tools_to_path()
+GSUTIL_PATH = os.path.join(DEPOT_PATH, 'gsutil.py')
+
+def RunCommand(command):
+  """Run command and return success (True) or failure."""
+
+  print 'Running %s' % (str(command))
+  if subprocess.call(command, shell=False) == 0:
+    return True
+  print 'Failed.'
+  return False
+
+def GetInstalledVersion(version_stamp):
+  version_file = os.path.join(INSTALL_DIR, version_stamp)
+  if not os.path.exists(version_file):
+    return None
+  with open(version_file) as f:
+    return f.read().strip()
+
+def VersionStampName(tools_name):
+  if sys.platform.startswith('linux'):
+    return 'VERSION_LINUX_' + tools_name.upper()
+  elif sys.platform == 'darwin':
+    return 'VERSION_MACOSX_' + tools_name.upper()
+  else:
+    raise Exception('Unsupported platform: ' + sys.platform)
+
+def UpdateTools(tools_name):
+  """Downloads zipped tools from Google Cloud Storage and extracts them,
+     stamping current version."""
+
+  # Read latest version.
+  version_stamp = VersionStampName(tools_name)
+  version = ''
+  with open(os.path.join(THIS_DIR, version_stamp)) as f:
+    version = f.read().strip()
+  # Return if installed binaries are up to date.
+  if version == GetInstalledVersion(version_stamp):
+    return
+
+  # Remove the old install directory checked out from git.
+  if os.path.exists(os.path.join(INSTALL_DIR, '.git')):
+    shutil.rmtree(INSTALL_DIR)
+  # Make sure that the install directory exists.
+  if not os.path.exists(INSTALL_DIR):
+    os.mkdir(INSTALL_DIR)
+  # Remove current installation.
+  tools_root = os.path.join(INSTALL_DIR, tools_name)
+  if os.path.exists(tools_root):
+    shutil.rmtree(tools_root)
+
+  # Download tools from GCS.
+  archive_path = os.path.join(INSTALL_DIR, tools_name + '.tar.gz')
+  download_cmd = ['python', GSUTIL_PATH, 'cp',
+                  'gs://mojo/android/tool/%s.tar.gz' % version,
+                  archive_path]
+  if not RunCommand(download_cmd):
+    print ('WARNING: Failed to download Android tools.')
+    return
+
+  print "Extracting Android tools (" + tools_name + ")"
+  with tarfile.open(archive_path) as arch:
+    arch.extractall(INSTALL_DIR)
+  os.remove(archive_path)
+  # Write version as the last step.
+  with open(os.path.join(INSTALL_DIR, version_stamp), 'w+') as f:
+    f.write('%s\n' % version)
+
+def main():
+  UpdateTools('sdk')
+  UpdateTools('ndk')
+
+if __name__ == '__main__':
+  sys.exit(main())

dev/bots/find_depot_tools.py

@@ -0,0 +1,45 @@
+# Copyright (c) 2011 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+"""Small utility function to find depot_tools and add it to the python path.
+
+Will throw an ImportError exception if depot_tools can't be found since it
+imports breakpad.
+"""
+
+import os
+import sys
+
+
+def IsRealDepotTools(path):
+  return os.path.isfile(os.path.join(path, 'gclient.py'))
+
+
+def add_depot_tools_to_path():
+  """Search for depot_tools and add it to sys.path."""
+  # First look if depot_tools is already in PYTHONPATH.
+  for i in sys.path:
+    if i.rstrip(os.sep).endswith('depot_tools') and IsRealDepotTools(i):
+      return i
+  # Then look if depot_tools is in PATH, common case.
+  for i in os.environ['PATH'].split(os.pathsep):
+    if IsRealDepotTools(i):
+      sys.path.append(i.rstrip(os.sep))
+      return i
+  # Rare case, it's not even in PATH, look upward up to root.
+  root_dir = os.path.dirname(os.path.abspath(__file__))
+  previous_dir = os.path.abspath(__file__)
+  while root_dir and root_dir != previous_dir:
+    i = os.path.join(root_dir, 'depot_tools')
+    if IsRealDepotTools(i):
+      sys.path.append(i)
+      return i
+    previous_dir = root_dir
+    root_dir = os.path.dirname(root_dir)
+  print >> sys.stderr, 'Failed to find depot_tools'
+  return None
+
+add_depot_tools_to_path()
+
+# pylint: disable=W0611
+import breakpad

dev/bots/setup.sh

@@ -0,0 +1,8 @@
+#!/bin/bash
+set -ex
+
+# disable analytics on the bots and download Flutter dependencies
+./bin/flutter config --no-analytics
+
+# run pub get in all the repo packages
+./bin/flutter update-packages

dev/bots/test.sh

@@ -0,0 +1,29 @@
+#!/bin/bash
+set -ex
+
+export PATH="$PWD/bin:$PWD/bin/cache/dart-sdk/bin:$PATH"
+
+# analyze all the Dart code in the repo
+flutter analyze --flutter-repo
+
+# verify that the tests actually return failure on failure and success on success
+(cd dev/automated_tests; ! flutter test test_smoke_test/fail_test.dart > /dev/null)
+(cd dev/automated_tests; flutter test test_smoke_test/pass_test.dart > /dev/null)
+
+# run tests
+(cd packages/flutter; flutter test)
+(cd packages/flutter_driver; dart -c test/all.dart)
+(cd packages/flutter_sprites; flutter test)
+(cd packages/flutter_test; flutter test)
+(cd packages/flutter_tools; dart -c test/all.dart)
+(cd packages/flx; dart -c test/all.dart)
+
+(cd dev/manual_tests; flutter test)
+(cd examples/hello_world; flutter test)
+(cd examples/layers; flutter test)
+(cd examples/flutter_gallery; flutter test)
+(cd examples/stocks; flutter test)
+
+# generate and analyze our large sample app
+dart dev/tools/mega_gallery.dart
+(cd dev/benchmarks/mega_gallery; flutter analyze --watch --benchmark)