.. _install-docker: Development Environment Setup ============================= BuildGrid comes with Docker support for local development. .. caution:: The Docker manifests are intended to be use for **local development only**. Do **not** use them in production. Please consult the `Get Started with Docker`_ guide if you are looking for instructions on how to setup Docker on your machine. .. _`Get Started with Docker`: https://www.docker.com/get-started .. _install-docker-build: Docker build ------------ BuildGrid ships a ``Dockerfile`` manifest for building images from source using ``docker build``. In order to produce a ``buildgrid:local`` base image, run: .. code-block:: sh git clone https://gitlab.com/BuildGrid/buildgrid.git cd buildgrid docker build --tag buildgrid:local . .. note:: The image built will contain the Python sources, including example configuration files. The main endpoint is the ``bgd`` CLI tools and the default command shall run the BuildGrid server loading default configuration. Once completed, you can check that build succeed by locally starting in a container the BuildGrid server with default configuration. Simply run: .. code-block:: sh docker run --interactive --publish 50051:50051 buildgrid:local .. hint:: You can run any of the BuildGrid CLI tool using that image, simply pass extra arguments to ``docker run`` the same way you would pass them to ``bgd``. Bear in mind that whenever the source code or the configuration files are updated, you **must** re-build the image. .. _install-docker-compose: Docker Compose -------------- BuildGrid ships a ``docker-compose.yml`` manifest for building and running a grid locally using ``docker-compose``. This is the recommended way of running a simple demo grid. In order to produce a ``buildgrid:local`` base image, run: .. code-block:: sh git clone https://gitlab.com/BuildGrid/buildgrid.git cd buildgrid docker-compose build Once completed, you can start a simple grid by running: .. code-block:: sh docker-compose up .. note:: The grid is composed of five containers: - A PostgreSQL database available at ``localhost:5432``. - An execution service available at ``http://localhost:50051``. - A CAS service available at ``http://localhost:50052``. - An ActionCache service available at ``http://localhost:50053``. - A single unnamed instance with one host-tools based worker bot attached. .. hint:: You can spin up more bots by using ``docker-compose`` scaling capabilities: .. code-block:: sh docker-compose up --scale bots=12 .. hint:: The contained services configuration files are bind mounted into the container, no need to rebuild the base image on configuration update. Configuration files are read from: - ``data/config/controller.yml`` for the Execution service. - ``data/config/storage.yml`` for the CAS service. - ``data/config/cache.yml`` for the ActionCache service Minimal Docker Compose ~~~~~~~~~~~~~~~~~~~~~~ BuildGrid also provides a ``docker-compose.all-in-one.yml`` manifest, which deploys a minimal BuildGrid that is easy to get working with Bazel. To use it, first build the ``buildgrid:local`` base image as above: .. code-block:: sh git clone https://gitlab.com/BuildGrid/buildgrid.git cd buildgrid docker-compose -f docker-compose.all-in-one.yml build Once completed, you can start a simple grid by running: .. code-block:: sh docker-compose -f docker-compose.all-in-one.yml up .. note:: The grid is composed of three containers: - A PostgreSQL database available at ``localhost:5432``. - Execution, CAS, and ActionCache services available at ``http://localhost:50051``. - A single unnamed instance with one host-tools based worker bot attached. .. hint:: You can spin up more bots by using ``docker-compose`` scaling capabilities: .. code-block:: sh docker-compose -f docker-compose.all-in-one.yml up --scale bots=12 .. hint:: The configuration file is bind mounted into the container, no need to rebuild the base image on configuration update. Configuration files are read from: - ``data/config/all-in-one.yml`` for the BuildGrid services. Interactive Development with tox -------------------------------- Support for development using tox_ is available out of the box with BuildGrid. Tox handles installing virtualenvs with all your dependencies, based on the use case. So for example to create a virtualenv with the dependencies sufficient to run a simple BuildGrid server, we provide the "server" environment: .. code-block:: sh tox -e server This will install the virtualenv and then run ``bgd server start data/config/default.yml -vvv`` within that env. For the bot, you can run a ``buildbox-worker``. .. code-block:: sh buildbox-casd --cas-remote=http://localhost:50051 --bind=127.0.0.1:50011 ~/casd & buildbox-worker --buildbox-run=buildbox-run-hosttools --bots-remote=http://localhost:50051 \ --cas-remote=http://127.0.0.1:50011 --request-timeout=30 my_bot For development, the `venv` virtualenv is useful as that does not run a preconfigured command. To run the tests from the virtualenv you simply specify the ``venv`` environment and then whatever command you wish to run after that: .. code-block:: sh tox -e venv -- pytest # Or run a single test file, for example test_index.py: tox -e venv -- pytest tests/cas/index/test_index.py Before running tests that use S3, start a ``moto_server`` instance for the tests to connect to. .. code-block:: sh .tox/venv/bin/moto_server You can use tox to build these docs themselves too, there is an environment for that: .. code-block:: sh tox -e docs A full list of available environments is available in tox.ini_. .. _tox: https://tox.readthedocs.io/en/latest/ .. _tox.ini: https://gitlab.com/BuildGrid/buildgrid/-/blob/master/tox.ini