Development Environment Setup

BuildGrid comes with Docker support for local development.

Caution

The Docker manifests are intended to be use for local development only. Do not use them in production.

Please consult the Get Started with Docker guide if you are looking for instructions on how to setup Docker on your machine.

Docker build

BuildGrid ships a Dockerfile manifest for building images from source using docker build. In order to produce a buildgrid:local base image, run:

git clone https://gitlab.com/BuildGrid/buildgrid.git
cd buildgrid
docker build --tag buildgrid:local .

Note

The image built will contain the Python sources, including example configuration files. The main endpoint is the bgd CLI tools and the default command shall run the BuildGrid server loading default configuration.

Once completed, you can check that build succeed by locally starting in a container the BuildGrid server with default configuration. Simply run:

docker run --interactive --publish 50051:50051 buildgrid:local

Hint

You can run any of the BuildGrid CLI tool using that image, simply pass extra arguments to docker run the same way you would pass them to bgd.

Bear in mind that whenever the source code or the configuration files are updated, you must re-build the image.

Docker Compose

BuildGrid ships a docker-compose.yml manifest for building and running a grid locally using docker-compose. This is the recommended way of running a simple demo grid.

In order to produce a buildgrid:local base image, run:

git clone https://gitlab.com/BuildGrid/buildgrid.git
cd buildgrid
docker-compose build

Once completed, you can start a simple grid by running:

docker-compose up

Note

The grid is composed of five containers: - A PostgreSQL database available at localhost:5432. - An execution service available at http://localhost:50051. - A CAS service available at http://localhost:50052. - An ActionCache service available at http://localhost:50053. - A single unnamed instance with one host-tools based worker bot attached.

Hint

You can spin up more bots by using docker-compose scaling capabilities:

docker-compose up --scale bots=12

Hint

The contained services configuration files are bind mounted into the container, no need to rebuild the base image on configuration update. Configuration files are read from:

  • data/config/controller.yml for the Execution service.

  • data/config/storage.yml for the CAS service.

  • data/config/cache.yml for the ActionCache service

Minimal Docker Compose

BuildGrid also provides a docker-compose-examples/all-in-one.yml manifest, which deploys a minimal BuildGrid that is easy to get working with Bazel. To use it, first build the buildgrid:local base image as above:

git clone https://gitlab.com/BuildGrid/buildgrid.git
cd buildgrid
docker-compose -f docker-compose-examples/all-in-one.yml build

Once completed, you can start a simple grid by running:

docker-compose -f docker-compose-examples/all-in-one.yml up

Note

The grid is composed of three containers:

  • A PostgreSQL database available at localhost:5432.

  • Execution, CAS, and ActionCache services available at http://localhost:50051.

  • A single unnamed instance with one host-tools based worker bot attached.

Hint

You can spin up more bots by using docker-compose scaling capabilities:

docker-compose -f docker-compose-examples/all-in-one.yml up --scale bots=12

Hint

The configuration file is bind mounted into the container, no need to rebuild the base image on configuration update. Configuration files are read from:

  • data/config/all-in-one.yml for the BuildGrid services.

Interactive Development with tox

Support for development using tox is available out of the box with BuildGrid.

Tox handles installing virtualenvs with all your dependencies, based on the use case. So for example to create a virtualenv with the dependencies sufficient to run a simple BuildGrid server, we provide the “server” environment:

tox -e server

This will install the virtualenv and then run bgd server start data/config/default.yml -vvv within that env.

For the bot, you can run a buildbox-worker.

buildbox-casd --cas-remote=http://localhost:50051 --bind=127.0.0.1:50011 ~/casd &
buildbox-worker --buildbox-run=buildbox-run-hosttools --bots-remote=http://localhost:50051 \
   --cas-remote=http://127.0.0.1:50011 --request-timeout=30 my_bot

For development, the venv virtualenv is useful as that does not run a preconfigured command. To run the tests from the virtualenv you simply specify the venv environment and then whatever command you wish to run after that:

tox -e venv -- pytest
# Or run a single test file, for example test_index.py:
tox -e venv -- pytest  tests/cas/index/test_index.py

Before running tests that use S3, start a moto_server instance for the tests to connect to.

.tox/venv/bin/moto_server

You can use tox to build these docs themselves too, there is an environment for that:

tox -e docs

A full list of available environments is available in tox.ini.