What is buildbox?

buildbox is a set of tools for remote worker build execution, conforming to the remote workers api, and the remote execution api. It is highly configurable and extensible while also having support for Linux, AIX, and Solaris. To learn more about buildbox, its architecture and how to build the various tools, please visit the buildbox landing page.

Using buildbox-worker

This section will explain how to connect a simple buildbox-worker to BuildGrid. buildbox-worker is a worker written in C++ that implements the Remote Workers API.

buildbox-worker runs its underlying job using a pluggable “runner.” For this example, we will be using the buildbox-run-hosttools runner, which simply runs the command you send to it directly with no sandboxing.

First, build buildbox-worker and buildbox-run-hosttools. Instructions for each can be found in the respective READMEs. As part of the build process, you will also need to build and install buildbox-common, which has several dependencies that need to be installed.

Then, start a BuildGrid instance. If you have followed the installation guide and also have the BuildGrid source on your machine, you can use the default configuration:

bgd server start data/config/default.conf

Running the worker is pretty simple –you just need to pass it the locations of the BuildGrid server, BuildGrid CAS server, and buildbox-run-hosttools. For example, if your server has an execution service and a CAS service both located at localhost:50051, you can invoke the worker like this:

/path/to/buildbox-worker --bots-remote=http://localhost:50051 --cas-remote=http://localhost:50051 --buildbox-run=/path/to/buildbox-run-hosttools my_bot

If all goes well, you should have a functioning worker named my_bot that’s ready to accept jobs.

Using a local cache (buildbox-casd)

Currently, whenever a worker needs to fetch blobs, it must make a request to the CAS. Since the CAS is most likely on a separate machine, this can be an expensive operation. The worker’s performance can be greatly improved if there is a local cache which minimized requests to the CAS.

buildbox-casd acts as a local cache; serving as a proxy to CAS for buildbox-worker. If it doesn’t have a blob which the worker requests, it will fetch it from CAS, cache it, and then provide it to the worker. Consequently, future requests for the blob (by any workers on the machine) will hit buildbox-casd's local cache instead of the remote CAS.

You will first need to build buildbox-casd. Instructions for how to do this can be found on it’s README. Note that buildbox-casd also depends on buildbox-common.

You can then launch buildbox casd with the following command:

/path/to/buildbox-casd --cas-remote=http://localhost:50051 --bind=127.0.0.1:50011 /location/to/store/local/cache

You can then launch the worker, using the following command:

/path/to/buildbox-worker --bots-remote=http://localhost:50051 --cas-remote=http://127.0.0.1:50011 --buildbox-run=/path/to/buildbox-run-hosttools --runner-arg=--prefix-staged-dir my_bot

You can additionally tell the worker where to run the action with the workspace path argument:

/path/to/buildbox-worker --bots-remote=http://localhost:50051 --cas-remote=http://127.0.0.1:50011 --buildbox-run=/path/to/buildbox-run-hosttools --runner-arg=--prefix-staged-dir --runner-arg=--workspace-path=/stage/location my_bot

You should now have local caching enabled on your worker setup.

Enabling the local cas protocol

You can further optimize the local caching by enabling the local cas protocol. In order to understand what this does, we will need some definitions.

Stage means placing the files needed to execute your action. Capture refers to saving the action’s outputs.

Enabling the local cas protocol means that buildbox-casd will stage and capture the files instead of buildbox-worker. The benefit of this is that buildbox-casd can do this much more quickly and efficiently, resulting in a huge performance gain.

In order to do this safely you will need to run buildbox-casd and buildbox-worker as separate users. This prevents the actions from corrupting the local cache.

To launch buildbox-casd, use the following command:

sudo -u casduser /path/to/buildbox-casd --cas-remote=http://localhost:50051 --bind=127.0.0.1:50011 /location/to/store/local/cache

To launch buildbox-worker, with the local cas protocol enabled, run:

sudo -u workeruser /path/to/buildbox-worker --bots-remote=http://localhost:50051 --cas-remote=http://127.0.0.1:50011 --buildbox-run=/path/to/buildbox-run-hosttools --runner-arg=--use-localcas --runner-arg=--prefix-staged-dir my_bot

The local cas protocol should now be enabled. Enjoy your super fast builds!

Sandboxed workers (buildbox-run-userchroot)

So far, we have been using buildbox-run-hosttools as our runner. While this is fine in simple cases, it presents a serious security risk. When using buildbox-run-hosttools, actions have full access to the entire filesystem of the machine. This means that they can corrupt the files on the machine, access potentially sensitive information, or just leave behind unwanted files.

By using buildbox-run-userchroot, you automatically use userchroot to sandbox the actions, making the entire system much safer. To build buildbox-run-userchroot please refer to the repo’s README. Please note that you will need userchroot installed and configured on your machine in order to use it.

It is highly recommeded that you are using buildbox-casd with the local cas protocol enabled before using buildbox-run-userchroot. Otherwise it can be very slow.

Before using buildbox-run-userchroot you will need to upload a chroot image to CAS and get its digest. You can use casupload to upload chroot images and get their respective root digests. Refer to the README on recc to build casupload; building recc will automatically build casupload.

You will need to specify the root digest as a platform property on the client side, buildgrid’s config file, and in the workers’ arguments. To launch a worker with buildbox-run-userchroot enabled, run the following command:

sudo -u workeruser /path/to/buildbox-worker --bots-remote=http://localhost:50051 --cas-remote=http://127.0.0.1:50011 --buildbox-run=/path/to/buildbox-run-userchroot --platform chrootRootDigest=root-digest-of-chroot --runner-arg=--use-localcas --runner-arg=--workspace-path=/path/to/userchroot/base my_bot

Congratulations! You now have sandboxed workers!