What is buildbox?
buildbox_ is a set of tools for remote worker build execution, conforming to the remote workers api, and the remote execution api. It is highly configurable and extensible while also having support for Linux, AIX, and Solaris. To learn more about buildbox, its architecture and how to build the various tools, please visit the buildbox landing page.
Using buildbox-worker
This section will explain how to connect a simple buildbox-worker to BuildGrid.
buildbox-worker is a worker written in C++ that implements the Remote Workers API.
buildbox-worker runs its underlying job using a pluggable “runner.” For this example,
we will be using the buildbox-run-hosttools runner, which simply runs the command
you send to it directly with no sandboxing.
First, build buildbox, instructions can be found in its README.
Then, start a BuildGrid instance. If you have followed the installation guide and also have the BuildGrid source on your machine, you can use the default configuration:
bgd server start data/config/default.yml
Running the worker is pretty simple –you just need to pass it the locations of the
BuildGrid server, BuildGrid CAS server, and buildbox-run-hosttools. For example,
if your server has an execution service and a CAS service both located at localhost:50051,
you can invoke the worker like this:
/path/to/buildbox-worker --bots-remote=http://localhost:50051 --cas-remote=http://localhost:50051 --buildbox-run=/path/to/buildbox-run-hosttools my_bot
If all goes well, you should have a functioning worker named my_bot that’s ready to accept jobs.
Using a local cache (buildbox-casd)
Currently, whenever a worker needs to fetch blobs, it must make a request to the CAS.
Since the CAS is most likely on a separate machine, this can be an expensive operation.
The worker’s performance can be greatly improved if there is a local cache which minimized
requests to the CAS.
buildbox-casd acts as a local cache; serving as a proxy to
CAS for buildbox-worker. If it doesn’t have a blob which the worker requests, it will
fetch it from CAS, cache it, and then provide it to the worker. Consequently, future
requests for the blob (by any workers on the machine) will hit buildbox-casd's local cache
instead of the remote CAS.
buildbox-casd is a part of buildbox, so it is built together with buildbox-worker.
You can then launch buildbox casd with the following command:
/path/to/buildbox-casd --cas-remote=http://localhost:50051 --bind=127.0.0.1:50011 /location/to/store/local/cache
You can then launch the worker, using the following command:
/path/to/buildbox-worker --bots-remote=http://localhost:50051 --cas-remote=http://127.0.0.1:50011 --buildbox-run=/path/to/buildbox-run-hosttools --runner-arg=--prefix-staged-dir my_bot
You can additionally tell the worker where to run the action with the workspace path argument:
/path/to/buildbox-worker --bots-remote=http://localhost:50051 --cas-remote=http://127.0.0.1:50011 --buildbox-run=/path/to/buildbox-run-hosttools --runner-arg=--prefix-staged-dir --runner-arg=--workspace-path=/stage/location my_bot
You should now have local caching enabled on your worker setup.
Enabling the local cas protocol
You can further optimize the local caching by enabling the local cas protocol. In order to understand what this does, we will need some definitions.
Stage means placing the files needed to execute your action.
Capture refers to saving the action’s outputs.
Enabling the local cas protocol means that buildbox-casd will stage and capture
the files instead of buildbox-worker. The benefit of this is that buildbox-casd can do
this much more quickly and efficiently, resulting in a huge performance gain.
In order to do this safely you will need to run buildbox-casd and buildbox-worker as
separate users. This prevents the actions from corrupting the local cache.
To launch buildbox-casd, use the following command:
sudo -u casduser /path/to/buildbox-casd --cas-remote=http://localhost:50051 --bind=127.0.0.1:50011 /location/to/store/local/cache
To launch buildbox-worker, with the local cas protocol enabled, run:
sudo -u workeruser /path/to/buildbox-worker --bots-remote=http://localhost:50051 --cas-remote=http://127.0.0.1:50011 --buildbox-run=/path/to/buildbox-run-hosttools --runner-arg=--use-localcas --runner-arg=--prefix-staged-dir my_bot
The local cas protocol should now be enabled. Enjoy your super fast builds!
Sandboxed workers (buildbox-run-userchroot)
So far, we have been using buildbox-run-hosttools as our runner.
While this is fine in simple cases, it presents a serious security risk. When using
buildbox-run-hosttools, actions have full access to the entire filesystem of the machine.
This means that they can corrupt the files on the machine, access potentially sensitive
information, or just leave behind unwanted files.
By using buildbox-run-userchroot, you automatically use userchroot to sandbox the actions, making the
entire system much safer. To build buildbox-run-userchroot please refer to the repo’s README. Please
note that you will need userchroot installed and configured on your machine in order to use it.
It is highly recommeded that you are using buildbox-casd with the local cas protocol
enabled before using buildbox-run-userchroot. Otherwise it can be very slow.
Before using buildbox-run-userchroot you will need to upload a chroot image to CAS and
get its digest. You can use casupload to upload chroot images and get their respective root digests.
Refer to the README on recc to build casupload; building recc will automatically build casupload.
You will need to specify the root digest as a platform property on the client side,
buildgrid’s config file, and in the workers’ arguments. To launch a worker with buildbox-run-userchroot
enabled, run the following command:
sudo -u workeruser /path/to/buildbox-worker --bots-remote=http://localhost:50051 --cas-remote=http://127.0.0.1:50011 --buildbox-run=/path/to/buildbox-run-userchroot --platform chrootRootDigest=root-digest-of-chroot --runner-arg=--use-localcas --runner-arg=--workspace-path=/path/to/userchroot/base my_bot
Congratulations! You now have sandboxed workers!