.. _using_buildbox: What is buildbox? ----------------- buildbox_ is a set of tools for remote worker build execution, conforming to the `remote workers api`_, and the `remote execution api`_. It is highly configurable and extensible while also having support for Linux, AIX, and Solaris. To learn more about buildbox, its architecture and how to build the various tools, please visit the `buildbox landing page`_. Using buildbox-worker --------------------- This section will explain how to connect a simple ``buildbox-worker`` to BuildGrid. ``buildbox-worker`` is a worker written in C++ that implements the Remote Workers API. ``buildbox-worker`` runs its underlying job using a pluggable "runner." For this example, we will be using the ``buildbox-run-hosttools`` runner, which simply runs the command you send to it directly with no sandboxing. First, build ``buildbox-worker`` and ``buildbox-run-hosttools``. Instructions for each can be found in the respective READMEs. As part of the build process, you will also need to build and install ``buildbox-common``, which has several dependencies that need to be installed. Then, start a BuildGrid instance. If you have followed the installation guide and also have the BuildGrid source on your machine, you can use the default configuration: bgd server start data/config/default.yml Running the worker is pretty simple --you just need to pass it the locations of the BuildGrid server, BuildGrid CAS server, and ``buildbox-run-hosttools``. For example, if your server has an execution service and a CAS service both located at localhost:50051, you can invoke the worker like this:: /path/to/buildbox-worker --bots-remote=http://localhost:50051 --cas-remote=http://localhost:50051 --buildbox-run=/path/to/buildbox-run-hosttools my_bot If all goes well, you should have a functioning worker named ``my_bot`` that's ready to accept jobs. Using a local cache (buildbox-casd) ----------------------------------- Currently, whenever a worker needs to fetch blobs, it must make a request to the ``CAS``. Since the ``CAS`` is most likely on a separate machine, this can be an expensive operation. The worker's performance can be greatly improved if there is a local cache which minimized requests to the ``CAS``. ``buildbox-casd`` acts as a local cache; serving as a proxy to ``CAS`` for ``buildbox-worker``. If it doesn't have a blob which the worker requests, it will fetch it from ``CAS``, cache it, and then provide it to the worker. Consequently, future requests for the blob (by any workers on the machine) will hit ``buildbox-casd's`` local cache instead of the remote ``CAS``. You will first need to build ``buildbox-casd``. Instructions for how to do this can be found on it's README. Note that ``buildbox-casd`` also depends on ``buildbox-common``. You can then launch buildbox casd with the following command:: /path/to/buildbox-casd --cas-remote=http://localhost:50051 --bind=127.0.0.1:50011 /location/to/store/local/cache You can then launch the worker, using the following command:: /path/to/buildbox-worker --bots-remote=http://localhost:50051 --cas-remote=http://127.0.0.1:50011 --buildbox-run=/path/to/buildbox-run-hosttools --runner-arg=--prefix-staged-dir my_bot You can additionally tell the worker where to run the action with the workspace path argument:: /path/to/buildbox-worker --bots-remote=http://localhost:50051 --cas-remote=http://127.0.0.1:50011 --buildbox-run=/path/to/buildbox-run-hosttools --runner-arg=--prefix-staged-dir --runner-arg=--workspace-path=/stage/location my_bot You should now have local caching enabled on your worker setup. Enabling the local cas protocol ------------------------------- You can further optimize the local caching by enabling the local cas protocol. In order to understand what this does, we will need some definitions. ``Stage`` means placing the files needed to execute your action. ``Capture`` refers to saving the action's outputs. Enabling the local cas protocol means that ``buildbox-casd`` will stage and capture the files instead of ``buildbox-worker``. The benefit of this is that ``buildbox-casd`` can do this much more quickly and efficiently, resulting in a huge performance gain. In order to do this safely you will need to run ``buildbox-casd`` and ``buildbox-worker`` as separate users. This prevents the actions from corrupting the local cache. To launch buildbox-casd, use the following command:: sudo -u casduser /path/to/buildbox-casd --cas-remote=http://localhost:50051 --bind=127.0.0.1:50011 /location/to/store/local/cache To launch buildbox-worker, with the local cas protocol enabled, run:: sudo -u workeruser /path/to/buildbox-worker --bots-remote=http://localhost:50051 --cas-remote=http://127.0.0.1:50011 --buildbox-run=/path/to/buildbox-run-hosttools --runner-arg=--use-localcas --runner-arg=--prefix-staged-dir my_bot The local cas protocol should now be enabled. Enjoy your super fast builds! Sandboxed workers (buildbox-run-userchroot) ------------------------------------------- So far, we have been using ``buildbox-run-hosttools`` as our runner. While this is fine in simple cases, it presents a serious security risk. When using ``buildbox-run-hosttools``, actions have full access to the entire filesystem of the machine. This means that they can corrupt the files on the machine, access potentially sensitive information, or just leave behind unwanted files. By using ``buildbox-run-userchroot``, you automatically use userchroot_ to sandbox the actions, making the entire system much safer. To build ``buildbox-run-userchroot`` please refer to the repo's README. Please note that you will need userchroot_ installed and configured on your machine in order to use it. It is highly recommeded that you are using ``buildbox-casd`` with the local cas protocol enabled before using ``buildbox-run-userchroot``. Otherwise it can be very slow. Before using ``buildbox-run-userchroot`` you will need to upload a chroot image to ``CAS`` and get its digest. You can use casupload_ to upload chroot images and get their respective root digests. Refer to the README on recc_ to build casupload; building recc will automatically build casupload. You will need to specify the root digest as a platform property on the client side, buildgrid's config file, and in the workers' arguments. To launch a worker with ``buildbox-run-userchroot`` enabled, run the following command:: sudo -u workeruser /path/to/buildbox-worker --bots-remote=http://localhost:50051 --cas-remote=http://127.0.0.1:50011 --buildbox-run=/path/to/buildbox-run-userchroot --platform chrootRootDigest=root-digest-of-chroot --runner-arg=--use-localcas --runner-arg=--workspace-path=/path/to/userchroot/base my_bot Congratulations! You now have sandboxed workers! Resources --------- - `buildbox-worker`_ - `buildbox-run-hosttools`_ - `buildbox-run-userchroot`_ - `buildbox-casd`_ - `buildbox-common`_ .. _buildbox landing page: https://buildgrid.gitlab.io/buildbox/buildbox-home/ .. _buildbox: https://gitlab.com/BuildGrid/buildbox .. _buildbox-worker: https://gitlab.com/BuildGrid/buildbox/buildbox-worker .. _buildbox-run-hosttools: https://gitlab.com/BuildGrid/buildbox/buildbox-run-hosttools .. _buildbox-run-userchroot: https://gitlab.com/BuildGrid/buildbox/buildbox-run-userchroot .. _buildbox-casd: https://gitlab.com/BuildGrid/buildbox/buildbox-casd .. _buildbox-common: https://gitlab.com/BuildGrid/buildbox/buildbox-common .. _recc: https://gitlab.com/BuildGrid/recc .. _casupload: https://gitlab.com/BuildGrid/recc/tree/master/src/bin .. _userchroot: https://github.com/bloomberg/userchroot .. _remote workers api: https://docs.google.com/document/d/1s_AzRRD2mdyktKUj2HWBn99rMg_3tcPvdjx3MPbFidU/edit#heading=h.1u2taqr2h940 .. _remote execution api: https://github.com/bazelbuild/remote-apis