BuildStream is a free software tool for building and integrating software stacks. It supports remote build execution using the remote execution API (REAPI) v2. The project’s documentation has a detailed section about its remote execution subsystem architecture that you are very recommanded to read first.
There is no stable release of BuildStream with support for remote execution yet. You’ll have to install it from sources in order to build with remote execution.
BuildStream uses YAML for build definition and configuration. It has two
levels of configuration: user and project level. User-level configuration is
stored in your
buildstream.conf file while project-level configuration
is defined in each project’s
At the moment, remote execution can only be configured at project-level.
In order to activate remote build execution at project-level, the project’s
project.conf file must declare two specific configuration nodes:
artifactsfor remote cache endpoint details.
remote-executionfor remote execution endpoint details.
BuildStream does not support multi-instance remote execution servers and will always submit remote execution request omitting the instance name parameter. Thus, you must declare an unnamed ‘’ instance in your server configuration to workaround this.
If you are using BuildStream’s artifact server, the server instance pointed
storage-service key must accept pushes from your client for
remote execution to be possible.
BuildStream performs builds in a sandbox on top of a project-defined
environment, not relying on any host-tools. BuildGrid supports this kind of
build using the
buildbox bot, a specific type of bot relying on BuildBox
for build execution.
BuildBox being a rather young project, it isn’t packaged yet and you’ll have to build it from source. You may want follow the manual instructions or you can build it with BuildStream using the dedicated integration project (recommanded).
Whatever the method you use to install BuildBox, you also have to install bubblewrap along, minimum required version being 0.1.8.
The BuildStream repository contains example projects used for testing purpose in the project’s usage documentation section. We’ll focus here on instructions on how to build the autotools example running BuildStream and BuildGrid on your local machine, compiling the GNU Automake hello example program in a sandbox on top of a minimal Alpine Linux environment.
First, you need to checkout the buildstream repository sources:
git clone https://gitlab.com/BuildStream/buildstream.git
Next, change the current directory to the BuildStream project root:
All the commands in the instructions below are expected to be executed from that root directory (the autotools example project’s root directory).
Before running BuildStream and building the example project, you’ll have to setup
and run a BuildGrid server and bot. A minimal server’s configuration is given
below, paste it in a
server.conf file in the root directory:
server: - !channel port: 50051 insecure-mode: true instances: - name: '' storages: - !lru-storage &main-storage size: 512MB services: - !action-cache &main-action storage: *main-storage max-cached-refs: 256 allow-updates: true - !execution storage: *main-storage action-cache: *main-action - !cas storage: *main-storage - !bytestream storage: *main-storage - !reference-cache storage: *main-storage max-cached-refs: 128
This defines a single unnamed server instance implementing a
ContentAddressableStorage (CAS) +
together with an
ActionCache service, both using the
same in-memory storage. You can then start the BuildGrid server daemon using
that configuration by running:
bgd server start server.conf
In order to perform the actual build work, you need to attach a
worker bot to that server for that unnamed instance. Once you’ve make sure
buildbox tool is functional on your machine (refer to
bgd bot --remote=http://localhost:50051 --instance-name= buildbox --fuse-dir=fuse --local-cas=cache
--remote option is used to specify the server location (running on the
same machine here, and listening to port 50051). The
--instance-name option is
used to specify the server instance you expect the bot to be attached to (empty
--local-cas are specific to the
and respectively specify the sandbox mount point and local CAS cache locations.
Refer to the CLI reference section for
for command line interface details.
The BuildGrid server is now ready to accept jobs and execute them. The example
project needs some configuration tweaks in
order to be build remotely. Below is the configuration fragment you should
append at the end of the
project.conf file from the root directory:
artifacts: url: http://localhost:50051 push: true type: storage remote-execution: execution-service: url: http://localhost:50051 storage-service: url: http://localhost:50051 client-key: '' client-cert: '' server-cert: '' action-cache-service: url: http://localhost:50051
This activates BuildGrid’s remote execution mode and points to the unnamed
remote execution server instance at
You can finally have BuildStream to build the example project by running:
bst build hello.bst
You can verify that the example has been successfully built by running the generated executable. Simply invoke:
bst shell hello.bst -- hello