BuildStream client

BuildStream is a free software tool for building and integrating software stacks. It supports remote build execution using the remote execution API (REAPI) v2. The project’s documentation has a detailed section about its remote execution subsystem architecture that you are very recommanded to read first.

Note

There is no stable release of BuildStream with support for remote execution yet. You’ll have to install it from sources in order to build with remote execution.

Configuration

BuildStream uses YAML for build definition and configuration. It has two levels of configuration: user and project level. User-level configuration is stored in your buildstream.yml file while project-level configuration is defined in each project’s project.yml file.

Note

At the moment, remote execution can only be configured at project-level.

Project configuration

In order to activate remote build execution at project-level, the project’s project.yml file must declare two specific configuration nodes:

Important

BuildStream does not support multi-instance remote execution servers and will always submit remote execution request omitting the instance name parameter. Thus, you must declare an unnamed ‘’ instance in your server configuration to workaround this.

Important

If you are using BuildStream’s artifact server, the server instance pointed by the storage-service key must accept pushes from your client for remote execution to be possible.

BuildBox tool

BuildStream performs builds in a sandbox on top of a project-defined environment, not relying on any host-tools. BuildGrid supports this kind of build using the buildbox-worker bot in conjunction with buildbox-run-bubblewrap.

BuildBox can execute build commands in a sandbox on top of an environment constructed from provided sources using FUSE. It also uses bubblewrap for sandboxing without requiring root privileges.

BuildBox being a rather young project, it isn’t packaged yet and you’ll have to build it from source. You may want follow the BuildBox installation instructions or you can build it with BuildStream using the dedicated integration project. You’ll need to build buildbox-common, buildbox-worker, and buildbox-run-bubblewrap.

Important

Whatever the method you use to install BuildBox, you also have to install bubblewrap along, minimum required version being 0.1.8.

Example build

The BuildStream repository contains example projects used for testing purpose in the project’s usage documentation section. We’ll focus here on instructions on how to build the autotools example running BuildStream and BuildGrid on your local machine, compiling the GNU Automake hello example program in a sandbox on top of a minimal Alpine Linux environment.

First, you need to checkout the buildstream repository sources:

git clone https://gitlab.com/BuildStream/buildstream.git

Next, change the current directory to the BuildStream project root:

cd buildstream/doc/examples/autotools

Hint

All the commands in the instructions below are expected to be executed from that root directory (the autotools example project’s root directory).

Before running BuildStream and building the example project, you’ll have to setup and run a BuildGrid server and bot. A minimal server’s configuration is given below, paste it in a server.yml file in the root directory:

server:
  - !channel
    address: localhost:50051
    insecure-mode: true

connections:
  - !sql-connection &sql
    connection-string: sqlite:///./example.db
    automigrate: yes
    connection-timeout: 15

storages:
  - !lru-storage &main-storage
    size: 512MB

caches:
  - !lru-action-cache &main-action
    storage: *main-storage
    max-cached-refs: 256
    allow-updates: true

schedulers:
  - !sql-scheduler &scheduler
    sql: *sql
    storage: *main-storage
    action-cache: *main-action
    poll-interval: 0.5
    max-execution-timeout: 7200

instances:
  - name: ''

    services:
      - !action-cache
        cache: *main-action

      - !execution
        scheduler: *scheduler

      - !cas
        storage: *main-storage

      - !bytestream
        storage: *main-storage

This defines a single unnamed server instance implementing a ContentAddressableStorage (CAS) + Reference + ByteStream service together with an Execution + ActionCache service, both using the same in-memory storage. You can then start the BuildGrid server daemon using that configuration by running:

bgd server start server.yml

In order to perform the actual build work, you need to attach a buildbox-worker bot to that server for that unnamed instance. Once you’ve built and installed buildbox-worker and buildbox-run-bubblewrap, you can run the bot with:

buildbox-worker \
   --buildbox-run=buildbox-run-bubblewrap \
   --bots-remote=http://localhost:50051 \
   --cas-remote=http://localhost:50051 \
   --request-timeout=30 \
   --instance-name= \
   --platform OSFamily=linux \
   --platform ISA=x86-64

The BuildGrid server is now ready to accept jobs and execute them. The example project needs some configuration tweaks in order to be build remotely. Below is the configuration fragment you should append at the end of the project.yml file from the root directory:

artifacts:
  url: http://localhost:50051
  push: true
  type: storage

remote-execution:
  execution-service:
    url: http://localhost:50051
  storage-service:
    url: http://localhost:50051
    client-key: ''
    client-cert: ''
    server-cert: ''
  action-cache-service:
    url: http://localhost:50051

This activates BuildGrid’s remote execution mode and points to the unnamed remote execution server instance at localhost:50051.

You can finally have BuildStream to build the example project by running:

bst build hello.bst

You can verify that the example has been successfully built by running the generated executable. Simply invoke:

bst shell hello.bst -- hello