What is BuildGrid?

BuildGrid is a Python remote execution service which implements Google’s Remote Execution API and the Remote Workers API. The project’s goal is to be able to execute build jobs remotely on a grid of computers in order to massively speed up build times. Workers on the grid should be able to run with different environments. It works with clients such as Bazel, BuildStream and RECC, and is designed to be able to work with any client that conforms to the above API protocols.

BuildGrid is designed to work with any worker conforming to the Remote Workers API specification. Workers actually execute the jobs on the backend while BuildGrid does the scheduling and storage. The BuildBox ecosystem provides a suite of workers and sandboxing tools that work with the Workers API and can be used with BuildGrid.

What’s Going On?

Recently we finished implementing basic CAS expiry for the S3 storage backend, when used alongside an SQL CAS index. This uses a new bgd cleanup command, use of which is described in the documentation.

“Permissive Bot Session mode” has been added to the Bots service, which makes it possible for bots to reconnect transparently after a server restart with an UpdateBotSession request rather than that causing an error and loss of in-progress leases. However, bots must still connect to the same server instance every request, otherwise the BotSession reaper may terminate leases that are actually progressing without issue.

A number of new metrics have been added in the last few months too, allowing monitoring of the state of CAS and the ActionCache.

We’ve also made the validation of the YAML configuration file more strict, to provide more useful error messages when something is wrong. A side-effect of this is that now all keys in the file must use - as the word separator. Previously some keys (notably the keys used with custom YAML tags) allowed _ to be used interchangably.

Next we’re working on adding support for streaming logs from workers back to clients as Actions are being executed.

See our release notes for the latest changes/updates.

Getting started

Please refer to the documentation for installation and usage instructions, plus guidelines for contributing to the project.