What is BuildGrid?

BuildGrid is a Python remote execution service which implements Google’s Remote Execution API and the Remote Workers API. The project’s goal is to be able to execute build jobs remotely on a grid of computers in order to massively speed up build times. Workers on the grid should be able to run with different environments. It works with clients such as Bazel, BuildStream and RECC, and is designed to be able to work with any client that conforms to the above API protocols.

BuildGrid is designed to work with any worker conforming to the Remote Workers API specification. Workers actually execute the jobs on the backend while BuildGrid does the scheduling and storage. The BuildBox ecosystem provides a suite of workers and sandboxing tools that work with the Workers API and can be used with BuildGrid.

What’s Going On?

Recently we’ve merged some work to update the S3 CAS and ActionCache implementations to use pycurl to send requests, rather than using boto3 directly. This approach has a lower Python CPU overhead, leading to less GIL contention, and also adds support for parallel requests providing further speedups to the Batch*Blobs requests. We’ve also addressed various other bugs and performance issues around the CAS implementations and metrics publishing.

We’ve also dropped support for Python 3.6, and added CI verification for Python 3.10.

See our release notes for the latest changes/updates.

Getting started

Please refer to the documentation for installation and usage instructions, plus guidelines for contributing to the project.