Notes for Developers

Warning

Invocation of the dependency generation script must happen within the “requirements” directory.

Adding dependencies

Simply add the dependencies to the relevant dependency lists in the pyproject.toml file, and run:

tox -e requirements

Warning

Do not run this script outside of a Linux amd64 system or pip-compile will choose the wrong platform specific deps.

Upgrading the gRPC protobuf files

Buildgrid’s gRPC stubs are not built as part of installation. Instead, they are precompiled and shipped with the source code. The protobufs are prone to compatibility-breaking changes, so we update them manually.

First bring the updated proto file into the protos source tree. For example, if updating the remote execution proto, replace the old protos/src/build/bazel/remote/execution/v2/remote_execution.proto with the newer one. Then, compile the protobufs.

tox -e protos

The regenerated protobufs will be available in buildgrid/server/_protos. If adding a new protobuf file, make sure to list the file in the protos/protos.yaml configuration!

Modifying the database schema

The database models are stored in buildgrid/server/sql/models.py. This is the source of truth for the database schema, and this file is what needs to be updated in order to modify the schema.

To update the schema, make any needed changes to the models.py file. Then, generate an alembic migration script using tox. Note that this requires docker installed on your machine.

tox -e migrate -- -m "Description of your change here"

This will create a new migration script in buildgrid/server/sql/alembic/versions. Check and tweak this script as necessary to make sure its making just the changes required for your model change.

Some particulars worth noting:

• If you are adding a new index, please be sure to add it CONCURRENTLY in Postgres. This ensures that a migration can be performed on a database that is being locked by other processes (perhaps a running BuildGrid). This is accomplished with the postgresql_concurrently flag.

Use this migration to regenerate the SQL for database initialization using the sql tox environment.

tox -e sql
tox -e sql -- $PREVIOUS_REVISION_NUMBER:$NEW_REVISION_NUMBER

This will generate two SQL files in data/revisions, all.sql and one with the two revision numbers in the name.

The previous and new revision numbers are contained in the migration script generated in the previous step.

Working with timestamps and timezones in Buildgrid

Currently, if the Index is enabled, Buildgrid will store timestamps in the Index Database. These timestamps are stored as timezone-unaware objects in the database. This means that the timestamps do not have any accompaning timezone information. As convention, Buildgrid treats all timestamps as UTC time.

This results in some important considerations one must make when contributing to Buildgrid. You should always default to using UTC time when dealing with timestamps. Not doing so can break behavior in Buildgrid which requires proper ordering of timestamps. This also means that all timestamp objects should also be timezone-unaware. If you use timezone-aware objects, some libraries like SQLAlchemy will convert them to local time before comparing them with timezone-unaware objects. This can break systems which rely on accurate timestamps in Buildgrid.

Consequently, when contributing, please verify that if datetime objects are being used, they are UTC time and timezone-unaware. For example, if you wish to get the current time, you should always be using datetime.utcnow(). Using variants which include timezone information can create subtle bugs!

Additionally, when updating timestamp-sensitive code, it is always best practice to write thorough unit tests. Even if the change may seem trivial, unit tests can reveal hidden assumptions you are making.