.. _notes_for_developers:

Notes for Developers
====================
.. warning:: **Invocation of the dependency generation script must happen within the "requirements" directory**.

.. _adding-updating-and-freezing-dependencies:

Adding dependencies
-------------------

Simply add the dependencies to the relevant dependency lists in the pyproject.toml file, and run:

    .. code-block:: sh

        tox -e requirements

.. warning:: Do not run this script outside of a Linux amd64 system or pip-compile will choose the wrong platform specific deps.

Upgrading the gRPC protobuf files
---------------------------------

Buildgrid's gRPC stubs are not built as part of installation. Instead, they are
precompiled and shipped with the source code. The protobufs are prone to
compatibility-breaking changes, so we update them manually.

First bring the updated proto file into the protos source tree. For example, if updating
the remote execution proto, replace the old
``protos/src/build/bazel/remote/execution/v2/remote_execution.proto``
with the newer one. Then, compile the protobufs.

    .. code-block:: shell

        tox -e protos

The regenerated protobufs will be available in ``buildgrid/server/_protos``. If adding a new protobuf
file, make sure to list the file in the ``protos/protos.yaml`` configuration!

.. _modifying-the-database-schema:

Modifying the database schema
-----------------------------

The database models are stored in ``buildgrid/server/persistence/sql/models.py``.
This is the source of truth for the database schema, and this file is what needs to
be updated in order to modify the schema.

To update the schema, make any needed changes to the ``models.py`` file. Then, you
need to generate a new revision and test the revision against a database. The easiest
way to do this is probably the postgres docker image (https://hub.docker.com/_/postgres).

    .. code-block:: shell

        docker pull postgres
        docker run --rm --name buildgrid-postgres -e POSTGRES_PASSWORD=pass -p 5432:5432 -d postgres

Now, install alembic (with ``pip install alembic``) and modify ``alembic.ini`` (the file in the root
of this repository) to point at our dockerized postgres database by editing the ``sqlalchemy.url`` field.

    .. code-block::

        sqlalchemy.url = postgres://postgres:pass@0.0.0.0:5432/postgres

Then, upgrade the database to the latest pre-revision state. Run this from the repository root.

    .. code-block:: shell

        alembic init alembic
        alembic upgrade head

Now, we can finally generate a new revision. Run this from the repository root.

    .. code-block:: shell

        alembic revision --autogenerate -m "a meaningful commit message"

This will generate a new revision file in ``buildgrid/server/persistence/sql/alembic/versions/``
that contains the difference between your old database and the new, updated model.

Some particulars worth noting:

- If you are adding a new index, please be sure to add it CONCURRENTLY in Postgres. This ensures
  that a migration can be performed on a database that is being locked by other processes (perhaps
  a running BuildGrid). This is accomplished with the `postgresql_concurrently`_ flag.

.. _postgresql_concurrently: https://docs.sqlalchemy.org/en/13/dialects/postgresql.html#indexes-with-concurrently

.. _working-with-timestamps:

Working with timestamps and timezones in Buildgrid
--------------------------------------------------

Currently, if the Index is enabled, Buildgrid will store timestamps in the Index Database.
These timestamps are stored as timezone-unaware objects in the database. This means that
the timestamps do not have any accompaning timezone information. As convention, Buildgrid
treats all timestamps as UTC time.

This results in some important considerations one must make when contributing to Buildgrid.
**You should always default to using UTC time when dealing with timestamps**. Not doing so
can break behavior in Buildgrid which requires proper ordering of timestamps. **This also means
that all timestamp objects should also be timezone-unaware**. If you use timezone-aware objects,
some libraries like SQLAlchemy will convert them to local time before comparing them with
timezone-unaware objects. This can break systems which rely on accurate timestamps
in Buildgrid.

Consequently, when contributing, please verify that if datetime objects are being used, they
are UTC time and timezone-unaware. For example, if you wish to get the current time, you should
always be using `datetime.utcnow()`_. Using variants which include timezone information can
create subtle bugs!

Additionally, when updating timestamp-sensitive code, it is always best practice to write
thorough unit tests. Even if the change may seem trivial, unit tests can reveal
hidden assumptions you are making.

.. _datetime.utcnow(): https://docs.python.org/3/library/datetime.html#datetime.datetime.utcnow