Migration

Some releases of BuildGrid introduce breaking changes which require a migration step, or changes which require consideration when upgrading to the new version.

The bgd migrate CLI can be used to apply any database migrations which have been added in a new release.

This section documents such changes in each release. If a release is missing from this page, it can be assumed to be too old to need documenting here, or to not have any such changes.

0.5.7

SQLIndex not using legacy table

This version contains the penultimate step of the SQLIndex migration; removing all reads and writes to the legacy index table.

With this change, index_metadata and the other tables added in 0.5.4 are the authoritative source for the contents of CAS. As such, this update should only be applied once the dual write and fallback reads have performed sufficient data migration that it is safe to rely on the new tables. A sensible time to wait is at least the length of your guaranteed blob lifespan in CAS. Alternatively, monitor the total size of the index calculated from each table, and only complete the migration once they are equal.

0.5.6

SQLIndex reading new tables

This version enables the new split index tables as the primary read location.

Reads which find a blob in the legacy table but not the new tables will replicate the entry into the new tables too.

No additional migration steps are necessary, but it is recommended to not change the inline blob threshold whilst this fallback behaviour exists. Doing so could result in double writes, which are functionally harmless but will increase the size of the index_access table through unnecessary timestamp churn.

0.5.4

SQLIndex schema changes

BuildGrid 0.5.4 includes a database migration which needs to be applied; which can be found at data/revisions/2b5f9653537e->689dc500937d.sql.

This migration adds three new tables for use by the SQLIndex; index_metadata, index_access, and index_blobs. These tables store the same data as the current table, but split across three tables to optimize the total size and metadata querying efficiency. The contents of the metadata and blobs tables are immutable, helping those tables to stay small even as the access timestamp is regularly updated.

Data is not migrated as part of this migration. Instead, a live migration path from the old table to the new tables will be provided via a series of releases.

In this first step of this change, the SQLIndex will dual write to the new tables but read only from the old table. In a future release we will switch to reading from the new table, with fallback to the old table, before a further change to disable all usage of the old table.

0.5.0

Removal of !assign-by-locality

The !assign-by-locality bot assignment strategy has been removed. Locality-based bot selection is now handled by !assign-by-capacity with a !sampling-config that includes locality in its !sampling-weights.

To migrate, replace any !assign-by-locality configuration with !assign-by-capacity using a !sampling-config. For example:

bot-assignment-strategy: !assign-by-capacity
  sampling: !sampling-config
    sample-size: 5
    weights: !sampling-weights
      locality: 5.0
      capacity: 1.0

The fallback parameter is no longer needed, as !assign-by-capacity with sampling weights naturally handles the case where no bot with a matching locality hint is available.

0.4.11

Scheduler schema changes

BuildGrid 0.4.11 includes a database migration which needs to be applied; which can be found at data/revisions/b1d6fb2ff810->2b5f9653537e.sql.

This migration adds a new index to the jobs table, to improve proactive scheduling performance (where bots pick up new jobs directly to help preserve queue order in high load scenarios).

0.4.10

Scheduler schema changes

BuildGrid 0.4.10 includes a database migration which needs to be applied; which can be found at data/revisions/146448692d39->b1d6fb2ff810.sql.

This migration adds a new priority column to the bots table, to support prioritising specific bots for assignment when using sampling assignment.

0.4.7

Scheduler schema changes

BuildGrid 0.4.7 includes a database migration which needs to be applied; which can be found at data/revisions/d1de6a7df71e->146448692d39.sql.

This migration adds new columns to the jobs table for the preferred property label and preferred requirements hash for the job. This is part of the preferred properties functionality added to the scheduler in this release, allowing some platform properties to be specified as preferred, but not strictly required, when considering which bot to assign a job to.

0.4.5

Minimum Python version

BuildGrid 0.4.5 bumps the minimum supported Python version to 3.12. The range of supported versions is now 3.12 - 3.14 inclusive.

0.4.1

Scheduler schema changes

BuildGrid 0.4.1 includes a database migration which needs to be applied; which can be found at data/revisions/3737630fc9cf->d1de6a7df71e.sql.

This migration adds a new table to keep a log of jobs cancelled by the cohort assigner when maintaining minimum quotas. See Eviction cooldown for details on how this log is used.

0.4.0

BuildGrid 0.4.0 introduces several changes which require care when migrating.

BuildBox version

Due to reintroduction of long-polling when bots have spare capacity, the minimum supported BuildBox version is now 1.3.51. Older versions of BuildBox will continue to work with BuildGrid 0.4.0, but bots with capacity of more than 1 will experience delays in communication of execution completion when they have spare capacity.

SQLIndex schema changes

The deleted column of the index table is removed. This column was used by the cleanup daemon to perform a three-step deletion process where blobs were marked as deleted in the index, removed from the backend storage, and then deleted from the index.

When migrating to BuildGrid 0.4.0 with an SQL index in use, any cleaners should be temporarily paused to avoid the upgrade effectively “undeleting” blobs which are part-way through this multi-step deletion.

Tip

Ensure that your cleaners are all updated to 0.4.0 before running the database migration to actually drop the deleted column.

A generated SQL database migration to apply this change can be found in data/revisions/ff09fbc30c3e->3737630fc9cf.sql.

Removal of SQLIndex fallback-on-get

The fallback-on-get configuration option for an SQL index is removed in this release. Usage of this option for data migration to a newly-added index should be replaced with a !replicated-storage with a replica for the newly indexed stack.