API reference

BuildGrid’s Application Programming Interface (API) reference documentation.

• buildgrid

Configuration Parser API

The tagged YAML nodes in the reference configuration are handled by the YAML parser using the following set of objects:

buildgrid.server.app.settings.parser.object_tag(kind: str, deprecated_keys: dict[str, str] | None = None) → Callable[[_Func], _Func]

Register a tag with custom decoder logic for a yaml object field.

buildgrid.server.app.settings.parser.string_tag(kind: str) → Callable[[_Func], _Func]

Register a tag with custom decoder logic for a yaml string value field.

buildgrid.server.app.settings.parser.load_channel(insecure_mode: bool, address: str, credentials: dict[str, str] | None = None) → ChannelConfig

Creates a GRPC channel.

The Channel class returns a grpc.Channel and is generated from the tag !channel. Creates either a secure or insecure channel.

Usage

- !channel
  address (str): Address for the channel. (For example,
    'localhost:50055' or 'unix:///tmp/sock')
  port (int): A port for the channel (only if no address was specified).
  insecure-mode: false
  credentials:
    tls-server-key: !expand-path ~/.config/buildgrid/server.key
    tls-server-cert: !expand-path ~/.config/buildgrid/server.cert
    tls-client-certs: !expand-path ~/.config/buildgrid/client.cert

Parameters:

• port (int) – A port for the channel.

• insecure_mode (bool) – If True, generates an insecure channel, even if there are credentials. Defaults to True.

• credentials (dict, optional) –

  A dictionary in the form:

  tls-server-key: /path/to/server-key
  tls-server-cert: /path/to/server-cert
  tls-client-certs: /path/to/client-certs
  

buildgrid.server.app.settings.parser.expand_path(path: str) → str

Returns a string of the user’s path after expansion.

The ExpandPath class returns a string and is generated from the tag !expand-path.

Usage

path: !expand-path ~/bgd-data/cas

Parameters:

path (str) – Can be used with strings such as: ~/dir/to/something or $HOME/certs

buildgrid.server.app.settings.parser.expand_vars(value: str) → str

Expand environment variables in a string.

The ExpandVars class returns a string and is generated from the tag !expand-vars.

Usage

endpoint: !expand-vars $ENDPOINT

Parameters:

path (str) – Can be used with strings such as: http://$ENDPOINT

buildgrid.server.app.settings.parser.read_file(path: str) → str

Returns a string of the contents of the specified file.

The ReadFile class returns a string and is generated from the tag !read-file.

Usage

secret_key: !read-file /var/bgd/s3-secret-key

Parameters:

path (str) – Can be used with strings such as: ~/path/to/some/file or $HOME/myfile or /path/to/file

buildgrid.server.app.settings.parser.load_disk_storage(path: str) → DiskStorage

Generates buildgrid.server.cas.storage.disk.DiskStorage using the tag !disk-storage.

Usage

- !disk-storage
  path: /opt/bgd/cas-storage

Parameters:

path (str) – Path to directory to storage.

buildgrid.server.app.settings.parser.load_lru_storage(size: str) → LRUMemoryCache

Generates buildgrid.server.cas.storage.lru_memory_cache.LRUMemoryCache using the tag !lru-storage.

Usage

- !lru-storage
  size: 2048M

Parameters:

size (int) – Size e.g 10kb. Size parsed with buildgrid.server.app.settings.parser._parse_size().

buildgrid.server.app.settings.parser.load_s3_storage(bucket: str, endpoint: str, access_key: str, secret_key: str, read_timeout_seconds_per_kilobyte: float | None = None, write_timeout_seconds_per_kilobyte: float | None = None, read_timeout_min_seconds: float = 120, write_timeout_min_seconds: float = 120, versioned_deletes: bool = False, hash_prefix_size: int | None = None, path_prefix_string: str | None = None) → S3Storage

Generates buildgrid.server.cas.storage.s3.S3Storage using the tag !s3-storage.

Usage

- !s3-storage
  bucket: bgd-bucket-{digest[0]}{digest[1]}
  endpoint: http://127.0.0.1:9000
  access_key: !read-file /var/bgd/s3-access-key
  secret_key: !read-file /var/bgd/s3-secret-key
  read_timeout_seconds_per_kilobyte: 0.01
  write_timeout_seconds_per_kilobyte: 0.01
  read_timeout_min_seconds: 120
  write_timeout_min_seconds: 120

Parameters:

• bucket (str) – Name of bucket

• endpoint (str) – URL of endpoint.

• access-key (str) – S3-ACCESS-KEY

• secret-key (str) – S3-SECRET-KEY

• read_timeout_seconds_per_kilobyte (float) – S3 Read timeout in seconds/kilobyte

• write_timeout_seconds_per_kilobyte (float) – S3 Write timeout in seconds/kilobyte

• read_timeout_min_seconds (float) – The minimal timeout for S3 read

• write_timeout_min_seconds (float) – The minimal timeout for S3 writes

• versioned_deletes (bool) – Query and use the VersionId when performing deletes.

• hash-prefix-size (int) – Number of hash characters to use as prefix in s3 object name.

• path-prefix-string (str) – Additional string for path prefix

buildgrid.server.app.settings.parser.load_redis_connection(host: str | None = None, port: int | None = None, password: str | None = None, db: int | None = None, dns_srv_record: str | None = None, sentinel_master_name: str | None = None, retries: int = 3) → RedisProvider

Generates buildgrid.server.redis.provider.RedisProvider using the tag !redis-connection

Usage

- !redis-connection
  host: redis
  port: 6379
  password: !read-file /var/bgd/redis-pass
  db: 0
  dns-srv-record: <Domain name of SRV record>
  sentinel-master-name: <service_name of Redis sentinel's master instance>
  retries: 3

Parameters:

• host (str | None) – The hostname of the Redis server to use.

• port (int | None) – The port that Redis is served on.

• password (str | None) – The Redis database password to use.

• db (int) – The Redis database number to use.

• dns-srv-record (str) – Domain name of SRV record used to discover host/port

• sentinel-master-name (str) – Service name of Redis master instance, used in a Redis sentinel configuration

• retries (int) – Max number of times to retry (default 3). Backoff between retries is about 2^(N-1), where N is the number of attempts

buildgrid.server.app.settings.parser.load_redis_storage(redis: RedisProvider) → RedisStorage

Generates buildgrid.server.cas.storage.redis.RedisStorage using the tag !redis-storage.

Usage

# This assumes that the YAML anchors are defined elsewhere
- !redis-storage
  redis: *redis-connection

Parameters:

redis (buildgrid.server.redis.provider.RedisProvider) – A configured Redis connection manager. This must be an object with an !redis-connection YAML tag.

buildgrid.server.app.settings.parser.load_redis_index(storage: StorageABC, redis: RedisProvider, prefix: str | None = None) → RedisIndex

Generates buildgrid.server.cas.storage.index.redis.RedisIndex using the tag !redis-index.

Usage

- !redis-index
  # This assumes that a storage instance is defined elsewhere
  # with a `&cas-storage` anchor
  storage: *cas-storage
  redis: *redis
  prefix: "B"

Parameters:

• storage (buildgrid.server.cas.storage.storage_abc.StorageABC) – Instance of storage to use. This must be a storage object constructed using a YAML tag ending in -storage, for example !disk-storage.

• redis (buildgrid.server.redis.provider.RedisProvider) – A configured Redis connection manager. This must be an object with an !redis-connection YAML tag.

• prefix (str) – An optional prefix to use to prefix keys written by this index. If not specified a prefix of “A” is used.

buildgrid.server.app.settings.parser.load_replicated_storage(storages: list[buildgrid.server.cas.storage.storage_abc.StorageABC], replication_queue_size: int = 0) → ReplicatedStorage

Generates buildgrid.server.cas.storage.replicated.ReplicatedStorage using the tag !replicated-storage.

Usage

- !replicated-storage
  storages:
    - &storageA
    - &storageB
  replication-queue-size: 10000

Parameters:

• Storages (list) – List of storages to mirror reads/writes for. A minimum of two storages is required.

• replication-queue-size (int) – Length of the replication queue used to replicate inconsistent blobs found during FMB. If not present or set to 0 the replication queue is disabled (default 0).

buildgrid.server.app.settings.parser.load_remote_storage(url: str, instance_name: str | None = None, credentials: ClientCredentials | None = None, channel_options: dict[str, Any] | None = None, retries: int = 3, max_backoff: int = 64, request_timeout: float | None = None) → RemoteStorage

Generates buildgrid.server.cas.storage.remote.RemoteStorage using the tag !remote-storage.

Usage

- !remote-storage
  url: https://storage:50052/
  instance-name: main
  credentials:
    tls-server-key: !expand-path ~/.config/buildgrid/server.key
    tls-server-cert: !expand-path ~/.config/buildgrid/server.cert
    tls-client-certs: !expand-path ~/.config/buildgrid/client.cert
    auth-token: /path/to/auth/token
    token-refresh-seconds: 6000
  channel-options:
    lb-policy-name: round_robin
  request-timeout: 15

Parameters:

• url (str) – URL to remote storage. If used with https, needs credentials.

• instance_name (str) – Instance of the remote to connect to. If none, defaults to the instance context.

• credentials (dict, optional) –

  A dictionary in the form:

  tls-client-key: /path/to/client-key
  tls-client-cert: /path/to/client-cert
  tls-server-cert: /path/to/server-cert
  auth-token: /path/to/auth/token
  token-refresh-seconds (int): seconds to wait before reading the token from the file again
  

• channel-options (dict, optional) –

  A dictionary of grpc channel options in the form:

  some-channel-option: channel_value
  other-channel-option: another-channel-value
  

• https (See) – //github.com/grpc/grpc/blob/master/include/grpc/impl/codegen/grpc_types.h

• options (for the valid channel) –

• retries (int) – Max number of times to retry (default 3). Backoff between retries is about 2^(N-1), where N is the number of attempts

• max_backoff (int) – Maximum backoff in seconds (default 64)

• request_timeout (float) – gRPC request timeout in seconds (default None)

buildgrid.server.app.settings.parser.load_with_cache_storage(cache: StorageABC, fallback: StorageABC, defer_fallback_writes: bool = False, fallback_writer_threads: int = 20) → WithCacheStorage

Generates buildgrid.server.cas.storage.with_cache.WithCacheStorage using the tag !with-cache-storage.

Usage

- !with-cache-storage
  cache:
    !lru-storage
    size: 2048M
  fallback:
    !disk-storage
    path: /opt/bgd/cas-storage
  defer-fallback-writes: no

Parameters:

• cache (StorageABC) – Storage instance to use as a cache

• fallback (StorageABC) – Storage instance to use as a fallback on cache misses

• defer-fallback-writes (bool) – If true, commit_write returns once writing to the cache is done, and the write into the fallback storage is done in a background thread

• fallback-writer-threads (int) – The maximum number of threads to use for writing blobs into the fallback storage. Defaults to 20.

class buildgrid.server.app.settings.parser.ShardType

Bases: TypedDict

name: str

storage: StorageABC

buildgrid.server.app.settings.parser.load_sharded_storage(shards: list[buildgrid.server.app.settings.parser.ShardType], thread_pool_size: int | None = None) → ShardedStorage

Generates buildgrid.server.cas.storage.Sharded.ShardedStorage using the tag !sharded-storage.

Usage

- !sharded-storage
  shards:
    - name: A
      storage: &storageA
    - name: B
      storage: !lru-storage
        size: 2048M
  thread-pool-size: 40

Parameters:

• shards (list) – List of dictionaries. The dictionaries are expected to have name and storage keys defining a storage shard. The name must be unique within a configuration and should be the same for any configuration using the same underlying storage.

• thread-pool-size (int|None) – Number of worker threads to use for bulk methods to allow parallel requests to each shard. If not set no threadpool is created and requests are made serially to each shard.

buildgrid.server.app.settings.parser.load_size_differentiated_storage(size_limited_storages: list[buildgrid.server.app.settings.parser._SizeLimitedStorageConfig], fallback: StorageABC, thread_pool_size: int | None = None) → SizeDifferentiatedStorage

Generates buildgrid.server.cas.storage.size_differentiated.SizeDifferentiatedStorage using the tag !size-differentiated-storage.

Usage

- !size-differentiated-storage
  size-limited-storages:
    - max-size: 1M
      storage:
        !lru-storage
        size: 2048M
  fallback:
    !disk-storage
    path: /opt/bgd/cas-storage
  thread-pool-size: 40

Parameters:

• size_limited_storages (list) – List of dictionaries. The dictionaries are expected to have max-size and storage keys, defining a storage provider to use to store blobs with size up to max-size.

• fallback (StorageABC) – Storage instance to use as a fallback for blobs which are too big for the options defined in size_limited_storages.

• thread-pool-size (int|None) – Number of worker threads to use for bulk methods to allow parallel requests to each storage. This thread pool is separate from the gRPC server thread-pool-size and should be tuned separately. If not set no threadpool is created and requests are made serially to each storage.

buildgrid.server.app.settings.parser.load_sql_storage(sql: SqlProvider, sql_ro: SqlProvider | None = None) → SQLStorage

Generates buildgrid.server.cas.storage.sql.SQLStorage using the tag !sql-storage.

Usage

- !sql-storage
  sql: *sql
  sql_ro: *sql

Parameters:

• sql (buildgrid.server.sql.provider.SqlProvider) – A configured SQL connection manager. This must be an object with an !sql-connection YAML tag.

• sql_ro (buildgrid.server.sql.provider.SqlProvider) – Similar to sql, but used for readonly backend transactions. If set, it should be configured with a replica of main DB using an optional but encouraged readonly role. Permission check is not executed by BuildGrid. If not set, readonly transactions are executed by sql object.

buildgrid.server.app.settings.parser.load_sql_connection(connection_string: str | None = None, connection_timeout: int = 5, lock_timeout: int = 5, connect_args: dict[str, Any] | None = None, max_overflow: int | None = None, pool_pre_ping: bool | None = None, pool_recycle: int | None = None, pool_size: int | None = None, pool_timeout: int | None = None, name: str = 'sql-provider') → SqlProvider

Generates buildgrid.server.sql.provider.SqlProvider using the tag !sql-connection.

Example

- !sql-connection &sql
  connection_string: postgresql://bgd:insecure@database/bgd
  connection_timeout: 5
  lock_timeout: 5
  pool-size: 5
  pool-timeout: 30
  max-overflow: 10
  name: "sql-pool"

buildgrid.server.app.settings.parser.load_sql_scheduler(storage: StorageABC, sql: SqlProvider, sql_ro: SqlProvider | None = None, sql_notifier: SqlProvider | None = None, property_set: PropertySet | None = None, pruner_job_max_age: dict[str, float] | None = None, pruner_period: dict[str, float] | None = None, pruner_max_delete_window: int | None = None, queue_timeout_job_max_age: dict[str, float] | None = None, queue_timeout_period: dict[str, float] | None = None, queue_timeout_max_window: int | None = None, action_cache: ActionCacheABC | None = None, action_browser_url: str | None = None, max_execution_timeout: int = 7200, metering_service_client: SyncMeteringServiceClient | None = None, metering_throttle_action: str | None = None, bot_session_keepalive_timeout: int = 600, logstream: dict[str, Any] | None = None, asset_client: AssetClient | None = None, queued_action_retention_hours: float | None = None, completed_action_retention_hours: float | None = None, action_result_retention_hours: float | None = None, max_job_attempts: int = 5, priority_assignment_percentage: int = 100, poll_interval: float = 1.0, job_assignment_interval: float = 1.0, max_queue_size: int | None = None, execution_timer_interval: float = 60.0, session_expiry_timer_interval: float = 10.0, instance_pools: list[list[str]] | None = None, property_keys: str | list[str] | None = None, wildcard_property_keys: str | list[str] | None = None) → Scheduler

Generates buildgrid.server.scheduler.Scheduler using the tag !sql-scheduler.

Example

- !sql-scheduler
  storage: *cas-storage
  sql: *sql
  pruner-job-max-age:
    days: 90

This usage example assumes that the cas-storage reference refers to a storage backend, eg. !disk-storage, and the sql reference refers to an SQL connection manager using !sql-connection.

Parameters:

• storage (buildgrid.server.cas.storage.storage_abc.StorageABC) – Instance of storage to use for getting actions and storing job results. This must be an object constructed using a YAML tag ending in -storage, for example !disk-storage.

• sql (buildgrid.server.sql.provider.SqlProvider) – A configured SQL connection manager. This must be an object with an !sql-connection YAML tag.

• sql_ro (buildgrid.server.sql.provider.SqlProvider) – Similar to sql, but used for readonly backend transactions. If set, it should be configured with a replica of main DB using an optional but encouraged readonly role. Permission check is not executed by BuildGrid. If not set, readonly transactions are executed by sql object.

• sql_notifier (buildgrid.server.sql.provider.SqlProvider) – Similar to sql, but used for operation notifier. If not set, transactions are executed by sql object.

• property_set (PropertySet) – Controls how execute requests are assigned to workers.

• pruner_job_max_age (dict) – Allow the storage to remove old entries by specifying the maximum amount of time that a row should be kept after its job finished. If this value is None, pruning is disabled and the background pruning thread is never created.

• pruner_period (dict) – How often to attempt to remove old entries. If pruning is enabled (see above) and this value is None, it is set to 5 minutes by default.

• pruner_max_delete_window (int) – Maximum number of records removed in a single cleanup pass. If pruning is enabled and this value is None, it is set to 10000 by default. This allows to put a limit on the time that the database will be blocked on a single invocation of the cleanup routine. (A smaller value reduces the performance impact of removing entries, but makes the recovery of storage space slower.)

• queue_timeout_job_max_age (dict) – If set, allow storage to abort jobs that have been queued for a long period of time.

• queue_timeout_period (dict) – How often to find aged queued jobs. If not set, default to 5 minutes.

• queue_timeout_max_window (int) – Maximum number of jobs to timeout per batch. If not set, default to 10000.

• action_cache (ActionCache) – Instance of action cache to use.

• action_browser_url (str) – The base URL to use to generate Action Browser links to users. If a single Web interface serves several Buildgrid installations then this URL should include the namespace configured for the current Buildgrid installation, see https://gitlab.com/BuildGrid/bgd-browser#multi-buildgrid-setup.

• max_execution_timeout (int) – The maximum time jobs are allowed to be in ‘OperationStage.EXECUTING’. This is a periodic check. When this time is exceeded in executing stage, the job will be cancelled.

• metering_service_client – Optional client to check whether resource usage of a client is above a predefined threshold

• metering_throttle_action – The action to perform when metering service returns that job should be throttled. Can be set to “deprioritize” or “reject”. Defaults to “deprioritize”.

• bot_session_keepalive_timeout (int) – The longest time (in seconds) we’ll wait for a bot to send an update before it assumes it’s dead. Defaults to 600s (10 minutes).

• logstream (Dict) – Configuration options for connecting a logstream instance to ongoing jobs. Is a dict with items “url”, “credentials”, and “instance-name”

• asset_client (AssetClient | None) – Client of remote-asset service

• queued_action_retention_hours (float | None) – Minimum retention for queued actions in hours

• completed_action_retention_hours (float | None) – Minimum retention for completed actions in hours

• action_result_retention_hours (float | None) – Minimum retention for action results in hours

• max_job_attempts (int) – The number of times a job will be assigned to workers before marking the job failed. Reassignment happens when a worker fails to report the outcome of a job. Minimum value allowed is 1. Default value is 5.

• priority_assignment_percentage (int) – A value between 0 and 100 (inclusive) representing the percentage of workers to assign jobs to in priority order. The remainder will be assigned work in oldest-first order. Defaults to 100, or all work assigned in priority order.

• poll_interval (float) – Duration to wait between polling operation updates.

• job_assignment_interval (float) – Duration to wait between scheduling queries.

• max_queue_size (int) – Maximum number of jobs queued per platform property set.

• execution_timer_interval (float) – Duration to wait between attempts to executions exceeding timeout.

• session_expiry_timer_interval (float) – Duration to wait between attempts to close unresponsive bot sessions

• instance_pools (list[list[str]] | None) – List of lists of instance names to schedule together

• property_keys (list) – Deprecated. Use a property_set instead.

• wildcard_property_keys (list) – Deprecated. Use a property_set instead.

buildgrid.server.app.settings.parser.load_dynamic_property_set(unique_property_keys: Iterable[str] | None = None, match_property_keys: Iterable[str] | None = None, wildcard_property_keys: Iterable[str] | None = None, label_key: str | None = None) → DynamicPropertySet

A dynamic property set allows scheduling jobs which may have unset values for properties. Dynamic queues can be flexible as they allow minimal configuration to add new properties, however, they have an exponential cost to scheduling. Using many different properties can lead to very slow scheduling rates.

Parameters:

• unique_property_keys (set[str]) – Properties which may only be set once. OSFamily is always considered unique.

• match_property_keys (set[str]) – Properties which must match on the worker and execute request. OSFamily and ISA property keys are always added as match keys even if unlisted.

• wildcard_property_keys (set[str]) – Properties which are available to workers, but not used for scheduling.

• label_key (str) – A key used to identify job types in logging and metrics. Defaults to OSFamily

buildgrid.server.app.settings.parser.load_static_property_set(property_labels: list[buildgrid.server.app.settings.parser._PropertyLabel], wildcard_property_keys: list[str] | None = None) → StaticPropertySet

A static property set allows scheduling jobs by resolving sane defaults for unspecified keys. Static queues can be more verbose as you require defining all sets of valid properties, however, they have a linear cost to scheduling. Using many different properties becomes less expensive to calculate assignment.

Parameters:

• property_labels (list[_PropertyLabel]) – Properties combinations which are allowed.

• wildcard_property_keys (list[str]) – Properties which are available to workers, but not used for scheduling.

buildgrid.server.app.settings.parser.load_sql_index(storage: StorageABC, sql: SqlProvider, window_size: int = 1000, inclause_limit: int = -1, fallback_on_get: bool = False, max_inline_blob_size: int = 0, refresh_accesstime_older_than: int = 0) → SQLIndex

Generates buildgrid.server.cas.storage.index.sql.SQLIndex using the tag !sql-index.

Usage

- !sql-index
  # This assumes that a storage instance is defined elsewhere
  # with a `&cas-storage` anchor
  storage: *cas-storage
  sql: *sql
  window-size: 1000
  inclause-limit: -1
  fallback-on-get: no
  max-inline-blob-size: 256
  refresh-accesstime-older-than: 0

Parameters:

• storage (buildgrid.server.cas.storage.storage_abc.StorageABC) – Instance of storage to use. This must be a storage object constructed using a YAML tag ending in -storage, for example !disk-storage.

• window_size (uint) – Maximum number of blobs to fetch in one SQL operation (larger resultsets will be automatically split into multiple queries)

• inclause_limit (int) – If nonnegative, overrides the default number of variables permitted per “in” clause. See the buildgrid.server.cas.storage.index.sql.SQLIndex comments for more details.

• fallback_on_get (bool) – By default, the SQL Index only fetches blobs from the underlying storage if they’re present in the index on get_blob/bulk_read_blobs requests to minimize interactions with the storage. If this is set, the index instead checks the underlying storage directly on get_blob/bulk_read_blobs requests, then loads all blobs found into the index.

• max_inline_blob_size (int) – Blobs of this size or smaller are stored directly in the index and not in the backing storage (must be nonnegative).

• refresh-accesstime-older-than (int) – When reading a blob, its access timestamp will not be updated if the current time is not at least refresh-accesstime-older-than seconds newer than the access timestamp. Set this to reduce load associated with frequent timestamp updates.

buildgrid.server.app.settings.parser.load_execution_controller(scheduler: Scheduler, operation_stream_keepalive_timeout: int = 600, endpoints: Sequence[str] = ('execution', 'operations', 'bots'), max_list_operations_page_size: int = 1000) → ExecutionController

Generates buildgrid.server.execution.service.ExecutionService using the tag !execution.

Usage

# This assumes that the YAML anchors are defined elsewhere
- !execution
  scheduler: *state-database
  operation-stream-keepalive-timeout: 600
  endpoints:
    - execution
    - operations
    - bots
  max-list-operations-page-size: 1000

Parameters:

• scheduler (Scheduler) – Instance of scheduler to use for the scheduler’s state.

• operation_stream_keepalive_timeout (int) – The longest time (in seconds) we’ll wait before sending the current status in an Operation response stream of an Execute or WaitExecution request. Defaults to 600s (10 minutes).

• endpoints (list) – List of service/endpoint types to enable. Possible services are execution, operations, and bots. By default all three are enabled.

• max_list_operations_page_size (int) – The maximum number of operations that can be returned in a ListOperations response. A page token will be returned with the response to allow the client to get the next page of results.

buildgrid.server.app.settings.parser.load_bots_controller(scheduler: Scheduler) → BotsInterface

Generates buildgrid.server.bots.instance.BotsInterface using the tag !bots.

Usage

# This assumes that the YAML anchors are defined elsewhere
- !bots
  scheduler: *state-database

Parameters:

scheduler (Scheduler) – Instance of scheduler to use for the scheduler’s state.

buildgrid.server.app.settings.parser.load_action_cache_controller(cache: ActionCacheABC) → ActionCache

Generates buildgrid.server.actioncache.service.ActionCacheService using the tag !action-cache.

Usage

# This assumes that the YAML anchors are defined elsewhere
- !action-cache
  cache: *lru-cache

Parameters:

cache (ActionCacheABC) – The ActionCache backend to use for this cache.

buildgrid.server.app.settings.parser.load_mirrored_action_cache(first: ActionCacheABC, second: ActionCacheABC) → MirroredCache

Generates:class:buildgrid.server.actioncache.caches.mirrored_cache.MirroredCache using the tag !mirrored-action-cache.

Usage

# This assumes that the YAML anchors are defined elsewhere
- !mirrored-action-cache
  first: *first-action-cache
  second: *second-action-cache

buildgrid.server.app.settings.parser.load_with_cache_action_cache(cache: ActionCacheABC, fallback: ActionCacheABC, allow_updates: bool = True, cache_failed_actions: bool = True) → WithCacheActionCache

Generates:class:buildgrid.server.actioncache.caches.with_cache.WithCacheActionCache using the tag !with-cache-action-cache.

Usage

# This assumes that the YAML anchors are defined elsewhere
- !with-cache-action-cache
  storage: *cas-storage
  cache: *cache-ac
  fallback: *fallback-ac

Parameters:

• cache (ActionCacheABC) – ActionCache instance to use as a local cache

• fallback (ActionCacheABC) – ActionCache instance to use as a fallback on local cache misses

• allow_updates (bool) – Allow updates pushed to the Action Cache. Defaults to True.

• cache_failed_actions (bool) – Whether to store failed (non-zero exit code) actions. Default to True.

buildgrid.server.app.settings.parser.load_lru_action_cache(storage: StorageABC, max_cached_refs: int, allow_updates: bool = True, cache_failed_actions: bool = True) → LruActionCache

Generates buildgrid.server.actioncache.caches.lru_cache.LruActionCache using the tag !lru-action-cache.

Usage

# This assumes that the YAML anchors are defined elsewhere
- !lru-action-cache
  storage: *cas-storage
  max-cached-refs: 1024
  cache-failed-actions: yes
  allow-updates: yes

Parameters:

• storage (buildgrid.server.cas.storage.storage_abc.StorageABC) – Instance of storage to use.

• max_cached_refs (int) – Max number of cached actions.

• allow_updates (bool) – Allow updates pushed to the Action Cache. Defaults to True.

• cache_failed_actions (bool) – Whether to store failed (non-zero exit code) actions. Default to True.

buildgrid.server.app.settings.parser.load_s3_action_cache(storage: StorageABC, allow_updates: bool = True, cache_failed_actions: bool = True, entry_type: str | None = None, migrate_entries: bool | None = False, bucket: str | None = None, endpoint: str | None = None, access_key: str | None = None, secret_key: str | None = None, cache_key_salt: str | None = None) → S3ActionCache

Generates buildgrid.server.actioncache.caches.s3_cache.S3ActionCache using the tag !s3action-cache.

Usage

# This assumes that the YAML anchors are defined elsewhere
- !s3action-cache
  storage: *cas-storage
  allow-updates: yes
  cache-failed-actions: yes
  entry-type: action-result-digest
  migrate-entries: no
  bucket: bgd-action-cache
  endpoint: http://localhost:9000/
  access-key: !read-file /var/bgd/s3-access-key
  secret-key: !read-file /var/bgd/s3-secret-key

Parameters:

• storage (buildgrid.server.cas.storage.storage_abc.StorageABC) – Instance of storage to use. This must be an object constructed using a YAML tag ending in -storage, for example !disk-storage.

• allow_updates (bool) – Allow updates pushed to the Action Cache. Defaults to True.

• cache_failed_actions (bool) – Whether to store failed (non-zero exit code) actions. Default to True.

• entry_type (str) – whether entries in S3 will store an 'action-result' or an 'action-result-digest' (default).

• migrate_entries (bool) – Whether to automatically update the values of entries that contain a different type of value to entry_type as they are queried. Default to False.

• bucket (str) – Name of bucket

• endpoint (str) – URL of endpoint.

• access-key (str) – S3-ACCESS-KEY

• secret-key (str) – S3-SECRET-KEY

• cache_key_salt (str) – Optional salt to use in S3 keys. Instances with the same salt will share cache contents.

buildgrid.server.app.settings.parser.load_remote_action_cache(url: str, instance_name: str | None = None, retries: int = 3, max_backoff: int = 64, request_timeout: float | None = None, credentials: ClientCredentials | None = None, channel_options: dict[str, Any] | None = None) → RemoteActionCache

Generates buildgrid.server.actioncache.caches.remote.RemoteActionCache using the tag !remote-action-cache.

Usage

- !remote-action-cache
  url: https://action-cache:50053
  instance-name: main
  credentials:
    tls-server-key: !expand-path ~/.config/buildgrid/server.key
    tls-server-cert: !expand-path ~/.config/buildgrid/server.cert
    tls-client-certs: !expand-path ~/.config/buildgrid/client.cert
    auth-token: /path/to/auth/token
    token-refresh-seconds: 6000
  channel-options:
    lb-policy-name: round_robin

Parameters:

• url (str) – URL to remote action cache

• instance_name (str | None) – Instance of the remote to connect to. Defaults to the instance context if none.

• credentials (dict, optional) –

  A dictionary in the form:

  tls-client-key: /path/to/client-key
  tls-client-cert: /path/to/client-cert
  tls-server-cert: /path/to/server-cert
  auth-token: /path/to/auth/token
  token-refresh-seconds (int): seconds to wait before reading the token from the file again
  

• channel-options (dict, optional) –

  A dictionary of grpc channel options in the form:

  some-channel-option: channel_value
  other-channel-option: another-channel-value
  

• https (See) – //github.com/grpc/grpc/blob/master/include/grpc/impl/codegen/grpc_types.h

• options (for the valid channel) –

buildgrid.server.app.settings.parser.load_write_once_action_cache(action_cache: ActionCacheABC) → WriteOnceActionCache

Generates buildgrid.server.actioncache.caches.write_once_cache.WriteOnceActionCache using the tag !write-once-action-cache.

This allows a single update for a given key, essentially making it possible to create immutable ActionCache entries, rather than making the cache read-only as the allow-updates property of other ActionCache implementations does.

Usage

# This assumes that the YAML anchors are defined elsewhere
- !write-once-action-cache
  action-cache: *remote-cache

Parameters:

action_cache (ActionCache) – The action cache instance to make immutable.

buildgrid.server.app.settings.parser.load_redis_action_cache(storage: StorageABC, redis: Any, allow_updates: bool = True, cache_failed_actions: bool = True, entry_type: str | None = None, migrate_entries: bool | None = False, cache_key_salt: str | None = None) → RedisActionCache

Generates buildgrid.server.actioncache.caches.redis_cache.RedisActionCache using the tag !redis-action-cache.

This creates an Action Cache which stores the mapping from Action digests to ActionResults in Redis.

Usage

# This assumes that the YAML anchors are defined elsewhere
- !redis-action-cache
  storage: *cas-storage
  allow-updates: yes
  cache-failed-actions: yes
  entry-type: action-result-digest
  migrate-entries: no
  redis: *redis-connection

Parameters:

• storage (buildgrid.server.cas.storage.storage_abc.StorageABC) – Instance of storage to use. This must be an object constructed using a YAML tag ending in -storage, for example !disk-storage.

• allow_updates (bool) – Allow updates pushed to the Action Cache. Defaults to True.

• cache_failed_actions (bool) – Whether to store failed (non-zero exit code) actions. Default to True.

• entry_type (str) – whether entries in Redis will store an 'action-result' or an 'action-result-digest' (default).

• migrate_entries (bool) – Whether to automatically update the values of entries that contain a different type of value to entry_type as they are queried. Default to False.

• redis (buildgrid.server.redis.provider.RedisProvider) – A configured Redis connection manager. This must be an object with an !redis-connection YAML tag.

• cache_key_salt (str) – Optional salt to use in Redis keys. Instances with the same salt will share cache contents.

class buildgrid.server.app.settings.parser.ActionCacheShardType

Bases: TypedDict

name: str

cache: ActionCacheABC

buildgrid.server.app.settings.parser.load_sharded_action_cache(shards: list[buildgrid.server.app.settings.parser.ActionCacheShardType], allow_updates: bool = True, cache_failed_actions: bool = True, cache_key_salt: str | None = None) → ShardedActionCache

Generates:class:buildgrid.server.actioncache.caches.sharded_cache.ShardedActionCache using the tag !sharded-action-cache.

This creates an Action Cache whose contents are split among multiple child caches.

Usage

# This assumes that the YAML anchors are defined elsewhere
- !sharded-action-cache
  shards:
    - name: a
      cache:
        !lru-action-cache
        storage: *cas-storage
        max-cached-refs: 1024
        cache-failed-actions: yes
        allow-updates: yes
    - name: b
      cache:
        !lru-action-cache
        storage: *cas-storage
        max-cached-refs: 1024
        cache-failed-actions: yes
        allow-updates: yes
  allow-updates: yes
  cache-failed-actions: no

Parameters:

• shards (list[ActionCacheShardType]) – List of shards. Shards are defined as dictionaries with a name and cache. Shard names must be unique per Sharded Action Cache.

• allow_updates (bool) – Allow updates to be pushed to the Action Cache. Both this and the shards’ allow_updates settings must be True to allow updates. Defaults to True.

• cache_failed_actions (bool) – Whether to store failed (non-zero exit code) actions. Both this and the shards’ cache_failed_actions settings must be True to enable caching of failed actions. Default to True.

• cache_key_salt (str) – Optional salt to use in shard key calculation. Instances with the same salt will map the same digest to the same shard name.

buildgrid.server.app.settings.parser.load_cas_controller(storage: StorageABC, read_only: bool = False, tree_cache_size: int | None = None, tree_cache_ttl_minutes: float = 60) → ContentAddressableStorageInstance

Generates buildgrid.server.cas.service.ContentAddressableStorageService using the tag !cas.

Usage

# This assumes that the YAML anchors are defined elsewhere
- !cas
  storage: *cas-storage

Parameters:

• storage (buildgrid.server.cas.storage.storage_abc.StorageABC) – Instance of storage to use. This must be an object constructed using a YAML tag ending in -storage, for example !disk-storage.

• tree_cache_size (int | None) – Size of GetTreeResponse cache, default to None. This feature is experimental for testing purposes. It could be deprecated in favor of a redis cache in future.

• tree_cache_ttl_minutes (float) – TTL of GetTreeResponse cache, default to 60 minutes. This feature is experimental for testing purposes. It could be deprecated in favor of a redis cache in future.

buildgrid.server.app.settings.parser.load_bytestream_controller(storage: StorageABC, read_only: bool = False, disable_overwrite_early_return: bool = False) → ByteStreamInstance

Generates buildgrid.server.cas.service.ByteStreamService using the tag !bytestream.

Usage

# This assumes that the YAML anchors are defined elsewhere
- !bytestream
  storage: *cas-storage

Parameters:

storage (buildgrid.server.cas.storage.storage_abc.StorageABC) – Instance of storage to use. This must be an object constructed using a YAML tag ending in -storage, for example !disk-storage.

buildgrid.server.app.settings.parser.load_memory_build_events_controller() → BuildEventStreamStorage

Generates buildgrid.server.build_events.storage.BuildEventStreamStorage using the tag !memory-build-events-storage.

Usage

- !memory-build-events

buildgrid.server.app.settings.parser.load_metering_service_client(base_url: str, token_path: str | None = None, retry_max_attempts: int = 0, retry_exp_base: float = 1.5, retry_multiplier: float = 1.0, retry_max_wait: float = 10.0, retry_http_statuses: list[int] | None = None, retry_exceptions: list[str] | None = None, retry_cause_exceptions: list[str] | None = None) → SyncMeteringServiceClient

Generates buildgrid_metering.client.SyncMeteringServiceClient using the tag !metering-service-client.

Usage

- !metering-service-client
  token-path: /tmp/path/to/token
  retry-max-attempts: 3
  retry-exp-base: 2
  retry-multiplier: 1
  retry-http-statuses: [503]
  retry-exceptions: ["metering-service-client-error"]

buildgrid.server.app.settings.parser.load_asset_client(url: str, credentials: ClientCredentials | None = None, instance_name: str = '', request_timeout: float = 5.0, retries: int = 3) → AssetClient

Generates buildgrid_metering.client.AssetClient using the tag !asset-client.

Usage

- !asset-client
  url: https://remote-asset.com
  instance-name: dev
  credentials:
    tls-client-cert: /path/to/cert
    auth-token: /path/to/token
  request-timeout: 5
  retries: 3

buildgrid.server.app.settings.parser.load_introspection_instance(scheduler: Scheduler) → IntrospectionInstance

buildgrid.server.app.settings.parser.load_sentry(dsn: str = '', sample_rate: float = 0, proxy: str = '') → Sentry

Generates buildgrid.server.sentry.Sentry using the tag !sentry.

Usage

- !sentry
  dsn: https://public@sentry.example.com/1
  sample-rate: 0.5
  proxy: https://proxy.example.com:80

buildgrid.server.app.settings.parser.load_limiter(concurrent_request_limit: int = 0) → Limiter

buildgrid.server.app.settings.parser.get_logstream_connection_info(logstream: Any) → tuple[str | None, dict[str, str] | None, str | None]

buildgrid.server.app.settings.parser.get_schema(strict: bool = False) → Any

Gets a schema for the buildgrid configuration. If in strict mode, all object definitions will set additionalProperties to false

buildgrid.server.app.settings.parser.validate_config(path: PathLike[str], strict: bool = False, fail_deprecations: bool = False) → None

Validate a buildgrid configuration against its schema. In this mode, no real components are loaded, it simply detects invalid argument values. If in strict mode, all object definitions will set additionalProperties to false

buildgrid.server.app.settings.parser.validate_config_value(data: str, strict: bool = False, fail_deprecations: bool = False) → None

Validate a buildgrid configuration against its schema. In this mode, no real components are loaded, it simply detects invalid argument values. If in strict mode, all object definitions will set additionalProperties to false

buildgrid.server.app.settings.parser.load_config(path: PathLike[str]) → Any

Load and validate a buildgrid configuration.

buildgrid.server.app.settings.parser.load_config_value(data: str) → Any

Load and validate a buildgrid configuration.