API reference

BuildGrid’s Application Programming Interface (API) reference documentation.

Configuration Parser API

The tagged YAML nodes in the reference configuration are handled by the YAML parser using the following set of objects:

class buildgrid._app.settings.parser.YamlFactory

Bases: yaml.YAMLObject

Base class for contructing maps or scalars from tags.

yaml_tag: Optional[str] = None
schema: Optional[str] = None
classmethod from_yaml(loader, node)

Convert a representation node to a Python object.

class buildgrid._app.settings.parser.Channel(port, insecure_mode, credentials=None)

Bases: buildgrid._app.settings.parser.YamlFactory

Creates a GRPC channel.

The Channel class returns a grpc.Channel and is generated from the tag !channel. Creates either a secure or insecure channel.

Usage
- !channel
  port: 50051
  insecure-mode: false
  credentials:
    tls-server-key: !expand-path ~/.config/buildgrid/server.key
    tls-server-cert: !expand-path ~/.config/buildgrid/server.cert
    tls-client-certs: !expand-path ~/.config/buildgrid/client.cert
Parameters
  • port (int) – A port for the channel.

  • insecure_mode (bool) – If True, generates an insecure channel, even if there are credentials. Defaults to True.

  • credentials (dict, optional) –

    A dictionary in the form:

    tls-server-key: /path/to/server-key
    tls-server-cert: /path/to/server-cert
    tls-client-certs: /path/to/client-certs
    

yaml_tag: Optional[str] = '!channel'
schema: Optional[str] = 'misc/channel.yaml'
class buildgrid._app.settings.parser.ExpandPath(path)

Bases: buildgrid._app.settings.parser.YamlFactory

Returns a string of the user’s path after expansion.

The ExpandPath class returns a string and is generated from the tag !expand-path.

Usage
path: !expand-path ~/bgd-data/cas
Parameters

path (str) – Can be used with strings such as: ~/dir/to/something or $HOME/certs

yaml_tag: Optional[str] = '!expand-path'
class buildgrid._app.settings.parser.ReadFile(path)

Bases: buildgrid._app.settings.parser.YamlFactory

Returns a string of the contents of the specified file.

The ReadFile class returns a string and is generated from the tag !read-file.

Usage
secret_key: !read-file /var/bgd/s3-secret-key
Parameters

path (str) – Can be used with strings such as: ~/path/to/some/file or $HOME/myfile or /path/to/file

yaml_tag: Optional[str] = '!read-file'
class buildgrid._app.settings.parser.Disk(path: str)

Bases: buildgrid._app.settings.parser.YamlFactory

Generates buildgrid.server.cas.storage.disk.DiskStorage using the tag !disk-storage.

Usage
- !disk-storage
  path: /opt/bgd/cas-storage
Parameters

path (str) – Path to directory to storage.

yaml_tag: Optional[str] = '!disk-storage'
schema: Optional[str] = 'storage/disk.yaml'
class buildgrid._app.settings.parser.LRU(size: str)

Bases: buildgrid._app.settings.parser.YamlFactory

Generates buildgrid.server.cas.storage.lru_memory_cache.LRUMemoryCache using the tag !lru-storage.

Usage
- !lru-storage
  size: 2048M
Parameters

size (int) – Size e.g 10kb. Size parsed with buildgrid._app.settings.parser._parse_size().

yaml_tag: Optional[str] = '!lru-storage'
schema: Optional[str] = 'storage/lru.yaml'
class buildgrid._app.settings.parser.S3(bucket: str, endpoint: str, access_key: str, secret_key: str)

Bases: buildgrid._app.settings.parser.YamlFactory

Generates buildgrid.server.cas.storage.s3.S3Storage using the tag !s3-storage.

Usage
- !s3-storage
  bucket: bgd-bucket-{digest[0]}{digest[1]}
  endpoint: http://127.0.0.1:9000
  access_key: !read-file /var/bgd/s3-access-key
  secret_key: !read-file /var/bgd/s3-secret-key
Parameters
  • bucket (str) – Name of bucket

  • endpoint (str) – URL of endpoint.

  • access-key (str) – S3-ACCESS-KEY

  • secret-key (str) – S3-SECRET-KEY

yaml_tag: Optional[str] = '!s3-storage'
schema: Optional[str] = 'storage/s3.yaml'
class buildgrid._app.settings.parser.Redis(host: str, port: int, password: Optional[str] = None, db: Optional[int] = None)

Bases: buildgrid._app.settings.parser.YamlFactory

Generates buildgrid.server.cas.storage.redis.RedisStorage using the tag !redis-storage.

Usage
- !redis-storage
  host: 127.0.0.1
  port: 6379
  password: !read-file /var/bgd/redis-pass
  db: 0
Parameters
  • host (str) – hostname of endpoint.

  • port (int) – port on host.

  • password (str) – redis database password

  • db (int) – db number

yaml_tag: Optional[str] = '!redis-storage'
schema: Optional[str] = 'storage/redis.yaml'
class buildgrid._app.settings.parser.Remote(url: str, instance_name: str, credentials: Optional[Dict[str, str]] = None)

Bases: buildgrid._app.settings.parser.YamlFactory

Generates buildgrid.server.cas.storage.remote.RemoteStorage using the tag !remote-storage.

Usage
- !remote-storage
  url: https://storage:50052/
  instance-name: main
  credentials:
    tls-server-key: !expand-path ~/.config/buildgrid/server.key
    tls-server-cert: !expand-path ~/.config/buildgrid/server.cert
    tls-client-certs: !expand-path ~/.config/buildgrid/client.cert
Parameters
  • url (str) – URL to remote storage. If used with https, needs credentials.

  • instance_name (str) – Instance of the remote to connect to.

  • credentials (dict, optional) –

    A dictionary in the form:

    tls-client-key: /path/to/client-key
    tls-client-cert: /path/to/client-cert
    tls-server-cert: /path/to/server-cert
    

yaml_tag: Optional[str] = '!remote-storage'
schema: Optional[str] = 'storage/remote.yaml'
class buildgrid._app.settings.parser.WithCache(cache: buildgrid.server.cas.storage.storage_abc.StorageABC, fallback: buildgrid.server.cas.storage.storage_abc.StorageABC)

Bases: buildgrid._app.settings.parser.YamlFactory

Generates buildgrid.server.cas.storage.with_cache.WithCacheStorage using the tag !with-cache-storage.

Usage
- !with-cache-storage
  cache:
    !lru-storage
    size: 2048M
  fallback:
    !disk-storage
    path: /opt/bgd/cas-storage
Parameters
  • cache (StorageABC) – Storage instance to use as a cache

  • fallback (StorageABC) – Storage instance to use as a fallback on cache misses

yaml_tag: Optional[str] = '!with-cache-storage'
schema: Optional[str] = 'storage/with-cache.yaml'
class buildgrid._app.settings.parser.SQLSchedulerConfig(storage: buildgrid.server.cas.storage.storage_abc.StorageABC, connection_string: Optional[str] = None, automigrate: bool = False, connection_timeout: int = 5, **kwargs)

Bases: buildgrid._app.settings.parser.YamlFactory

Generates buildgrid.server.persistence.sql.impl.SQLDataStore using the tag !sql-scheduler.

Usage
- !sql-scheduler
  # This assumes that a storage instance is defined elsewhere
  # with a `&cas-storage` anchor
  storage: *cas-storage
  connection_string: postgresql://bgd:insecure@database/bgd
  automigrate: yes
  connection_timeout: 5
Parameters
  • storage (buildgrid.server.cas.storage.storage_abc.StorageABC) – Instance of storage to use for getting actions and storing job results. This must be an object constructed using a YAML tag ending in -storage, for example !disk-storage.

  • connection_string (str) – SQLAlchemy connection string to use for connecting to the database.

  • automigrate (bool) – Whether to attempt to automatically upgrade an existing DB schema to the newest version (this will also create everything from scratch if given an empty database).

  • connection_timeout (int) – Time to wait for an SQLAlchemy connection to be available in the pool before timing out.

yaml_tag: Optional[str] = '!sql-scheduler'
schema: Optional[str] = 'scheduler/sql.yaml'
class buildgrid._app.settings.parser.SQLDataStoreConfig(storage: buildgrid.server.cas.storage.storage_abc.StorageABC, connection_string: Optional[str] = None, automigrate: bool = False, connection_timeout: int = 5, **kwargs)

Bases: buildgrid._app.settings.parser.YamlFactory

Generates buildgrid.server.persistence.sql.impl.SQLDataStore using the tag !sql-data-store.

Warning

This is deprecated and only used for compatibility with old configs.

Usage
- !sql-data-store
  # This assumes that a storage instance is defined elsewhere
  # with a `&cas-storage` anchor
  storage: *cas-storage
  connection_string: postgresql://bgd:insecure@database/bgd
  automigrate: yes
  connection_timeout: 5
Parameters
  • storage (buildgrid.server.cas.storage.storage_abc.StorageABC) – Instance of storage to use for getting actions and storing job results. This must be an object constructed using a YAML tag ending in -storage, for example !disk-storage.

  • connection_string (str) – SQLAlchemy connection string to use for connecting to the database.

  • automigrate (bool) – Whether to attempt to automatically upgrade an existing DB schema to the newest version (this will also create everything from scratch if given an empty database).

  • connection_timeout (int) – Time to wait for an SQLAlchemy connection to be available in the pool before timing out.

yaml_tag: Optional[str] = '!sql-data-store'
schema: Optional[str] = 'scheduler/sql.yaml'
class buildgrid._app.settings.parser.MemorySchedulerConfig(storage: buildgrid.server.cas.storage.storage_abc.StorageABC)

Bases: buildgrid._app.settings.parser.YamlFactory

Generates buildgrid.server.persistence.mem.impl.MemoryDataStore using the tag !memory-scheduler.

Usage
- !memory-scheduler
  # This assumes that a storage instance is defined elsewhere
  # with a `&cas-storage` anchor
  storage: *cas-storage
Parameters

storage (buildgrid.server.cas.storage.storage_abc.StorageABC) – Instance of storage to use for getting actions and storing job results. This must be an object constructed using a YAML tag ending in -storage, for example !disk-storage.

yaml_tag: Optional[str] = '!memory-scheduler'
schema: Optional[str] = 'scheduler/memory.yaml'
class buildgrid._app.settings.parser.MemoryDataStoreConfig(storage: buildgrid.server.cas.storage.storage_abc.StorageABC)

Bases: buildgrid._app.settings.parser.YamlFactory

Generates buildgrid.server.persistence.mem.impl.MemoryDataStore using the tag !memory-data-store.

Warning

This is deprecated and only used for compatibility with old configs. Use MemorySchedulerConfig instead.

Usage
- !memory-data-store
  # This assumes that a storage instance is defined elsewhere
  # with a `&cas-storage` anchor
  storage: *cas-storage
Parameters

storage (buildgrid.server.cas.storage.storage_abc.StorageABC) – Instance of storage to use for getting actions and storing job results. This must be an object constructed using a YAML tag ending in -storage, for example !disk-storage.

yaml_tag: Optional[str] = '!memory-data-store'
schema: Optional[str] = 'scheduler/memory.yaml'
class buildgrid._app.settings.parser.SQL_Index(storage: buildgrid.server.cas.storage.storage_abc.StorageABC, connection_string: str, automigrate: bool = False, window_size: int = 1000, inclause_limit: int = - 1, fallback_on_get: bool = False, **kwargs)

Bases: buildgrid._app.settings.parser.YamlFactory

Generates buildgrid.server.cas.storage.index.sql.SQLIndex using the tag !sql-index.

Usage
- !sql-index
  # This assumes that a storage instance is defined elsewhere
  # with a `&cas-storage` anchor
  storage: *cas-storage
  connection_string: postgresql://bgd:insecure@database/bgd
  automigrate: yes
  window-size: 1000
  inclause-limit: -1
  fallback-on-get: no
Parameters
  • storage (buildgrid.server.cas.storage.storage_abc.StorageABC) – Instance of storage to use. This must be a storage object constructed using a YAML tag ending in -storage, for example !disk-storage.

  • connection_string (str) – SQLAlchemy connection string

  • automigrate (bool) – Attempt to automatically upgrade an existing DB schema to the newest version.

  • window_size (uint) – Maximum number of blobs to fetch in one SQL operation (larger resultsets will be automatically split into multiple queries)

  • inclause_limit (int) – If nonnegative, overrides the default number of variables permitted per “in” clause. See the buildgrid.server.cas.storage.index.sql.SQLIndex comments for more details.

  • fallback_on_get (bool) – By default, the SQL Index only fetches blobs from the underlying storage if they’re present in the index on get_blob/bulk_read_blobs requests to minimize interactions with the storage. If this is set, the index instead checks the underlying storage directly on get_blob/bulk_read_blobs requests, then loads all blobs found into the index.

yaml_tag: Optional[str] = '!sql-index'
schema: Optional[str] = 'storage/sql-index.yaml'
class buildgrid._app.settings.parser.Execution(storage, action_cache=None, action_browser_url=None, data_store=None, scheduler=None, property_keys=None, bot_session_keepalive_timeout=600, permissive_bot_session=False, endpoints='execution', 'operations', 'bots', discard_unwatched_jobs=False, max_execution_timeout=7200)

Bases: buildgrid._app.settings.parser.YamlFactory

Generates buildgrid.server.execution.service.ExecutionService using the tag !execution.

Usage
# This assumes that the YAML anchors are defined elsewhere
- !execution
  storage: *cas-storage
  action-cache: *remote-cache
  action-browser-url: http://localhost:8080
  scheduler: *state-database
  property-keys:
    - chrootDigest
  bot-session-keepalive-timeout: 600
  endpoints:
    - execution
    - operations
    - bots
  discard-unwatched-jobs: no
  max-execution-timeout: 7200
Parameters
  • storage (buildgrid.server.cas.storage.storage_abc.StorageABC) – Instance of storage to use. This must be an object constructed using a YAML tag ending in -storage, for example !disk-storage.

  • action_cache (ActionCache) – Instance of action cache to use.

  • action_browser_url (str) – The base URL to use to generate Action Browser links to users

  • scheduler (DataStoreInterface) – Instance of data store to use for the scheduler’s state.

  • property_keys (list) – The allowed property keys for jobs

  • bot_session_keepalive_timeout (int) – The longest time (in seconds) we’ll wait for a bot to send an update before it assumes it’s dead. Defaults to 600s (10 minutes).

  • permissive_bot_session – Whether to accept UpdateBotSession requests from bots that haven’t explicitly called CreateBotSession with this particular server instance.

  • endpoints (list) – List of service/endpoint types to enable. Possible services are execution, operations, and bots. By default all three are enabled.

  • discard_unwatched_jobs (bool) – Allow operation id to persist without a client connection.

  • max_execution_timeout (int) – The maximum time jobs are allowed to be in ‘OperationStage.EXECUTING’. This is a lazy check tha happens before job deduplication and when new peers register for an ongoing Operation. When this time is exceeded in executing stage, the job will be cancelled.

yaml_tag: Optional[str] = '!execution'
schema: Optional[str] = 'services/execution.yaml'
class buildgrid._app.settings.parser.Bots(storage, action_cache=None, bot_session_keepalive_timeout=600, data_store=None, scheduler=None, permissive_bot_session=False)

Bases: buildgrid._app.settings.parser.YamlFactory

Generates buildgrid.server.bots.instance.BotsInterface using the tag !bots.

Usage
# This assumes that the YAML anchors are defined elsewhere
- !bots
  storage: *cas-storage
  action-cache: *remote-cache
  scheduler: *state-database
  bot-session-keepalive-timeout: 600
  permissive-bot-session: yes
Parameters
  • storage (buildgrid.server.cas.storage.storage_abc.StorageABC) – Instance of storage to use. This must be an object constructed using a YAML tag ending in -storage, for example !disk-storage.

  • action_cache (Action) – Instance of action cache to use.

  • scheduler (DataStoreInterface) – Instance of data store to use for the scheduler’s state.

  • bot_session_keepalive_timeout (int) – The longest time (in seconds) we’ll wait for a bot to send an update before it assumes it’s dead. Defaults to 600s (10 minutes).

  • permissive_bot_session – Whether to accept UpdateBotSession requests from bots that haven’t explicitly called CreateBotSession with this particular server instance.

yaml_tag: Optional[str] = '!bots'
schema: Optional[str] = 'services/bots.yaml'
class buildgrid._app.settings.parser.Action(storage: buildgrid.server.cas.storage.storage_abc.StorageABC, max_cached_refs: int, allow_updates: bool = True, cache_failed_actions: bool = True)

Bases: buildgrid._app.settings.parser.YamlFactory

Generates buildgrid.server.actioncache.service.ActionCacheService using the tag !action-cache.

Usage
# This assumes that the YAML anchors are defined elsewhere
- !action-cache
  storage: *cas-storage
  max-cached-refs: 1024
  cache-failed-actions: yes
  allow-updates: yes
Parameters
  • storage (buildgrid.server.cas.storage.storage_abc.StorageABC) – Instance of storage to use.

  • max_cached_refs (int) – Max number of cached actions.

  • allow_updates (bool) – Allow updates pushed to CAS. Defaults to True.

  • cache_failed_actions (bool) – Whether to store failed (non-zero exit code) actions. Default to True.

yaml_tag: Optional[str] = '!action-cache'
schema: Optional[str] = 'services/action-cache.yaml'
class buildgrid._app.settings.parser.S3Action(storage: buildgrid.server.cas.storage.storage_abc.StorageABC, allow_updates: bool = True, cache_failed_actions: bool = True, bucket: str = None, endpoint: Optional[str] = None, access_key: Optional[str] = None, secret_key: Optional[str] = None)

Bases: buildgrid._app.settings.parser.YamlFactory

Generates buildgrid.server.actioncache.service.S3ActionCache using the tag !s3-action-cache.

Usage
# This assumes that the YAML anchors are defined elsewhere
- !s3action-cache
  storage: *cas-storage
  allow-updates: yes
  cache-failed-actions: yes
  bucket: bgd-action-cache
  endpoint: http://localhost:9000/
  access-key: !read-file /var/bgd/s3-access-key
  secret-key: !read-file /var/bgd/s3-secret-key
Parameters
  • storage (buildgrid.server.cas.storage.storage_abc.StorageABC) – Instance of storage to use. This must be an object constructed using a YAML tag ending in -storage, for example !disk-storage.

  • allow_updates (bool) – Allow updates pushed to CAS. Defaults to True.

  • cache_failed_actions (bool) – Whether to store failed (non-zero exit code) actions. Default to True.

  • bucket (str) – Name of bucket

  • endpoint (str) – URL of endpoint.

  • access-key (str) – S3-ACCESS-KEY

  • secret-key (str) – S3-SECRET-KEY

yaml_tag: Optional[str] = '!s3action-cache'
schema: Optional[str] = 'services/s3-action-cache.yaml'
class buildgrid._app.settings.parser.RemoteAction(url: str, instance_name: str, credentials: Optional[Dict[str, str]] = None)

Bases: buildgrid._app.settings.parser.YamlFactory

Generates buildgrid.server.actioncache.remote.RemoteActionCache using the tag !remote-action-cache.

Usage
- !remote-action-cache
  url: https://action-cache:50053
  instance-name: main
  credentials:
    tls-server-key: !expand-path ~/.config/buildgrid/server.key
    tls-server-cert: !expand-path ~/.config/buildgrid/server.cert
    tls-client-certs: !expand-path ~/.config/buildgrid/client.cert
Parameters
  • url (str) – URL to remote action cache. If used with https, needs credentials.

  • instance_name (str) – Instance of the remote to connect to.

  • credentials (dict, optional) –

    A dictionary in the form:

    tls-client-key: /path/to/client-key
    tls-client-cert: /path/to/client-cert
    tls-server-cert: /path/to/server-cert
    

yaml_tag: Optional[str] = '!remote-action-cache'
schema: Optional[str] = 'services/remote-action-cache.yaml'
class buildgrid._app.settings.parser.WriteOnceAction(action_cache)

Bases: buildgrid._app.settings.parser.YamlFactory

Generates buildgrid.server.actioncache.remote.WriteOnceActionCache using the tag !write-once-action-cache.

This allows a single update for a given key, essentially making it possible to create immutable ActionCache entries, rather than making the cache read-only as the allow-updates property of other ActionCache implementations does.

Usage
# This assumes that the YAML anchors are defined elsewhere
- !write-once-action-cache
  action-cache: *remote-cache
Parameters

action_cache (ActionCache) – The action cache instance to make immutable.

yaml_tag: Optional[str] = '!write-once-action-cache'
schema: Optional[str] = 'services/write-once-action-cache.yaml'
class buildgrid._app.settings.parser.Reference(storage: buildgrid.server.cas.storage.storage_abc.StorageABC, max_cached_refs: int, allow_updates: bool = True)

Bases: buildgrid._app.settings.parser.YamlFactory

Generates buildgrid.server.referencestorage.service.ReferenceStorageService using the tag !reference-cache.

Note

This is intended for use by old versions of BuildStream as an artifact cache, and isn’t part of the REAPI/RWAPI specifications. There’s no need for this component if you aren’t using BuildStream.

Usage
# This assumes that the YAML anchors are defined elsewhere
- !reference-storage
  storage: *cas-storage
  max-cached-refs: 1024
  allow-updates: yes
Parameters
  • storage (buildgrid.server.cas.storage.storage_abc.StorageABC) – Instance of storage to use. This must be an object constructed using a YAML tag ending in -storage, for example !disk-storage.

  • max_cached_refs (int) – Max number of cached actions.

  • allow_updates (bool) – Allow updates pushed to CAS. Defaults to True.

yaml_tag: Optional[str] = '!reference-cache'
schema: Optional[str] = 'services/reference-cache.yaml'
class buildgrid._app.settings.parser.CAS(storage: buildgrid.server.cas.storage.storage_abc.StorageABC, read_only: bool = False)

Bases: buildgrid._app.settings.parser.YamlFactory

Generates buildgrid.server.cas.service.ContentAddressableStorageService using the tag !cas.

Usage
# This assumes that the YAML anchors are defined elsewhere
- !cas
  storage: *cas-storage
Parameters

storage (buildgrid.server.cas.storage.storage_abc.StorageABC) – Instance of storage to use. This must be an object constructed using a YAML tag ending in -storage, for example !disk-storage.

yaml_tag: Optional[str] = '!cas'
schema: Optional[str] = 'services/cas.yaml'
class buildgrid._app.settings.parser.ByteStream(storage: buildgrid.server.cas.storage.storage_abc.StorageABC, read_only: bool = False)

Bases: buildgrid._app.settings.parser.YamlFactory

Generates buildgrid.server.cas.service.ByteStreamService using the tag !bytestream.

Usage
# This assumes that the YAML anchors are defined elsewhere
- !bytestream
  storage: *cas-storage
Parameters

storage (buildgrid.server.cas.storage.storage_abc.StorageABC) – Instance of storage to use. This must be an object constructed using a YAML tag ending in -storage, for example !disk-storage.

yaml_tag: Optional[str] = '!bytestream'
schema: Optional[str] = 'services/bytestream.yaml'
class buildgrid._app.settings.parser.LogStream(prefix='')

Bases: buildgrid._app.settings.parser.YamlFactory

Generates buildgrid.server.logstream.instance.LogStreamInstance using the tag !logstream.

Usage
- !logstream
  prefix: test
yaml_tag: Optional[str] = '!logstream'
schema: Optional[str] = 'services/logstream.yaml'
buildgrid._app.settings.parser.get_parser()
buildgrid._app.settings.parser.get_schema()
buildgrid._app.settings.parser.get_validator(schema=None)