buildgrid.server.persistence.sql.impl module

buildgrid.server.persistence.sql.impl.sqlite_on_connect(conn, record)
class buildgrid.server.persistence.sql.impl.SQLDataStore(storage, *, connection_string=None, automigrate=False, connection_timeout=5, poll_interval=1, **kwargs)

Bases: buildgrid.server.persistence.interface.DataStoreInterface

activate_monitoring()

Enable the monitoring features of the data store.

deactivate_monitoring()

Disable the monitoring features of the data store.

This method also performs any necessary cleanup of stored metrics.

session(*, sqlite_lock_immediately=False, reraise=False)
get_job_by_action(action_digest)

Return the job corresponding to an Action digest.

This method looks for a job object corresponding to the given Action digest in the data store. If a job is found it is returned, otherwise None is returned.

Parameters

action_digest (Digest) – The digest of the Action to find the corresponding job for.

Returns

The job with the given Action digest, if it exists. Otherwise None.

Return type

buildgrid.server.job.Job or None

get_job_by_name(name)

Return the job with the given name.

This method looks for a job with the specified name in the data store. If there is a matching Job it is returned, otherwise this returns None.

Parameters

name (str) – The name of the job to return.

Returns

The job with the given name, if it exists. Otherwise None.

Return type

buildgrid.server.job.Job or None

get_job_by_operation(operation_name)

Return the Job for a given Operation.

This method takes an Operation name, and returns the Job which corresponds to that Operation. If the Operation isn’t found, or if the data store doesn’t contain a corresponding job, this returns None.

Parameters

operation (str) – Name of the Operation whose corresponding Job is to be returned.

Returns

The job related to the given operation, if it exists. Otherwise None.

Return type

buildgrid.server.job.Job or None

get_all_jobs()

Return a list of all jobs in the data store.

This method returns a list of all incomplete jobs in the data store.

Returns

List of all incomplete jobs in the data store.

Return type

list

get_jobs_by_stage(operation_stage)

Return a list of jobs in the given stage.

This method returns a list of all jobs in a specific operation stage.

Parameters

operation_stage (OperationStage) – The stage that the returned list of jobs should all be in.

Returns

List of all jobs in the specified operation stage.

Return type

list

create_job(job)

Add a new job to the data store.

NOTE: This method just stores the job in the data store. In order to enqueue the job to make it available for scheduling execution, the queue_job method should also be called.

Parameters

job (buildgrid.server.job.Job) – The job to be stored.

queue_job(job_name)

Add an existing job to the queue of jobs waiting to be assigned.

This method adds a job with the given name to the queue of jobs. If the job is already in the queue, then this method ensures that the position of the job in the queue is correct.

update_job(job_name, changes)

Update a job in the data store.

This method takes a job name and a dictionary of changes to apply to the job in the data store, and updates the job with those changes. The dictionary should be keyed by the attribute names which need to be updated, with the values being the new values for the attributes.

Parameters
  • job_name (str) – The name of the job that is being updated.

  • changes – (dict): The dictionary of changes

delete_job(job_name)

Delete a job from the data store.

This method removes a job from the data store.

Parameters

job_name (str) – The name of the job to be removed.

wait_for_job_updates()
store_response(job)

Store the job’s ExecuteResponse in the data store.

This method stores the response message for the job in the data store, in order to allow it to be retrieved when getting jobs in the future.

This is separate from update_job as implementations will likely need to always have a special case for handling persistence of the response message.

Parameters

job (buildgrid.server.job.Job) – The job to store the response message of.

get_operations_by_stage(operation_stage)

Return a set of Job names in a specific operation stage.

Find the operations in a given stage and return a set containing the names of the Jobs related to those operations.

Parameters

operation_stage (OperationStage) – The stage that the operations should be in.

Returns

Set of all job names with operations in the specified state.

Return type

set

get_all_operations()

Return a list of all operation names in the data store.

Returns

List of all operation names in the data store.

Return type

list

create_operation(operation, job_name)

Add a new operation to the data store.

Parameters
  • operation (Operation) – The Operation protobuf object representing the operation to add to the data store.

  • job_name (str) – The name of the Job representing the execution of this operation.

update_operation(operation_name, changes)

Update an operation in the data store.

This method takes an operation name and a dictionary of changes to apply to the operation in the data store, and updates the operation with those changes. The dictionary should be keyed by the attribute names which need to be updated, with the values being the new values for the attributes.

Parameters
  • operation_name (str) – The name of the operation that is being updated.

  • changes – (dict): The dictionary of changes to be applied.

delete_operation(operation_name)

Delete a operation from the data store.

This method removes a operation from the data store.

Parameters

operation_name (str) – The name of the operation to be removed.

get_leases_by_state(lease_state)

Return the set of IDs of leases in a given state.

Parameters

lease_state (LeaseState) – The state that the leases should be in.

Returns

Set of strings containing IDs of leases in the given state.

Return type

set

get_metrics()

Return a dictionary of metrics for jobs, operations, and leases.

The returned dictionary is keyed by buildgrid._enums.MetricCategories values, and the values are dictionaries of counts per operation stage (or lease state, in the case of leases).

create_lease(lease)

Add a new lease to the data store.

Parameters

lease (Lease) – The Lease protobuf object representing the lease to be added to the data store.

update_lease(job_name, changes)

Update a lease in the data store.

This method takes a job name and a dictionary of changes to apply to the lease for that job in the data store, and updates the lease with those changes. The dictionary should be keyed by the attribute names which need to be updated, with the values being the new values for the attributes.

The job name is used as leases have no unique identifier; instead there should always be at most one active lease for the job. It is the responsibility of data store implementations to ensure this.

Parameters
  • job_name (str) – The name of the job whose lease is being updated.

  • changes – (dict): The dictionary of changes to be applied.

load_unfinished_jobs()
assign_lease_for_next_job(capabilities, callback, timeout=None)

Return a list of leases for the highest priority jobs that can be run by a worker.

NOTE: Currently the list only ever has one or zero leases.

Query the jobs table to find queued jobs which match the capabilities of a given worker, and return the one with the highest priority. Takes a dictionary of worker capabilities to compare with job requirements.

Parameters
  • capabilities (dict) – Dictionary of worker capabilities to compare with job requirements when finding a job.

  • callback (function) – Function to run on the next runnable job, should return a list of leases.

  • timeout (int) – time to wait for new jobs, caps if longer than MAX_JOB_BLOCK_TIME.

Returns

List of leases

flatten_capabilities(capabilities: Dict[str, List[str]]) → List[Tuple[str, str]]

Flatten a capabilities dictionary, assuming all of its values are lists. E.g.

{‘OSFamily’: [‘Linux’], ‘ISA’: [‘x86-32’, ‘x86-64’]}

becomes

[(‘OSFamily’, ‘Linux’), (‘ISA’, ‘x86-32’), (‘ISA’, ‘x86-64’)]

get_partial_capabilities(capabilities: Dict[str, List[str]]) → Iterable[Dict[str, List[str]]]

Given a capabilities dictionary with all values as lists, yield all partial capabilities dictionaries.

get_partial_capabilities_hashes(capabilities: Dict) → List[str]

Given a list of configurations, obtain each partial configuration for each configuration, obtain the hash of each partial configuration, compile these into a list, and return the result.