buildgrid.utils module

class buildgrid.utils.BrowserURL(base_url, instance_name=None)

Bases: object

for_message(message_type, message_digest)

Completes browser URL initialization for a protobuf message.


Generates a browser URL string.

class buildgrid.utils.Condition(lock=None)

Bases: threading.Condition

Subclass of threading.Condition with wait overridden.

In this implementation, wait only releases the lock if other threads are actually waiting for the lock, otherwise it does nothing.


Wait if other threads are trying to acquire the lock.

If other threads have attempted to acquire the lock for this Condition using with, this method behaves the same as wait on a regular threading.Condition.

If only one thread has attempted to acquire the lock, then that thread must be the current thread. In that case, this method doesn’t release the lock or wait at all, it simply returns True as if it had been woken by threading.Condition.notify.

class buildgrid.utils.TypedEvent

Bases: object

Wrapper around a threading.Event to support event ‘types’

class buildgrid.utils.JobState(job)

Bases: object

class buildgrid.utils.JobWatchSpec(job)

Bases: object

Structure to track what operations are being watched for a given job.

This also contains the event used for notifying watchers of changes, and the state that the job was in after a change was last detected.

property peers

Returns a copy of the list of peers for the given operation.

If the operation is not being watched, or for some reason has no “peers” key, the empty list is returned.


operation_name (string) – The name of the operation to get the list of peers for.

add_peer(operation_name, peer)

Add a peer to the set of peers watching the job this spec is for.

Takes an operation name and a peer and tracks that the peer is watching the given operation.

  • operation_name (string) – The name of the operation that the peer is watching for updates on.

  • peer (string) – The peer that is starting to watch for updates.

remove_peer(operation_name, peer)

Remove a peer from the list watching an operation for this job.

The inverse of add_peer. Takes an operation name and a peer and removes that peer from the list of peers watching that operation. If this leaves the operation with no peers watching it, the operation is removed from the JobWatchSpec.

  • operation_name (string) – The name of the operation that is no longer being watched by the peer.

  • peer (string) – The name of the peer that is stopping watching.

buildgrid.utils.acquire_lock_or_timeout(lock, timeout=60)

Returns the hostname of the machine executing that function.


Hostname for the current machine.

Return type



Returns the hash type.


Computes the Digest of a piece of data.

The Digest of a data is a function of its hash and size.


bytes_to_digest (bytes) – byte data to digest.


The Digest for the given byte data.

Return type



Creates a Digest from a digest string.

A digest string should alway be: {hash}/{size_bytes}.


digest_string (str) – the digest string.


The Digest read from the string or None if

digest_string is not a valid digest string.

Return type



Loads raw file content in memory.


file_path (str) – path to the target file.


Raw file’s content until EOF.

Return type



OSError – If file_path does not exist or is not readable.

buildgrid.utils.write_file(file_path, content)

Dumps raw memory content to a file.

  • file_path (str) – path to the target file.

  • content (bytes) – raw file’s content.


OSError – If file_path does not exist or is not writable.


Walks a local folder tree, generating FileNode and DirectoryNode.


directory_path (str) – absolute or relative path to a local directory.


Message, bytes, str – a tutple of either a FileNode or DirectoryNode message, the corresponding blob and the corresponding node path.

buildgrid.utils.output_file_maker(file_path, input_path, file_digest)

Creates an OutputFile from a local file and possibly upload it.


file_path must point inside or be relative to input_path.

  • file_path (str) – absolute or relative path to a local file.

  • input_path (str) – absolute or relative path to the input root directory.

  • file_digest (Digest) – the underlying file’s digest.


a new OutputFile object for the file pointed by file_path.

Return type


buildgrid.utils.output_directory_maker(directory_path, working_path, tree_digest)

Creates an OutputDirectory from a local directory.


directory_path must point inside or be relative to input_path.

  • directory_path (str) – absolute or relative path to a local directory.

  • working_path (str) – absolute or relative path to the working directory.

  • tree_digest (Digest) – the underlying folder tree’s digest.


a new OutputDirectory for the directory pointed by directory_path.

Return type


buildgrid.utils.convert_values_to_sorted_lists(dictionary: Dict[str, Union[str, Sequence[str]]])None

Given a dictionary, do the following:

  1. Turn strings into singleton lists

  2. Turn all other sequence types into sorted lists with list()

This is a mutating operation. No value is returned.

buildgrid.utils.hash_from_dict(dictionary: Dict[str, List[str]])str

Get the hash represntation of a dictionary

buildgrid.utils.get_unique_objects_by_attribute(objects: Sequence[T], attribute: str)Iterable[T]

Return a list of unique objects based on a hashable attribute or chained attributes.

Note that this does not provide any sanitization, and any problematic elements will only raise exceptions when iterated on.

buildgrid.utils.retry_delay(retry_attempt: int, delay_base: int = 1)float
buildgrid.utils.flatten_capabilities(capabilities: Mapping[str, Set[str]])List[Tuple[str, str]]

Flatten a capabilities dictionary.

This method takes a capabilities dictionary and flattens it into a list of key/value tuples describing all the platform properties that the capabilities map to. To do this, it assumes that all of the dictionary’s values are iterable.

For example,

{'OSFamily': {'Linux'}, 'ISA': {'x86-32', 'x86-64'}}


[('OSFamily', 'Linux'), ('ISA', 'x86-32'), ('ISA', 'x86-64')]


capabilities (dict) – The capabilities dictionary to flatten.


list containing the flattened dictionary key-value tuples.

buildgrid.utils.combinations_with_unique_keys(iterator: Sequence[Tuple[str, str]], size: int)Iterable[Iterable[Tuple]]

Return an iterator of the unique combinations of the input without duplicated keys.

The input iterator is a sequence of key-value tuples. This function behaves similarly to itertools.combinations(), except combinations containing more than one tuple with the same first element are not included in the result.

The size argument specifies how many elements should be included in the resulting combinations.

For example,

>>> capabilities = [('OSFamily', 'linux'), ('ISA', 'x86-64'), ('ISA', 'x86-32')]
>>> platforms = combinations_with_unique_keys(capabilities, 2)
>>> for item in platforms:
...     print(item)
(('OSFamily', 'linux'), ('ISA', 'x86-64'))
(('OSFamily', 'linux'), ('ISA', 'x86-32'))
  • iterator (list) – The list of key-value tuples to return combinations of.

  • size (int) – How many elements to include in each combination.


An iterator of the combinations of the input in which each key appears at most once.