buildgrid.utils module¶

class buildgrid.utils.BrowserURL(base_url, instance_name=None)¶

Bases: object

for_message(message_type, message_digest)¶: Completes browser URL initialization for a protobuf message.

generate()¶: Generates a browser URL string.

class buildgrid.utils.TypedEvent¶

Bases: object

Wrapper around a threading.Event to support event ‘types’

set(event_type=None)¶

clear()¶

notify_change()¶

notify_stop()¶

wait(last_received=None, timeout=None)¶

class buildgrid.utils.JobState(job)¶: Bases: object

class buildgrid.utils.JobWatchSpec(job)¶

Bases: object

Structure to track what operations are being watched for a given job.

This also contains the event used for notifying watchers of changes, and the state that the job was in after a change was last detected.

property peers¶

peers_for_operation(operation_name)¶

Returns a copy of the list of peers for the given operation.

If the operation is not being watched, or for some reason has no “peers” key, the empty list is returned.

Parameters: operation_name (string) – The name of the operation to get the list of peers for.

add_peer(operation_name, peer)¶

Add a peer to the set of peers watching the job this spec is for.

Takes an operation name and a peer and tracks that the peer is watching the given operation.

Parameters

operation_name (string) – The name of the operation that the peer is watching for updates on.
peer (string) – The peer that is starting to watch for updates.

remove_peer(operation_name, peer)¶

Remove a peer from the list watching an operation for this job.

The inverse of add_peer. Takes an operation name and a peer and removes that peer from the list of peers watching that operation. If this leaves the operation with no peers watching it, the operation is removed from the JobWatchSpec.

Parameters

operation_name (string) – The name of the operation that is no longer being watched by the peer.
peer (string) – The name of the peer that is stopping watching.

buildgrid.utils.acquire_lock_or_timeout(lock, timeout=60)¶

buildgrid.utils.get_hostname()¶

Returns the hostname of the machine executing that function.

Returns: Hostname for the current machine.
Return type: str

buildgrid.utils.get_hash_type()¶: Returns the hash type.

buildgrid.utils.create_digest(bytes_to_digest)¶

Computes the Digest of a piece of data.

The Digest of a data is a function of its hash and size.

Parameters: bytes_to_digest (bytes) – byte data to digest.
Returns: The Digest for the given byte data.
Return type: Digest

buildgrid.utils.create_digest_from_file(file_obj: BinaryIO) → build.bazel.remote.execution.v2.remote_execution_pb2.Digest¶

Computed the Digest of a file-like object.

The Digest contains a hash of the file’s contents and the size of those contents. This function only reads the content in chunks for hashing, so is safe to use on large files.

Parameters: file_obj (BinaryIO) – A file-like object of some kind.
Returns: The Digest for the given file object.
Return type: Digest

buildgrid.utils.parse_digest(digest_string)¶

Creates a Digest from a digest string.

A digest string should alway be: {hash}/{size_bytes}.

Parameters

digest_string (str) – the digest string.

Returns

The Digest read from the string or None if: digest_string is not a valid digest string.

Return type

Digest

buildgrid.utils.validate_digest_data(digest: build.bazel.remote.execution.v2.remote_execution_pb2.Digest, data: bytes)¶: Validate that the given digest corresponds to the given data.

buildgrid.utils.read_file(file_path)¶

Loads raw file content in memory.

Parameters: file_path (str) – path to the target file.
Returns: Raw file’s content until EOF.
Return type: bytes
Raises: OSError – If file_path does not exist or is not readable.

buildgrid.utils.write_file(file_path, content)¶

Dumps raw memory content to a file.

Parameters

file_path (str) – path to the target file.
content (bytes) – raw file’s content.

Raises

OSError – If file_path does not exist or is not writable.

buildgrid.utils.read_and_rewind(read_head: IO) → Optional[AnyStr]¶

Reads from an IO object and returns the data found there after rewinding the object to the beginning.

Parameters: read_head (IO) – readable IO head
Returns: readable content from read_head.
Return type: AnyStr

buildgrid.utils.merkle_tree_maker(directory_path)¶

Walks a local folder tree, generating FileNode and DirectoryNode.

Parameters: directory_path (str) – absolute or relative path to a local directory.
Yields: Message, bytes, str – a tutple of either a FileNode or DirectoryNode message, the corresponding blob and the corresponding node path.

buildgrid.utils.output_file_maker(file_path, input_path, file_digest)¶

Creates an OutputFile from a local file and possibly upload it.

Note

file_path must point inside or be relative to input_path.

Parameters

file_path (str) – absolute or relative path to a local file.
input_path (str) – absolute or relative path to the input root directory.
file_digest (Digest) – the underlying file’s digest.

Returns

a new OutputFile object for the file pointed by file_path.

Return type

OutputFile

buildgrid.utils.output_directory_maker(directory_path, working_path, tree_digest)¶

Creates an OutputDirectory from a local directory.

Note

directory_path must point inside or be relative to input_path.

Parameters

directory_path (str) – absolute or relative path to a local directory.
working_path (str) – absolute or relative path to the working directory.
tree_digest (Digest) – the underlying folder tree’s digest.

Returns

a new OutputDirectory for the directory pointed by directory_path.

Return type

OutputDirectory

buildgrid.utils.convert_values_to_sorted_lists(dictionary: Mapping[str, Union[str, Sequence[str], Set[str]]]) → Dict[str, List[str]]¶

Given a dictionary, do the following:

Turn strings into singleton lists
Turn all other sequence types into sorted lists with list()

This returns the converted dictionary and does not change the dictionary that was passed in.

buildgrid.utils.hash_from_dict(dictionary: Mapping[str, List[str]]) → str¶: Get the hash represntation of a dictionary

buildgrid.utils.get_unique_objects_by_attribute(objects: Sequence[T], attribute: str) → Iterable[T]¶

Return a list of unique objects based on a hashable attribute or chained attributes.

Note that this does not provide any sanitization, and any problematic elements will only raise exceptions when iterated on.

buildgrid.utils.retry_delay(retry_attempt: int, delay_base: int = 1) → float¶

buildgrid.utils.flatten_capabilities(capabilities: Mapping[str, Union[Set[str], List[str]]]) → List[Tuple[str, str]]¶

Flatten a capabilities dictionary.

This method takes a capabilities dictionary and flattens it into a list of key/value tuples describing all the platform properties that the capabilities map to. To do this, it assumes that all of the dictionary’s values are iterable.

For example,

{'OSFamily': {'Linux'}, 'ISA': {'x86-32', 'x86-64'}}

becomes

[('OSFamily', 'Linux'), ('ISA', 'x86-32'), ('ISA', 'x86-64')]

Parameters: capabilities (dict) – The capabilities dictionary to flatten.
Returns: list containing the flattened dictionary key-value tuples.

buildgrid.utils.combinations_with_unique_keys(iterator: Sequence[Tuple[str, str]], size: int) → Iterable[Iterable[Tuple]]¶

Return an iterator of the unique combinations of the input without duplicated keys.

The input iterator is a sequence of key-value tuples. This function behaves similarly to itertools.combinations(), except combinations containing more than one tuple with the same first element are not included in the result.

The size argument specifies how many elements should be included in the resulting combinations.

For example,

>>> capabilities = [('OSFamily', 'linux'), ('ISA', 'x86-64'), ('ISA', 'x86-32')]
>>> platforms = combinations_with_unique_keys(capabilities, 2)
>>> for item in platforms:
...     print(item)
...
(('OSFamily', 'linux'), ('ISA', 'x86-64'))
(('OSFamily', 'linux'), ('ISA', 'x86-32'))

Parameters

iterator (list) – The list of key-value tuples to return combinations of.
size (int) – How many elements to include in each combination.

Returns

An iterator of the combinations of the input in which each key appears at most once.