buildgrid.utils module

class buildgrid.utils.BrowserURL(base_url, instance_name=None)

Bases: object

for_message(message_type, message_digest)

Completes browser URL initialization for a protobuf message.

generate()

Generates a browser URL string.

class buildgrid.utils.TypedEvent

Bases: object

Wrapper around a threading.Event to support event ‘types’

set(event_type=None)
clear()
notify_change()
notify_stop()
wait(last_received=None)
class buildgrid.utils.JobState(job)

Bases: object

class buildgrid.utils.JobWatchSpec(job)

Bases: object

Structure to track what operations are being watched for a given job.

This also contains the event used for notifying watchers of changes, and the state that the job was in after a change was last detected.

property peers
peers_for_operation(operation_name)

Returns a copy of the list of peers for the given operation.

If the operation is not being watched, or for some reason has no “peers” key, the empty list is returned.

Parameters

operation_name (string) – The name of the operation to get the list of peers for.

add_peer(operation_name, peer)

Add a peer to the set of peers watching the job this spec is for.

Takes an operation name and a peer and tracks that the peer is watching the given operation.

Parameters
  • operation_name (string) – The name of the operation that the peer is watching for updates on.

  • peer (string) – The peer that is starting to watch for updates.

remove_peer(operation_name, peer)

Remove a peer from the list watching an operation for this job.

The inverse of add_peer. Takes an operation name and a peer and removes that peer from the list of peers watching that operation. If this leaves the operation with no peers watching it, the operation is removed from the JobWatchSpec.

Parameters
  • operation_name (string) – The name of the operation that is no longer being watched by the peer.

  • peer (string) – The name of the peer that is stopping watching.

buildgrid.utils.acquire_lock_or_timeout(lock, timeout=60)
buildgrid.utils.get_hostname()

Returns the hostname of the machine executing that function.

Returns

Hostname for the current machine.

Return type

str

buildgrid.utils.get_hash_type()

Returns the hash type.

buildgrid.utils.create_digest(bytes_to_digest)

Computes the Digest of a piece of data.

The Digest of a data is a function of its hash and size.

Parameters

bytes_to_digest (bytes) – byte data to digest.

Returns

The Digest for the given byte data.

Return type

Digest

buildgrid.utils.parse_digest(digest_string)

Creates a Digest from a digest string.

A digest string should alway be: {hash}/{size_bytes}.

Parameters

digest_string (str) – the digest string.

Returns

The Digest read from the string or None if

digest_string is not a valid digest string.

Return type

Digest

buildgrid.utils.validate_digest_data(digest: build.bazel.remote.execution.v2.remote_execution_pb2.Digest, data: bytes)

Validate that the given digest corresponds to the given data.

buildgrid.utils.read_file(file_path)

Loads raw file content in memory.

Parameters

file_path (str) – path to the target file.

Returns

Raw file’s content until EOF.

Return type

bytes

Raises

OSError – If file_path does not exist or is not readable.

buildgrid.utils.write_file(file_path, content)

Dumps raw memory content to a file.

Parameters
  • file_path (str) – path to the target file.

  • content (bytes) – raw file’s content.

Raises

OSError – If file_path does not exist or is not writable.

buildgrid.utils.read_and_rewind(read_head: IO)Optional[AnyStr]

Reads from an IO object and returns the data found there after rewinding the object to the beginning.

Parameters

read_head (IO) – readable IO head

Returns

readable content from read_head.

Return type

AnyStr

buildgrid.utils.merkle_tree_maker(directory_path)

Walks a local folder tree, generating FileNode and DirectoryNode.

Parameters

directory_path (str) – absolute or relative path to a local directory.

Yields

Message, bytes, str – a tutple of either a FileNode or DirectoryNode message, the corresponding blob and the corresponding node path.

buildgrid.utils.output_file_maker(file_path, input_path, file_digest)

Creates an OutputFile from a local file and possibly upload it.

Note

file_path must point inside or be relative to input_path.

Parameters
  • file_path (str) – absolute or relative path to a local file.

  • input_path (str) – absolute or relative path to the input root directory.

  • file_digest (Digest) – the underlying file’s digest.

Returns

a new OutputFile object for the file pointed by file_path.

Return type

OutputFile

buildgrid.utils.output_directory_maker(directory_path, working_path, tree_digest)

Creates an OutputDirectory from a local directory.

Note

directory_path must point inside or be relative to input_path.

Parameters
  • directory_path (str) – absolute or relative path to a local directory.

  • working_path (str) – absolute or relative path to the working directory.

  • tree_digest (Digest) – the underlying folder tree’s digest.

Returns

a new OutputDirectory for the directory pointed by directory_path.

Return type

OutputDirectory

buildgrid.utils.convert_values_to_sorted_lists(dictionary: Dict[str, Union[str, Sequence[str]]])None

Given a dictionary, do the following:

  1. Turn strings into singleton lists

  2. Turn all other sequence types into sorted lists with list()

This is a mutating operation. No value is returned.

buildgrid.utils.hash_from_dict(dictionary: Dict[str, List[str]])str

Get the hash represntation of a dictionary

buildgrid.utils.get_unique_objects_by_attribute(objects: Sequence[T], attribute: str)Iterable[T]

Return a list of unique objects based on a hashable attribute or chained attributes.

Note that this does not provide any sanitization, and any problematic elements will only raise exceptions when iterated on.

buildgrid.utils.retry_delay(retry_attempt: int, delay_base: int = 1)float
buildgrid.utils.flatten_capabilities(capabilities: Mapping[str, Set[str]])List[Tuple[str, str]]

Flatten a capabilities dictionary.

This method takes a capabilities dictionary and flattens it into a list of key/value tuples describing all the platform properties that the capabilities map to. To do this, it assumes that all of the dictionary’s values are iterable.

For example,

{'OSFamily': {'Linux'}, 'ISA': {'x86-32', 'x86-64'}}

becomes

[('OSFamily', 'Linux'), ('ISA', 'x86-32'), ('ISA', 'x86-64')]

Parameters

capabilities (dict) – The capabilities dictionary to flatten.

Returns

list containing the flattened dictionary key-value tuples.

buildgrid.utils.combinations_with_unique_keys(iterator: Sequence[Tuple[str, str]], size: int)Iterable[Iterable[Tuple]]

Return an iterator of the unique combinations of the input without duplicated keys.

The input iterator is a sequence of key-value tuples. This function behaves similarly to itertools.combinations(), except combinations containing more than one tuple with the same first element are not included in the result.

The size argument specifies how many elements should be included in the resulting combinations.

For example,

>>> capabilities = [('OSFamily', 'linux'), ('ISA', 'x86-64'), ('ISA', 'x86-32')]
>>> platforms = combinations_with_unique_keys(capabilities, 2)
>>> for item in platforms:
...     print(item)
...
(('OSFamily', 'linux'), ('ISA', 'x86-64'))
(('OSFamily', 'linux'), ('ISA', 'x86-32'))
Parameters
  • iterator (list) – The list of key-value tuples to return combinations of.

  • size (int) – How many elements to include in each combination.

Returns

An iterator of the combinations of the input in which each key appears at most once.