buildgrid.server.cas.storage.index.sql module

SQLIndex

A SQL index implementation. This can be pointed to either a remote SQL server or a local SQLite database.

class buildgrid.server.cas.storage.index.sql.SQLIndex(storage: buildgrid.server.cas.storage.storage_abc.StorageABC, connection_string: str, automigrate: bool = False, window_size: int = 1000, inclause_limit: int = - 1, connection_timeout: int = 5, **kwargs)

Bases: buildgrid.server.cas.storage.index.index_abc.IndexABC

session(*, sqlite_lock_immediately: bool = False, reraise: bool = False, pass_exceptions: Optional[List[Type[sqlalchemy.exc.SQLAlchemyError]]] = None) → Any

Context manager for convenience use of sessions. Automatically commits when the context ends and rolls back failed transactions.

Setting sqlite_lock_immediately will only yield a session once the SQLite database has been locked for exclusive use.

Setting reraise to True causes this to reraise any exceptions after rollback so they can be handled directly by client logic.

has_blob(digest: build.bazel.remote.execution.v2.remote_execution_pb2.Digest) → bool

Return True if the blob with the given instance/digest exists.

get_blob(digest: build.bazel.remote.execution.v2.remote_execution_pb2.Digest) → Optional[BinaryIO]

Return a file-like object containing the blob. Most implementations will read the entire file into memory and return a BytesIO object. Eventually this should be corrected to handle files which cannot fit into memory.

If the blob isn’t present in storage, return None.

delete_blob(digest: build.bazel.remote.execution.v2.remote_execution_pb2.Digest) → None

Delete a blob from the index. Return True if the blob was deleted, or False otherwise.

TODO: This method will be promoted to StorageABC in a future commit.

begin_write(digest: build.bazel.remote.execution.v2.remote_execution_pb2.Digest) → BinaryIO

Return a file-like object to which a blob’s contents could be written.

commit_write(digest: build.bazel.remote.execution.v2.remote_execution_pb2.Digest, write_session: BinaryIO) → None

Commit the write operation. write_session must be an object returned by begin_write.

The storage object is not responsible for verifying that the data written to the write_session actually matches the digest. The caller must do that.

missing_blobs(digests: List[build.bazel.remote.execution.v2.remote_execution_pb2.Digest]) → List[build.bazel.remote.execution.v2.remote_execution_pb2.Digest]

Return a container containing the blobs not present in CAS.

bulk_update_blobs(blobs: List[Tuple[build.bazel.remote.execution.v2.remote_execution_pb2.Digest, bytes]]) → List[google.rpc.status_pb2.Status]

Given a container of (digest, value) tuples, add all the blobs to CAS. Return a list of Status objects corresponding to the result of uploading each of the blobs.

Unlike in commit_write, the storage object will verify that each of the digests matches the provided data.

bulk_read_blobs(digests: List[build.bazel.remote.execution.v2.remote_execution_pb2.Digest]) → Dict[str, BinaryIO]

Given an iterable container of digests, return a {hash: file-like object} dictionary corresponding to the blobs represented by the input digests.

least_recent_digests() → Iterator[build.bazel.remote.execution.v2.remote_execution_pb2.Digest]

Generator to iterate through the digests in LRU order

get_total_size() → int

Return the sum of the size of all blobs within the index

mark_n_bytes_as_deleted(n_bytes: int, dry_run: bool = False) → List[build.bazel.remote.execution.v2.remote_execution_pb2.Digest]

Mark a given number of bytes’ worth of index entries as deleted.

The entries are marked as deleted in LRU order until the total size marked as deleted is at least the requested number of bytes. If any entries are already marked as deleted, they are used first when attempting to reach the required number of bytes.

Parameters

n_bytes (int) – The number of bytes of index entries to mark as deleted. When the sum of the size_bytes of index entries meets or exceeds this value, the affected digests will be returned.

Returns

The list of digests that were marked as deleted.

Return type

list

bulk_delete(digests: List[build.bazel.remote.execution.v2.remote_execution_pb2.Digest]) → None

Delete a list of blobs from storage.