buildgrid.server.cas.storage.index.sql module¶

SQLIndex¶

A SQL index implementation. This can be pointed to either a remote SQL server or a local SQLite database.

class buildgrid.server.cas.storage.index.sql.SQLIndex(storage: buildgrid.server.cas.storage.storage_abc.StorageABC, connection_string: str, automigrate: bool = False, window_size: int = 1000, inclause_limit: int = - 1, connection_timeout: int = 5, **kwargs)¶

Bases: buildgrid.server.cas.storage.index.index_abc.IndexABC

session(*, sqlite_lock_immediately: bool = False, exceptions_to_not_raise_on: Optional[List[Type[sqlalchemy.exc.SQLAlchemyError]]] = None, exceptions_to_not_rollback_on: Optional[List[Type[sqlalchemy.exc.SQLAlchemyError]]] = None) → Any¶

Context manager for convenience use of sessions. Automatically commits when the context ends and rolls back failed transactions.

Setting sqlite_lock_immediately will only yield a session once the SQLite database has been locked for exclusive use.

exceptions_to_not_raise_on is a list of exceptions which the session will not re-raise on. However, the session will still rollback on these errors.

exceptions_to_not_rollback_on is a list of exceptions which the session manager will not re-raise or rollback on. Being very careful specifying this option. It should only be used in cases where you know the exception will not put the database in a bad state.

has_blob(digest: build.bazel.remote.execution.v2.remote_execution_pb2.Digest) → bool¶: Return True if the blob with the given instance/digest exists.

get_blob(digest: build.bazel.remote.execution.v2.remote_execution_pb2.Digest) → Optional[BinaryIO]¶

Return a file-like object containing the blob. Most implementations will read the entire file into memory and return a BytesIO object. Eventually this should be corrected to handle files which cannot fit into memory.

If the blob isn’t present in storage, return None.

delete_blob(digest: build.bazel.remote.execution.v2.remote_execution_pb2.Digest) → None¶

Delete a blob from the index. Return True if the blob was deleted, or False otherwise.

TODO: This method will be promoted to StorageABC in a future commit.

begin_write(digest: build.bazel.remote.execution.v2.remote_execution_pb2.Digest) → BinaryIO¶: Return a file-like object to which a blob’s contents could be written.

commit_write(digest: build.bazel.remote.execution.v2.remote_execution_pb2.Digest, write_session: BinaryIO) → None¶

Commit the write operation. write_session must be an object returned by begin_write.

The storage object is not responsible for verifying that the data written to the write_session actually matches the digest. The caller must do that.

missing_blobs(digests: List[build.bazel.remote.execution.v2.remote_execution_pb2.Digest]) → List[build.bazel.remote.execution.v2.remote_execution_pb2.Digest]¶: Return a container containing the blobs not present in CAS.

bulk_update_blobs(blobs: List[Tuple[build.bazel.remote.execution.v2.remote_execution_pb2.Digest, bytes]]) → List[google.rpc.status_pb2.Status]¶

Given a container of (digest, value) tuples, add all the blobs to CAS. Return a list of Status objects corresponding to the result of uploading each of the blobs.

Unlike in commit_write, the storage object will verify that each of the digests matches the provided data.

bulk_read_blobs(digests: List[build.bazel.remote.execution.v2.remote_execution_pb2.Digest]) → Dict[str, BinaryIO]¶: Given an iterable container of digests, return a {hash: file-like object} dictionary corresponding to the blobs represented by the input digests.

least_recent_digests() → Iterator[build.bazel.remote.execution.v2.remote_execution_pb2.Digest]¶: Generator to iterate through the digests in LRU order

get_total_size() → int¶: Return the sum of the size of all blobs within the index

mark_n_bytes_as_deleted(n_bytes: int, dry_run: bool = False, protect_blobs_after: Optional[datetime.datetime] = None) → List[build.bazel.remote.execution.v2.remote_execution_pb2.Digest]¶

Mark a given number of bytes’ worth of index entries as deleted.

The entries are marked as deleted in LRU order until the total size marked as deleted is at least the requested number of bytes. If any entries are already marked as deleted, they are used first when attempting to reach the required number of bytes.

Parameters: n_bytes (int) – The number of bytes of index entries to mark as deleted. When the sum of the size_bytes of index entries meets or exceeds this value, the affected digests will be returned.
Returns: The list of digests that were marked as deleted.
Return type: list

bulk_delete(digests: List[build.bazel.remote.execution.v2.remote_execution_pb2.Digest]) → List[buildgrid.utils.Failure]¶: Delete a list of blobs from storage.