Using the LogStream service

BuildGrid’s LogStream service can be used both for streaming stdout/stderr logs from Actions in a BuildGrid execution service, and as a more generic standalone LogStream service implementation.

Streaming build logs

If deployed as part of a larger BuildGrid remote execution service, the LogStream service will be used by BuildBox workers to stream a combined stdout/stderr log. The resource names are contained in the metadata field of the Operation messages sent in response to Execute or WaitExecution requests.

The resource names can then be used by a ByteStream client to make a Read request for the resource to the LogStream service. Once an Action has finished executing, there is no guarantee that the stream will exist, and using the stdout/stderr inlined in the ActionResult (or the relevant digests to get logs from CAS) should be preferred.

One such ByteStream client is provided as part of BuildGrid,

bgd logstream read resource-name

An alternative is the logstream-tail tool, which is part of buildbox-tools.

Standalone LogStream service

When deployed as a standalone service, the same broad concepts hold. However, there will need to be a way to create streams (using a CreateLogStream request), and communicate the read and write resource names to clients who need them.

There is a BuildGrid command to create a stream,

bgd logstream create some-identifier-to-include-in-the-name

Once a stream has been created, it can be written to by making a ByteStream Write request to the LogStream server using the writeable resource name returned by CreateLogStream.

There is also a BuildGrid command to get an interactive prompt for writing to a stream,

bgd logstream write write-resource-name

When using this tool sending an empty message will finalize and commit the stream, which will cause the stream to be deleted when all readers are finished reading.

A more generally useful tool for use with a standalone LogStream is the outputstreamer tool from buildbox. This streams the contents of stdin to a LogStream server’s ByteStream endpoint.

The buildgrid-logstream repository contains a simple docker-compose file containing a standalone LogStream server for demo/experimentation purposes. This can be spun up using (in the root of the repository)

This serves the LogStream and ByteStream APIs at http://localhost:50051.