LogStream API

[Eric Burnett]

I took an AI a couple meetings ago, to publish our 'LogStream' API. Better late than never, here ya go!

Proto

Design sketch

In short, it defines a one-method API - CreateLogStream - and semantics around concurrently readable/writable streams of bytes, to be read/written with ByteStream. This is what we intended to use with stdout_stream_name / stderr_stream_name, though we haven't yet gotten to adding bazel-side support yet. (Though we do use these from our workers for all stdout they produce). But conveniently, from a reader perspective these are simply bytestreams, no special semantics required.

Hope this is helpful. If there's interest in moving this to remote-apis proper, happy to support that as well.

Cheers,

Eric (and Peter, in abstentia)

[George Gensure]

Thanks very much for this, Eric! Appreciate it from a consensus perspective to be able to present a useful common interface.

-George

--
You received this message because you are subscribed to the Google Groups "Remote Execution APIs Working Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to remote-execution...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/remote-execution-apis/CAJZTgz2S_LVd9XkB%3D9GGD1UNuqhMebCgSKtART%2B4rx0LuMAv2Q%40mail.gmail.com.

[Ed Baunton]

Thank you Eric! We are looking at adding streaming logs to BuildGrid
soon. Keeping in mind the use cases brought up in the last meeting and
this additional info, there might be some proposals to the API on the
way.

Thanks,

Ed

On Thu, 19 Mar 2020 at 17:31, 'George Gensure' via Remote Execution

APIs Working Group <remote-exe...@googlegroups.com> wrote:
>
> Thanks very much for this, Eric! Appreciate it from a consensus perspective to be able to present a useful common interface.
>
> -George
>
> On Thu, Mar 19, 2020 at 5:14 PM 'Eric Burnett' via Remote Execution APIs Working Group <remote-exe...@googlegroups.com> wrote:
>>
>> I took an AI a couple meetings ago, to publish our 'LogStream' API. Better late than never, here ya go!
>>
>> Proto
>> Design sketch
>>
>> In short, it defines a one-method API - CreateLogStream - and semantics around concurrently readable/writable streams of bytes, to be read/written with ByteStream. This is what we intended to use with stdout_stream_name / stderr_stream_name, though we haven't yet gotten to adding bazel-side support yet. (Though we do use these from our workers for all stdout they produce). But conveniently, from a reader perspective these are simply bytestreams, no special semantics required.
>>
>> Hope this is helpful. If there's interest in moving this to remote-apis proper, happy to support that as well.
>>
>> Cheers,
>> Eric (and Peter, in abstentia)
>>
>> --
>> You received this message because you are subscribed to the Google Groups "Remote Execution APIs Working Group" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to remote-execution...@googlegroups.com.
>> To view this discussion on the web visit https://groups.google.com/d/msgid/remote-execution-apis/CAJZTgz2S_LVd9XkB%3D9GGD1UNuqhMebCgSKtART%2B4rx0LuMAv2Q%40mail.gmail.com.
>
> --
> You received this message because you are subscribed to the Google Groups "Remote Execution APIs Working Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to remote-execution...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/remote-execution-apis/CAB5czhfn6CkXsYfrvATsJDPkcDLOVD3n2nwjp_ut_SQmo_oAyw%40mail.gmail.com.

[Santiago Gil]

Hi.

Thanks for sharing, Eric.

As Ed mentioned, we are looking to support streaming outputs in
BuildGrid and buildbox.

For illustration, this doc describes how the LogStream API would be
used:

https://docs.google.com/document/d/1jI-7Gm3VgAmohN4EATJflGywQbismSYe83dX9NoqUEU

We are also keeping an eye on the other discussion on this list for some
of the details like interleaving stdout/err and the formatting of the
transmitted logs.

Best,
Santiago

On 2020-03-19 21:14, 'Eric Burnett' via Remote Execution APIs Working

Group wrote:
> I took an AI a couple meetings ago, to publish our 'LogStream' API.
> Better late than never, here ya go!
>

> Proto [1]
> Design sketch [2]

>
> In short, it defines a one-method API - CreateLogStream - and
> semantics around concurrently readable/writable streams of bytes, to
> be read/written with ByteStream. This is what we intended to use with
> stdout_stream_name / stderr_stream_name, though we haven't yet gotten
> to adding bazel-side support yet. (Though we do use these from our
> workers for all stdout they produce). But conveniently, from a reader
> perspective these are simply bytestreams, no special semantics
> required.
>
> Hope this is helpful. If there's interest in moving this to
> remote-apis proper, happy to support that as well.
>
> Cheers,
> Eric (and Peter, in abstentia)
>

> --
> You received this message because you are subscribed to the Google
> Groups "Remote Execution APIs Working Group" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to remote-execution...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/remote-execution-apis/CAJZTgz2S_LVd9XkB%3D9GGD1UNuqhMebCgSKtART%2B4rx0LuMAv2Q%40mail.gmail.com

> [3].
>
>
> Links:
> ------
> [1]
> https://gist.github.com/EricBurnett/a094e0b8474b834d098e86bf20fa202b
> [2]
> https://docs.google.com/document/d/1UYWevPLquZBCWTJXrfy9TW1bPjQNKq9Y22ZQoYrmB6A/edit#
> [3]
> https://groups.google.com/d/msgid/remote-execution-apis/CAJZTgz2S_LVd9XkB%3D9GGD1UNuqhMebCgSKtART%2B4rx0LuMAv2Q%40mail.gmail.com?utm_medium=email&utm_source=footer

[Eric Burnett]

Thanks for sharing Santiago! I added a couple comments to your doc, but from a quick read it all looked sensible to me.

To view this discussion on the web visit https://groups.google.com/d/msgid/remote-execution-apis/dea5e0d797462aff84c6e2edd200b2de%40codethink.co.uk.

[Santiago Gil]

On 2020-04-20 19:03, Eric Burnett wrote:
> Thanks for sharing Santiago! I added a couple comments to your doc,
> but from a quick read it all looked sensible to me.
>
> On Mon, Apr 20, 2020 at 12:51 PM Santiago Gil
> <santia...@codethink.co.uk> wrote:
>

<snip>

>> As Ed mentioned, we are looking to support streaming outputs in
>> BuildGrid and buildbox.
>>
>> For illustration, this doc describes how the LogStream API would be
>> used:
>>
>>
> https://docs.google.com/document/d/1jI-7Gm3VgAmohN4EATJflGywQbismSYe83dX9NoqUEU
>>

<snip>

Hi.

We finished implementing this in BuildGrid and buildbox.

There are two slight differences from the original draft. One is that
BuildGrid now provides its own LogStream service [0] instead of relying
on a separate server.

The other is that we ended up leveraging ByteStream's
`QueryWriteStatus()` to avoid having workers stream logs that will
remain unread: BuildGrid blocks on that call until the first reader
arrives, and only when the call returns successfully a worker will start
writing to the LogStream [1].

Thanks again for the feedback, Eric.

Best,
Santiago


[0] https://buildgrid.build/user/using_logstream.html

[1]
https://buildgrid.gitlab.io/buildbox/buildbox-home/standard-outputs-streaming.html#on-demand-streaming

[Eric Burnett]

We finished implementing this in BuildGrid and buildbox.

There are two slight differences from the original draft. One is that
BuildGrid now provides its own LogStream service [0] instead of relying
on a separate server.

Very nice! That looks super polished already (out of the box client tools!), sweet.

The other is that we ended up leveraging ByteStream's
`QueryWriteStatus()` to avoid having workers stream logs that will
remain unread: BuildGrid blocks on that call until the first reader
arrives, and only when the call returns successfully a worker will start
writing to the LogStream [1].

Pretty creative hack, I think I can see how that'd work. I'm curious, did you consider pursuing having the client signal upfront when it's going to want streamed logs?  I know the REAPI doesn't have a field for that today, but IIRC it was floated in one of the past discussions (one around structured log outputs? Same discussion that considered optionally allowing the client to request other files be streamed back too, I think), and might be a less race-y option for you in the long-term. Given that ~nobody implements output streams yet (before you, anyways!), updating the API to say "clients SHOULD set <blah> if they want stream handles passed back" I don't think would inconvenience anyone in practice. We could propose it and find out, anyways, if that sounds like something you'd want.

[Santiago Gil]

On 2020-09-26 00:20, 'Eric Burnett' via Remote Execution APIs Working
Group wrote:
<snip>

>
> Pretty creative hack, I think I can see how that'd work.

Thank you!

> I'm curious, did you consider pursuing having the client signal upfront
> when it's
> going to want streamed logs?

<snip>

We thought about it. But the thing is that, because BuildGrid does job
de-duplication, it could happen that one client submits a job without
requesting the streamed logs and a bit later another client submits the
same job requesting streaming. So to be able to honor the second request
we'd also need a way to relay the message to the worker to start
streaming once it's already executing.

The QueryWriteStatus() mechanism solved that problem without adding
additional logic to the worker, which always receives resource names and
unconditionally sets everything up for streaming before it starts
executing the command.