Is stdout/stderr digest present even when contents are inlined?

[Tiago Quelhas]

Hi all,

I'm seeking clarification on the semantics of the `stdout_digest` and `stderr_digest` fields on ActionResult.

Are these fields intended to be present even when the contents are inlined (i.e., when the `stdout_raw` and `stderr_raw` fields are set)? The "Clients SHOULD NOT populate this field when uploading to the cache" provision hints at the answer being yes, but it's a rather oblique inference. Should we formalize this guarantee?

The context is that, if the contents are inlined but the digest is missing, Bazel itself needs to compute the digest in some "build without the bytes" scenarios. This has a (relatively minor) performance impact and, more importantly, is a code path that could be entirely avoided.

Thanks,

Tiago

[Mostyn Bramley-Moore]

On 10/3/23 15:05, 'Tiago Quelhas' via Remote Execution APIs Working Group wrote:
> Hi all,
>
> I'm seeking clarification on the semantics of the `stdout_digest` and `stderr_digest` fields on ActionResult.
>
> Are these fields intended to be present even when the contents are inlined (i.e., when the `stdout_raw` and `stderr_raw` fields are set)? The "Clients SHOULD NOT populate this field when uploading to the cache" provision hints at the answer being yes, but it's a rather oblique inference. Should we formalize this guarantee?

I think it would be good to formalize this.

Originally the text did not mention anything about clients not uploading the *_raw fields. I then made bazel-remote's implementation support *_raw fields being set in uploads, and implemented some (non-opensource) clients that took advantage of this, and then ran into trouble when using those clients with other cache servers. Then we clarified in the spec that clients should not populate the *_raw fields when uploading ActionResults. Nobody knew of any other clients doing this, so we suspect that most if not all other clients always specify the digests when uploading, and most if not all servers leave those digests intact.

I think it's OK for servers to choose to be defensive when receiving uploads, eg by allowing inlined *_raw fields, and to calculate the *_digest fields from inlined *_raw fields (and de-inlining where necessary).

Servers should never return an ActionResult without the *_digest fields set (either the client should have specified them, or the server should have populated them, or the server should have rejected the upload).

Clients should be able to reasonably assume that the *_digest fields are always set when downloading an ActionResult, but it's also OK for them to be defensive and infer this from inlined *_raw fields.

> The context is that, if the contents are inlined but the digest is missing, Bazel itself needs to compute the digest in some "build without the bytes" scenarios. This has a (relatively minor) performance impact and, more importantly, is a code path that could be entirely avoided.

If it were up to me, I would probably make Bazel quietly handle this case, because the alternatives are worse IMO:
* Fall back to local execution, possibly with a warning (bigger performance impact than hashing the *_raw fields)
* Fail the build (and require a server-side change to resolve)


-Mostyn.
--
Mostyn Bramley-Moore
mos...@antipode.se