Fetch API proposal for review

[Sander Striker]

Hi,

We've been iterating on the Fetch API proposal for a while now, and feel it is in a state that it is ready for broader review.

We welcome your feedback, especially when you have concerns about cases that are excluded by the design approach.

I suggest we close review on Dec 31st, so that we can get this over the line this year ;).  That is of course assuming no major concerns are raised that won't have been addressed at that point.

Cheers,

Sander

[Eric Burnett]

Thanks especially to Sander for jumping in and co-authoring revisions to this API (it's now been through many!), and continuing to push it forward in general; and many more folk for continued ideas and feedback. I'm quite happy where it's settled, especially vs what I thought was fairly close at recent working group meetings :).

Notable changes since this was last reviewed with the group:

  (1) The API has been pared down to two concrete return types - Blob and Directory, without 'Any' protos or similar. Simple use-cases like "fetch and cache a URL" are especially clean now.

  (2) The 'Pusher' APIs are now in scope as well, so use-cases requiring key-value storage or simplified sharing amongst peers now can be accommodated via only the standard API.

+1 to your choice of Dec 31. Let's do this!

--
You received this message because you are subscribed to the Google Groups "Remote Execution APIs Working Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to remote-execution...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/remote-execution-apis/CAND%2B6xmGQCh9evDNBbePsxQVQU4%2BJfZY7mCqLUyngfdqFshN4g%40mail.gmail.com.

[Jakob Buchgraber]

Thanks Sander, Eric as well as John for writing the original proposal and flying to BazelCon all the way from Tokyo only to push for progress on this :-).

The API proposal looks good to me. The Fetcher service is mostly identical to John's original proposal anyway. I understand that the Pusher service is motivated by BuildStream at this point?

- Jakob

To view this discussion on the web visit https://groups.google.com/d/msgid/remote-execution-apis/CAJZTgz27Pz2iww1oqQQ17oMzwpLe1KhZm%2BkvjyavyrknO%2BRSOw%40mail.gmail.com.

[Ed Baunton]

I am also very happy with the current state of the proposal; thank you for pushing this along all.

There are a few use cases for the Pusher service: there are details on the BuildStream use case in the doc. We have an additional use case for the pusher: imagine a fetcher service that is wired up to pull from a large central repository that contains only *published* artifacts. If a user wants to perform an integration build consisting of some data that is not already in the central repository (i.e. local edits), they can push it into the CAS and associate it as a published artifact using the Pusher service. For a concrete example, in the Bazel world, I am imagining something like a workspace rule (e.g. git_repository) with a series of local patches (not published). If the patches are applied after the data has been fetched from the fetch server, the tree would need be reuploaded and reassociated with the new patches via the Pusher API.

I am not a Bazel expert but hopefully that example makes sense,

Ed

To view this discussion on the web visit https://groups.google.com/d/msgid/remote-execution-apis/CAGQ4vn3MW_uy9Bo_h85ej%3DC%2BrdDMxf7tB-Asqz4K9RuzrxX-ow%40mail.gmail.com.

[Jakob Buchgraber]

On Tue, Dec 17, 2019 at 2:36 PM Ed Baunton <edba...@gmail.com> wrote:

I am also very happy with the current state of the proposal; thank you for pushing this along all.

There are a few use cases for the Pusher service: there are details on the BuildStream use case in the doc. We have an additional use case for the pusher: imagine a fetcher service that is wired up to pull from a large central repository that contains only *published* artifacts. If a user wants to perform an integration build consisting of some data that is not already in the central repository (i.e. local edits), they can push it into the CAS and associate it as a published artifact using the Pusher service. For a concrete example, in the Bazel world, I am imagining something like a workspace rule (e.g. git_repository) with a series of local patches (not published). If the patches are applied after the data has been fetched from the fetch server, the tree would need be reuploaded and reassociated with the new patches via the Pusher API.

Thanks Ed. That does make sense. I think in the Bazel world we wouldn't want repo rules to write to a service. The deployment use case makes sense though.

- Jakob

[Sander Striker]

On Fri, Dec 13, 2019 at 11:39 PM Eric Burnett <ericb...@google.com> wrote:

+1 to your choice of Dec 31. Let's do this!

And we've arrived at the end of 2019...  I haven't seen any objections raised to moving forward at this point.  I'll be clearing out comments and changing the status later this week.

Thanks for all your patience and support.  Happy New Year!

Cheers,

Sander 

[Sander Striker]

Hi,

As per usual I arrived at one of the hardest problems in CS: naming things.  While adding google.api.http url template annotations to the proto, it occurred to me that a better name is perhaps the "Remote Asset API", containing a "Fetch" and a "Push" service.  That seems to be more descriptive.

Thoughts?  Objections?

I'll give it 48 hours, then I'll update the PR I've put in for the API addition and un-WIP it.

Thanks in advance!

Cheers,

Sander

[Jakob Buchgraber]

No objections. Alternatives to consider could be

Remote Artifact API

Remote Dependency Cache API

Best,

Jakob

--

You received this message because you are subscribed to the Google Groups "Remote Execution APIs Working Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to remote-execution...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/remote-execution-apis/CAND%2B6xmnyvzKOhni-dGKPp7%2B61kO2jtUx8P%3DgEi0a72ZrT%2B-_Q%40mail.gmail.com.

[Sander Striker]

24h left on the clock.

I'm going to disqualify Remote Dependency Cache API as this API could be used for outputs as well as inputs.  I also like to think of this API as a bit stronger than Cache - at least leave it open to the implementation whether to persist forever maybe even beyond the lifetime of the origin.

The current tally:

Remote Fetch API: 0
Remote Asset API: 1 (Sander Striker)

Remote Artifact API: 1 (Erik Mavrinac)

To compare how they looks like:

https://github.com/sstriker/remote-apis/tree/remote-fetch-api

https://github.com/sstriker/remote-apis/tree/remote-asset-api

https://github.com/sstriker/remote-apis/tree/remote-artifact-api

Please cast your explicit votes.

Cheers,

Sander

[Steven Bergsieker]

Remote Asset API has my vote.

--

You received this message because you are subscribed to the Google Groups "Remote Execution APIs Working Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to remote-execution...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/remote-execution-apis/CAND%2B6xkg5JiViU%2BLUDQkNn28EQkdtmgqZ2aHv%2BsguS-X6576LA%40mail.gmail.com.

[Justin Wei (Mobile Infra)]

+1 for Remote Asset API

To view this discussion on the web visit https://groups.google.com/d/msgid/remote-execution-apis/CAA3Cs40nvKh7k%2BWg-WUX3mZUBT0WMLmg7c2HowaxH2ReW%3DkHcA%40mail.gmail.com.

[Eric Burnett]

I don't feel strongly in any direction, but Asset SGTM.

To view this discussion on the web visit https://groups.google.com/d/msgid/remote-execution-apis/CAMwNJxxvtbYzDOm8o5pW6-VkSSgnFmjzPfSmrb336QAUmkyuug%40mail.gmail.com.

[Mostyn Bramley-Moore]

+1 for Asset, it's easier to spell than Artifact.

-Mostyn.

To view this discussion on the web visit https://groups.google.com/d/msgid/remote-execution-apis/CAJZTgz22_w379xdOkHs9UaFuo56a1Tg0cL5p2-oNANXENquZ9g%40mail.gmail.com.

[Peter Ebden]

I would be happy enough with any of them, but Asset seems like a good choice.

To view this discussion on the web visit https://groups.google.com/d/msgid/remote-execution-apis/CACfMsGHZZ2g83%3De%2BYAzq8RdjUcFK9sNz%3DbS%3DQf9QuMRrpk%2Bbvw%40mail.gmail.com.

[Darius Makovsky]

Hi Sander,

I've looked at the diffs of the protos and I think 'asset' has the advantage of being less ambiguous; I found 'artifact' confusing because in other contexts it has othere specific meanings). However I wonder if it's worth preserving the noun naming of the services eg 'Fetcher' rather than 'Fetch'. Apologies if that's already been discussed at length and ultimately either is fine provided it becomes convention.

On 23/01/2020 19:51, Sander Striker wrote:
> 24h left on the clock.
>
> I'm going to disqualify Remote Dependency Cache API as this API could be used
> for outputs as well as inputs. I also like to think of this API as a bit
> stronger than Cache - at least leave it open to the implementation whether to
> persist forever maybe even beyond the lifetime of the origin.
>
> The current tally:
> Remote Fetch API: 0
> Remote Asset API: 1 (Sander Striker)
> Remote Artifact API: 1 (Erik Mavrinac)
>
> To compare how they looks like:
> https://github.com/sstriker/remote-apis/tree/remote-fetch-api
> https://github.com/sstriker/remote-apis/tree/remote-asset-api
> https://github.com/sstriker/remote-apis/tree/remote-artifact-api
>
> Please cast your explicit votes.
>
> Cheers,
>
> Sander
>

--
Best Regards,
Darius


For Codethink's privacy-policy please see https://www.codethink.co.uk/privacy.html

[Sander Striker]

I think it's clear enough to call it.  The "Remote Asset API" it is.   #112 is now up to date, and in anticipation of landing.  Those with karma to make this happen, please have a look.

Have a great weekend!

Cheers,

Sander