Remote Execution Graph API

[Steven Bergsieker]

Several of us talked about the various proposed graph execution APIs at the London Build meetup this week. Here's a summary of what we talked about.

- There are multiple proposals for this (George, Ola, Sander, Danny, probably more), most of which haven't been circulated yet.

- At least for Google and Bloomberg, a graph API is a forward-looking problem, but isn't an immediate priority. There's strong interest in making sure that we end up with a good API, but no immediate plans to implement it.

- Pertinent dimensions include: whether this is an extension of the current Execute API vs. a new "Execute Graph" API, whether the graph is implicit, explicit, or sidechannel in the Execute API (or Execute Graph API), and what granularity of information and metadata should be returned (particularly in the case of failures).

- I think there was a general consensus pointing towards a separate API for graph execution, primarily because richer output semantics will likely be required, although potential downstream use cases are sketchy enough that we can't design for them now. We want to have a place to extend the API with graph-specific functionality that doesn't pollute the Action API.

- Next steps: everyone with a design should publish it soon. We also need to create a list of requirements. Then we can discuss the various designs and try to settle on the path forward.

[Sander Striker]

Thanks Steven for bringing this back to the list.  I think your summary is a fair representation of the current state of affairs.  More inline.

On Wed, Oct 2, 2019 at 2:48 PM 'Steven Bergsieker' via Remote Execution APIs Working Group <remote-exe...@googlegroups.com> wrote:

Several of us talked about the various proposed graph execution APIs at the London Build meetup this week. Here's a summary of what we talked about.

- There are multiple proposals for this (George, Ola, Sander, Danny, probably more), most of which haven't been circulated yet.

- At least for Google and Bloomberg, a graph API is a forward-looking problem, but isn't an immediate priority. There's strong interest in making sure that we end up with a good API, but no immediate plans to implement it.

I would welcome anyone with shorter timelines than 3-6 months to speak up; I think we are currently operating with that time horizon in mind for establishing this API.

 

- Pertinent dimensions include: whether this is an extension of the current Execute API vs. a new "Execute Graph" API, whether the graph is implicit, explicit, or sidechannel in the Execute API (or Execute Graph API), and what granularity of information and metadata should be returned (particularly in the case of failures).

- I think there was a general consensus pointing towards a separate API for graph execution, primarily because richer output semantics will likely be required, although potential downstream use cases are sketchy enough that we can't design for them now. We want to have a place to extend the API with graph-specific functionality that doesn't pollute the Action API.

- Next steps: everyone with a design should publish it soon. We also need to create a list of requirements. Then we can discuss the various designs and try to settle on the path forward.

As our discussion lead to new insights, my proposal is in state of flux.  Nevertheless, it's here https://docs.google.com/document/d/1EMSB7CAWrLNB_Gui3cwGs4v0NLmecho5O1bqGGUlY2M/edit#

Cheers,

Sander 

--
You received this message because you are subscribed to the Google Groups "Remote Execution APIs Working Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to remote-execution...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/remote-execution-apis/CAA3Cs419LecGFMesWLg9Pt2JdEdtmszkqVFrffWBbmo36L6R%3Dg%40mail.gmail.com.

[Ola Rozenfeld]

My proposal (or, rather, three of them...) is here. It is still a rough draft! I need to expand the section on comparing it to other proposals, and also on what data would be nice to have in order to make further decisions. I will also try to expand the Performance Goals section into a separate list of requirements document.

Danny's proposal is here.

Cheers,

Ola

[Sander Striker]

Thanks Ola.  Compliments for your "rough draft", it captures a lot :).  I'll take another pass reading before issuing comments.

Cheers,

Sander

[Son Luong Ngoc]

Hi folks,

Sorry for an attempt to revive an old thread but I am quite interested in a Remote Action Graph API as well.

# Context

Many organizations who are migrating to Bazel today in 2022 rely on "large actions" that are often relatively expensive to setup:

- Action depending on large files (ML dataset)
- Action depending on complex dependency with slow startup time (Docker Daemon, QEMU VMs, ...)

It would be nice to be able to speculatively schedule these actions at a warmed-up waiting state before all the prior parent actions have been completed.

For that to happen, I believe the ability to move Action Scheduling from the client side to the server side could be hugely beneficial.

Additionally, a Remote Action Graph API would go a long way.

Within Bazel ecosystem, there have been different solutions to calculate the diff in the graph between 2 revisions.

Having a way to query the action graph in a remote and centralized fashion would help ease users from having to set up their local workspace correctly.

# Question

1. Is this something that belongs to REv3 or is it too Bazel specific?

2. What are the latest thoughts/blockers around this topic? Has there been a newer attempt/effort that I could follow?

Cheers,

Son Luong.

[Peter Ebden]

I don't see this as a Bazel-specific concern; we would be interested too (mostly from the perspective of reducing the latency of a string of dependent actions). I think it may be a challenge to describe it in REAPI in a generic way - hopefully not impossible though.

To view this discussion on the web visit https://groups.google.com/d/msgid/remote-execution-apis/54f70891-c193-4eac-9783-c0cdb694e71cn%40googlegroups.com.