remote-apis-testing: a refresh

[Christopher Phang]

Dear all,

Over the last few weeks, I and a few others have been working on revamping the existing remote-apis-testing[0] repository, which was covered last year by Laurence[1]. For those that aren't familiar with the project, remote-apis-testing provides an independent integration test suite for REAPI client and server implementations. The aim of the project is for users and developers of REAPI implementations to be able to identify whether their implementations are interoperable with one another, and where there are any potential issues report these to the relevant upstream communities[2] as quickly as possible.

One thing to note is that these tests can be run as easily locally as they can on gitlab.com CI. The only host-side dependencies required are docker and docker-compose. Therefore, for developers that are using REAPI clients and servers for the first time we hope that the  deployments that are offered here will be of use as a 'quickstart' and for testing.

Currently, we have support for testing goma, recc and bazel clients, and buildbarn, buildfarm and buildgrid servers. We are aiming to introduce testing for the remote asset API very soon as well[3]. If your client and/or server is not currently being tested, we would love to hear from you. Details on how to add new implementations can be found in our CONTRIBUTING[4] guide.

Many thanks,

Chris

[0] https://gitlab.com/remote-apis-testing/remote-apis-testing

[1] https://groups.google.com/d/msg/remote-execution-apis/qSZR05YFjjM/mB6rDtVzBAAJ

[2] https://github.com/bazelbuild/bazel-buildfarm/issues/492

[3] https://gitlab.com/remote-apis-testing/remote-apis-testing/-/issues/87

[4] https://gitlab.com/remote-apis-testing/remote-apis-testing/-/blob/master/CONTRIBUTING.md#adding-new-client-and-server-implementations

[Ulf Adams]

Hi Chris,

Thanks, that looks interesting. I have some feedback, and I currently don't have a Gitlab login, so I'm posting it here.

- Have you considered how to onboard commercial services? Right now, it looks like the test flow only supports open source products. (You can also say you're not planning to add commercial services, of course.)

- The Absl build seems pretty small - something like TensorFlow would be more interesting (it hits a corner case for us that is technically in violation of the spec as written, but seems to work with some other implementation).

- Goma tries hard to fall back to local execution if the remote service doesn't work. This may skew the results towards it working even when it doesn't actually work. This actually happened when I ran a simple test.

- I was actually expecting more of a synthetic test suite than real client runs. Are there any plans to add synthetic tests to the suite? We internally have a tool that can generate synthetic test loads.

- Are you planning to upstream any of the Goma server patches?

Fun fact - you're enabling a flag in Goma that I added back in March (or so): -insecure-skip-verify. IIUC it's actually not necessary because you're disabling TLS for the service connection.

Thanks,

-- Ulf

--
You received this message because you are subscribed to the Google Groups "Remote Execution APIs Working Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to remote-execution...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/remote-execution-apis/3dd6fd0d-4613-40c0-b796-1e9eb5d7c6aeo%40googlegroups.com.

[Mostyn Bramley-Moore]

On 16/8/20 12:49 am, Ulf Adams wrote:
> Hi Chris,
>
> Thanks, that looks interesting. I have some feedback, and I currently don't have a Gitlab login, so I'm posting it here.
>
> - Have you considered how to onboard commercial services? Right now, it looks like the test flow only supports open source products. (You can also say you're not planning to add commercial services, of course.)
> - The Absl build seems pretty small - something like TensorFlow would be more interesting (it hits a corner case for us that is technically in violation of the spec as written, but seems to work with some other implementation).
> - Goma tries hard to fall back to local execution if the remote service doesn't work. This may skew the results towards it working even when it doesn't actually work. This actually happened when I ran a simple test.

I think that's a one-line fix:
https://gitlab.com/remote-apis-testing/remote-apis-testing/-/merge_requests/159

> - I was actually expecting more of a synthetic test suite than real client runs. Are there any plans to add synthetic tests to the suite? We internally have a tool that can generate synthetic test loads.

This sounds like it would be a nice addition.

> - Are you planning to upstream any of the Goma server patches?

The no-auth patch is definitely worth upstreaming- I'm doing something almost identical in a local fork.

I have also been debugging some client side authentication timeouts, even with authentication disabled on the server. You might need a longer running testcase to hit these problems (or maybe my server patch is wonky).


-Mostyn.

> Fun fact - you're enabling a flag in Goma that I added back in March (or so): |-insecure-skip-verify|. IIUC it's actually not necessary because you're disabling TLS for the service connection.
>
> Thanks,
>
> -- Ulf

>
> On Fri, Aug 14, 2020 at 7:04 PM Christopher Phang <christop...@codethink.co.uk <mailto:christop...@codethink.co.uk>> wrote:
>
> Dear all,
>
> Over the last few weeks, I and a few others have been working on revamping the existing remote-apis-testing[0] repository, which was covered last year by Laurence[1]. For those that aren't familiar with the project, remote-apis-testing provides an independent integration test suite for REAPI client and server implementations. The aim of the project is for users and developers of REAPI implementations to be able to identify whether their implementations are interoperable with one another, and where there are any potential issues report these to the relevant upstream communities[2] as quickly as possible.
>

> One thing to note is that these tests can be run as easily locally as they can on gitlab.com <http://gitlab.com> CI. The only host-side dependencies required are docker and docker-compose. Therefore, for developers that are using REAPI clients and servers for the first time we hope that the  deployments that are offered here will be of use as a 'quickstart' and for testing.

>
> Currently, we have support for testing goma, recc and bazel clients, and buildbarn, buildfarm and buildgrid servers. We are aiming to introduce testing for the remote asset API very soon as well[3]. If your client and/or server is not currently being tested, we would love to hear from you. Details on how to add new implementations can be found in our CONTRIBUTING[4] guide.
>
> Many thanks,
>
> Chris
>
> [0] https://gitlab.com/remote-apis-testing/remote-apis-testing
> [1] https://groups.google.com/d/msg/remote-execution-apis/qSZR05YFjjM/mB6rDtVzBAAJ
> [2] https://github.com/bazelbuild/bazel-buildfarm/issues/492

> [3] <https://gitlab.com/remote-apis-testing/remote-apis-testing/-/blob/master/CONTRIBUTING.md#adding-new-client-and-server-implementations>https://gitlab.com/remote-apis-testing/remote-apis-testing/-/issues/87

> <https://gitlab.com/remote-apis-testing/remote-apis-testing/-/issues/87>
> [4] https://gitlab.com/remote-apis-testing/remote-apis-testing/-/blob/master/CONTRIBUTING.md#adding-new-client-and-server-implementations
>
> --
> You received this message because you are subscribed to the Google Groups "Remote Execution APIs Working Group" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to remote-execution...@googlegroups.com <mailto:remote-execution...@googlegroups.com>.
> To view this discussion on the web visit https://groups.google.com/d/msgid/remote-execution-apis/3dd6fd0d-4613-40c0-b796-1e9eb5d7c6aeo%40googlegroups.com <https://groups.google.com/d/msgid/remote-execution-apis/3dd6fd0d-4613-40c0-b796-1e9eb5d7c6aeo%40googlegroups.com?utm_medium=email&utm_source=footer>.

>
> --
> You received this message because you are subscribed to the Google Groups "Remote Execution APIs Working Group" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to remote-execution...@googlegroups.com <mailto:remote-execution...@googlegroups.com>.
> To view this discussion on the web visit https://groups.google.com/d/msgid/remote-execution-apis/CAO8_sfm%2B0RACdLPrEvNqZ4fXF8fbQBv7hsYBUuXYvR7aF5BsHQ%40mail.gmail.com <https://groups.google.com/d/msgid/remote-execution-apis/CAO8_sfm%2B0RACdLPrEvNqZ4fXF8fbQBv7hsYBUuXYvR7aF5BsHQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.


--
Mostyn Bramley-Moore
mos...@antipode.se

[christop...@codethink.co.uk]

Hi Ulf,

Thanks for the feedback, very much appreciated!

> - Have you considered how to onboard commercial services? Right now, it looks like the test flow only supports open source products. (You can also say you're not planning to add commercial services, of course.)

Not yet, although approaches on how we might do this would be appreciated. Currently our test harnesses are kept simple because we deploy server and client implementations on the same runner with independent docker-compose files. It would be slightly trickier to do this whilst additionally considering a commerical service, but certainly not impossible. Happy to discuss further in an issue!

> - The Absl build seems pretty small - something like TensorFlow would be more interesting (it hits a corner case for us that is technically in violation of the spec as written, but seems to work with some other implementation).

> - I was actually expecting more of a synthetic test suite than real client runs. Are there any plans to add synthetic tests to the suite? We internally have a tool that can generate synthetic test loads.

That's interesting, currently as we are using gitlab shared runners these builds have to stay fairly small. However, I think the wider point is that absl-hello might not cover enough of the API calls that this is a meaningful test. I would be interested to consider other Bazel projects that we could test. My only thought would be that TensorFlow would be a bit too big, I think if we were to consider such a test that would have to be done as a nightly.

I think this lends itself to the next point about having a synthetic test suite. I think that is incredibly important. This has already been discussed to some extent in https://gitlab.com/remote-apis-testing/remote-apis-testing/-/issues/73 but I would be interested to hear your thoughts on this, especially if you've already done some work in this area.

I'll reply to your other questions in my reply to Mostyn, thanks very much again for your feedback.

Chris

[christop...@codethink.co.uk]

Hi Mostyn,

>> - Goma tries hard to fall back to local execution if the remote service doesn't work. This may skew the results towards it working even when it doesn't actually work. This actually happened when I ran a simple test.

> I think that's a one-line fix:
> https://gitlab.com/remote-apis-testing/remote-apis-testing/-/merge_requests/159

Indeed, thanks very much for submitting the patch.

>> - Are you planning to upstream any of the Goma server patches?

> The no-auth patch is definitely worth upstreaming- I'm doing something almost identical in a local fork.
> I have also been debugging some client side authentication timeouts, even with authentication disabled on the server. You might need a longer running testcase to hit these problems (or maybe my server patch is wonky).

Indeed, I was planning on getting the no-auth patch submitted as a changeset this week. The other patches I think will require some discussion on chromium-dev to see if the approach is worthwhile.

I'd be interested to hear more about your timeouts issue. Again that's probably worth discussing in another forum.

Many thanks,

Chris

 

[Ulf Adams]

Hi all,

The remote-apis-testing website currently shows all red 404 badges.

Cheers,

-- Ulf

--

You received this message because you are subscribed to the Google Groups "Remote Execution APIs Working Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to remote-execution...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/remote-execution-apis/90641c7d-8324-429e-b6b0-0dea8de25f85n%40googlegroups.com.

[christop...@codethink.co.uk]

Hi Ulf

Yep I've just seen this, thanks for picking this up and apologies for this. I'll make sure that this gets fixed asap tomorrow morning.

Chris

[christop...@codethink.co.uk]

Hi Ulf

This is now fixed, for those interested the relevant MR is https://gitlab.com/remote-apis-testing/remote-apis-testing/-/merge_requests/172

Many thanks,

Chris

On Thursday, 17 September 2020 at 22:55:47 UTC+1 ulf...@gmail.com wrote: