Checking if my "dist test" results are on par with Go builders

[Davis Goodin]

Hi all,

I'm working on infrastructure to build a copy of Go and run dist tests on Ubuntu (for Microsoft). I looked at the Go builder infra and did my best to run the build/tests the same way on our existing machine pools with Docker containers, and now I'm looking for a way to confirm the tests are actually running the way they should. My biggest concern is that some tests might skip even though they wouldn't skip on an ordinary Go builder. For example, if a longtest sees I'm missing a dependency and skips, how can I detect that and figure out whether it's normal?

My first thought was to grab logs from https://build.golang.org and diff against mine, but logs aren't preserved there unless the run fails. I found an open issue tracking preserving successful logs: https://github.com/golang/go/issues/34119 (last comment 2019/10/31).

However, even if I had those logs, "go tool dist test" output isn't verbose, so diffing against my results would miss info that doesn't end up on top. I'm able to put "-v" into my local "dist test" code to make my tests log a lot more information, but that isn't very useful to me--it's a lot of logs to sift through (~11 MB) and I don't have verbose output from https://build.golang.org to compare against.

Is there some way to essentially audit the "go tool dist test" results?

Thanks,

Davis

[Ian Lance Taylor]

I don't know of a way to do the kind of audit that you are looking for.

But you should at least be able to use -v and look for SKIP lines.
Check those to make sure that they aren't specific to your
environment. That will find any cases that are skipped due to missing
dependencies. (There are very few such tests in any case.)

Ian

[Davis Goodin]

Ah, that's unfortunate. Yeah, "dist test -v" doesn't show many skips, and that's why I modified it to pass "-v" to the subcommands--I knew there were some skips that weren't being logged. (Especially in short mode.)

When I grep "SKIP" in my extra-verbose linux-amd64-longtest log, I see ~550 skips:

https://gist.github.com/dagood/2bc08e37da295c2022b9196572d378a4

I suppose it's not necessarily impossible to review--it does look like each one has an explanation on the line above--but it's a lot. :) A baseline to get rid of some of them is what I was hoping for when I was looking around on https://build.golang.org.

Am I overthinking the verbosity? Is "-v" on the inner test commands not necessary to do this kind of check?

The next thing that comes to mind is for me to try to set up the Go buildlets/coordinator/etc. locally to try to get an extra-verbose baseline. There's still the possibility of my setup going wrong somewhere and giving me a bad baseline anyway... but it seems like it might be my best option.

[Ian Lance Taylor]

On Mon, Apr 19, 2021 at 3:20 PM 'Davis Goodin' via golang-dev
<golan...@googlegroups.com> wrote:
>
> Ah, that's unfortunate. Yeah, "dist test -v" doesn't show many skips, and that's why I modified it to pass "-v" to the subcommands--I knew there were some skips that weren't being logged. (Especially in short mode.)
>
> When I grep "SKIP" in my extra-verbose linux-amd64-longtest log, I see ~550 skips:
> https://gist.github.com/dagood/2bc08e37da295c2022b9196572d378a4

OK, that's more than I expected.

I did a spot check of ten at random, and they were all unimportant.
But it would be tedious to check all of them.

> I suppose it's not necessarily impossible to review--it does look like each one has an explanation on the line above--but it's a lot. :) A baseline to get rid of some of them is what I was hoping for when I was looking around on https://build.golang.org.

I don't think we have that information.

> Am I overthinking the verbosity? Is "-v" on the inner test commands not necessary to do this kind of check?

I would be surprised if it's necessary, but I can't prove that it's
not required.

> The next thing that comes to mind is for me to try to set up the Go buildlets/coordinator/etc. locally to try to get an extra-verbose baseline. There's still the possibility of my setup going wrong somewhere and giving me a bad baseline anyway... but it seems like it might be my best option.

Or just set up a Linux system and run the tests. You don't need a
whole buildlet.

Ian

[Davis Goodin]

Thanks for taking a look.

About setting up a Linux system, I'm not sure I understand. I'm running on Ubuntu, and I've also run them on a container based on golang/build env/linux-x86-stretch/Dockerfile (Debian). I also ran them on Fedora (without a Docker container). The basic tests run fine with the src/run.bash script, and I've been using "go tool dist test" with "GO_BUILDER_NAME" and other env vars set to emulate how the tests run on the builders.

I've been poking around at "dashboard/builders.go" in golang/build to figure out what I need to do to run each style of tests. From what I've seen in e.g. "cmd/coordinator/coordinator.go", it appears to be tightly linked to buildlets. I haven't been able to find a way to run the "go tool dist test" commands locally rather than on a buildlet.

Is there a way to run the tests without a buildlet and be sure the tests are running the same way they would in a buildlet?

[Ian Lance Taylor]

On Mon, Apr 19, 2021 at 4:10 PM 'Davis Goodin' via golang-dev
<golan...@googlegroups.com> wrote:
>
> About setting up a Linux system, I'm not sure I understand. I'm running on Ubuntu, and I've also run them on a container based on golang/build env/linux-x86-stretch/Dockerfile (Debian). I also ran them on Fedora (without a Docker container). The basic tests run fine with the src/run.bash script, and I've been using "go tool dist test" with "GO_BUILDER_NAME" and other env vars set to emulate how the tests run on the builders.
>
> I've been poking around at "dashboard/builders.go" in golang/build to figure out what I need to do to run each style of tests. From what I've seen in e.g. "cmd/coordinator/coordinator.go", it appears to be tightly linked to buildlets. I haven't been able to find a way to run the "go tool dist test" commands locally rather than on a buildlet.
>
> Is there a way to run the tests without a buildlet and be sure the tests are running the same way they would in a buildlet?

Likely I don't understand what you are trying to do. I thought you
were comparing tests across different kinds of systems.

If you just want to compare your test runs against the builders on the
same kind of system, then I wouldn't bother. The builders try to
basically do the same thing as run.bash, so just running run.bash
should be fine. There are some exceptions, but those are designed to
not matter. You can remove those exceptions by setting the
environment variable GO_BUILDER_NAME to some non-empty string; it
shouldn't matter what.

Ian

> On Monday, April 19, 2021 at 5:42:41 PM UTC-5 Ian Lance Taylor wrote:
>>
>> On Mon, Apr 19, 2021 at 3:20 PM 'Davis Goodin' via golang-dev
>> <golan...@googlegroups.com> wrote:
>> >
>> > Ah, that's unfortunate. Yeah, "dist test -v" doesn't show many skips, and that's why I modified it to pass "-v" to the subcommands--I knew there were some skips that weren't being logged. (Especially in short mode.)
>> >
>> > When I grep "SKIP" in my extra-verbose linux-amd64-longtest log, I see ~550 skips:
>> > https://gist.github.com/dagood/2bc08e37da295c2022b9196572d378a4
>>
>> OK, that's more than I expected.
>>
>> I did a spot check of ten at random, and they were all unimportant.
>> But it would be tedious to check all of them.
>>
>>
>> > I suppose it's not necessarily impossible to review--it does look like each one has an explanation on the line above--but it's a lot. :) A baseline to get rid of some of them is what I was hoping for when I was looking around on https://build.golang.org.
>>
>> I don't think we have that information.
>>
>>
>> > Am I overthinking the verbosity? Is "-v" on the inner test commands not necessary to do this kind of check?
>>
>> I would be surprised if it's necessary, but I can't prove that it's
>> not required.
>>
>>
>> > The next thing that comes to mind is for me to try to set up the Go buildlets/coordinator/etc. locally to try to get an extra-verbose baseline. There's still the possibility of my setup going wrong somewhere and giving me a bad baseline anyway... but it seems like it might be my best option.
>>
>> Or just set up a Linux system and run the tests. You don't need a
>> whole buildlet.
>>
>> Ian
>

> --
> You received this message because you are subscribed to the Google Groups "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/278ce66e-b79c-4d60-8e96-19a6269f1f81n%40googlegroups.com.

[Davis Goodin]

> I thought you were comparing tests across different kinds of systems.

I am, the systems are just very similar--Ubuntu (my image) vs. Debian (golang/build repo's image). Both x86-64, same dependencies as far as I can tell. My goal is to find some data that will help me figure out if my test run is being affected by any remaining differences that I'm not aware of.

Running the coordinator and buildlets as directly as possible (using the Debian image, etc.) is just a way that I think will get me some data about what's intended to happen. Then I can compare that against what happens on my build pool. The environment variables set by the coordinator and the specialized behavior for e.g. racecompile (doesn't actually run tests!) catch my eye, in particular.

My ultimate goal is for the CI system I'm setting up to validate the same things as the Linux builders on https://build.golang.org. For now, it's just to be as sure as possible our unmodified build of Go works. However, we do plan to do some work that we want to upstream, and I was thinking that going beyond run.bash is useful to continuously validate that our changes won't cause problems that would only show up in the Go infrastructure when they hit the more complex builders.

For example, we're interested in making it possible for Go's crypto libraries to use OpenSSL, for FIPS compliance reasons. I don't know enough about Go yet to know if that will have wide-reaching test impact. Maybe the -nocgo builders in particular will hit issues if build conditions aren't set up correctly. (I'm aware this example isn't easy to upstream, it's just the work we've been looking at first.)

> If you just want to compare your test runs against the builders on the

> same kind of system, then I wouldn't bother. The builders try to
> basically do the same thing as run.bash, so just running run.bash
> should be fine. There are some exceptions, but those are designed to
> not matter.

This is useful insight, thanks. With non-Go tests I've worked with before, there were sometimes big, intentional differences between a local run and a run in CI infra. This is why I've been so focused on replicating the builders so far--they're what shows up on the dashboard, and I figure they're the "full" tests. Good to know that isn't the intent here.

> You can remove those exceptions by setting the

> environment variable GO_BUILDER_NAME to some non-empty string; it
> shouldn't matter what.

I saw some tests that check for a prefix/suffix in testenv.Builder(), or compare the whole string. But, there are only a handful, so I suppose they're exceptions for very specific cases. The general rule makes sense, thanks.

[Ian Lance Taylor]

On Tue, Apr 20, 2021 at 9:45 AM 'Davis Goodin' via golang-dev
<golan...@googlegroups.com> wrote:
>
> For example, we're interested in making it possible for Go's crypto libraries to use OpenSSL, for FIPS compliance reasons. I don't know enough about Go yet to know if that will have wide-reaching test impact. Maybe the -nocgo builders in particular will hit issues if build conditions aren't set up correctly. (I'm aware this example isn't easy to upstream, it's just the work we've been looking at first.)

On this particular topic, see the dev.boringcrypto branch
(https://go.googlesource.com/go/+/refs/heads/dev.boringcrypto/README.boringcrypto.md)
and https://golang.org/issue/33281. (Apologies if you are already
aware of these.)

Ian

[Davis Goodin]

Yep, we also have our eyes on https://github.com/golang/go/pull/43900 that adds OpenSSL support (via dlopen) to the boringcrypto branch. Thanks for making sure, though.