Extracting golden files from test results

[Andrew Allen]

I've been playing with doing testing based on golden files. Things are going great thus far, but I would like to make it easier to update my goldens. Looking through the test encyclopedia, there is an entry for "TEST_UNDECLARED_OUTPUTS_DIR". That looks extremely promising for what I want. 

The fact that there are undeclared outputs implies to me that there are declared test output files too. Is it possible to declare from inside my test that some output file should be saved? Or should I go down the route of dumping all the test files in there and saving them via a bash script? Where would I even look to get those files?

It would be REALLY cool if the file could be uploaded to Dash for collaborative debugging. Is that a thing that could be done given the current state of Bazel?

/** ~Andrew Z Allen */

[Eric Fellheimer]

Hmm, what about passing in a flag to your test that instructs it to save the file? You could pass in --test_arg=--save_golden_file=<path> ?

Eric Fellheimer
fe...@google.com

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAAVj6hwcT7wnJXyAF_rCACpOx7EUM6VgjRGfDyXoKNn5XOn5OA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Austin Schuh]

I've been doing it by using bazel run when I want to update the test data instead of trying to get bazel test to update it.

Austin

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAJMmbx%2BK%3DyixfXZqGu5cPvbva7pa8z%2B57DJ2Y8trHjr8zuOxZA%40mail.gmail.com.

[Andrew Allen]

Interesting advice.

/** ~Andrew Z Allen */

On Thu, Feb 11, 2016 at 12:44 PM, Austin Schuh <austin...@gmail.com> wrote:

I've been doing it by using bazel run when I want to update the test data instead of trying to get bazel test to update it.

So bazel run runs in the context of the real repo instead of in the sandbox? Or are you reaching into bazel-out (or wherever the files pop out) and copying them into your repo?

 

Austin

On Thu, Feb 11, 2016 at 10:22 AM 'Eric Fellheimer' via bazel-discuss <bazel-...@googlegroups.com> wrote:

Hmm, what about passing in a flag to your test that instructs it to save the file? You could pass in --test_arg=--save_golden_file=<path> ?

To quote the test encyclopedia:

"The root directory observed by a test may or may not be the real root directory. /proc shall be mounted. All build tools shall be present at the absolute paths under /usr used by a local installation. Paths starting with /home may not be available. Tests should not access any such paths. /tmp and /export/hda3/tmp shall be writable, but tests should avoid using these paths. Tests must not assume that any constant path is available for their exclusive use. Tests must not assume that atimes are enabled for any mounted filesystem."

I'm not sure if the sandboxing precludes writing to the normal file system. Based on my understanding of Bazel I think this would work but I'd like to make sure that I'm doing this in a way that won't silently break in the future.

[Austin Schuh]

Andrew,

On Thu, Feb 11, 2016 at 8:13 PM Andrew Allen <m...@andrewzallen.com> wrote:

Interesting advice.

/** ~Andrew Z Allen */

On Thu, Feb 11, 2016 at 12:44 PM, Austin Schuh <austin...@gmail.com> wrote:

I've been doing it by using bazel run when I want to update the test data instead of trying to get bazel test to update it.

So bazel run runs in the context of the real repo instead of in the sandbox? Or are you reaching into bazel-out (or wherever the files pop out) and copying them into your repo?

`bazel run` runs in the context of the host, after having built in the context of the sandbox.  The directory that it runs from seems to be the runfiles directory of the binary, FYI.  I use absolute paths when I want to write somewhere so I don't have to think as hard...  You could very likely make relative paths work.

This doesn't solve your desire to log the output upon test failure though, which seems like a neat idea.

 

 

Austin

On Thu, Feb 11, 2016 at 10:22 AM 'Eric Fellheimer' via bazel-discuss <bazel-...@googlegroups.com> wrote:

Hmm, what about passing in a flag to your test that instructs it to save the file? You could pass in --test_arg=--save_golden_file=<path> ?

To quote the test encyclopedia:

"The root directory observed by a test may or may not be the real root directory. /proc shall be mounted. All build tools shall be present at the absolute paths under /usr used by a local installation. Paths starting with /home may not be available. Tests should not access any such paths. /tmp and /export/hda3/tmp shall be writable, but tests should avoid using these paths. Tests must not assume that any constant path is available for their exclusive use. Tests must not assume that atimes are enabled for any mounted filesystem."

I'm not sure if the sandboxing precludes writing to the normal file system. Based on my understanding of Bazel I think this would work but I'd like to make sure that I'm doing this in a way that won't silently break in the future.

 

Sandboxing should preclude writing to the host.  The namespace is created with a lot of stuff bind mounted read-only.  If that isn't true, it would most likely be considered a bug.

I hope this helps!

Austin

[Brian Silverman]

Hi,

I've got a bit more info about how some of this stuff works.

On Thu, Feb 11, 2016 at 11:19 PM, Austin Schuh <austin...@gmail.com> wrote:

Andrew,

On Thu, Feb 11, 2016 at 8:13 PM Andrew Allen <m...@andrewzallen.com> wrote:

Interesting advice.

/** ~Andrew Z Allen */

On Thu, Feb 11, 2016 at 12:44 PM, Austin Schuh <austin...@gmail.com> wrote:

I've been doing it by using bazel run when I want to update the test data instead of trying to get bazel test to update it.

So bazel run runs in the context of the real repo instead of in the sandbox? Or are you reaching into bazel-out (or wherever the files pop out) and copying them into your repo?

`bazel run` runs in the context of the host, after having built in the context of the sandbox.  The directory that it runs from seems to be the runfiles directory of the binary, FYI.  I use absolute paths when I want to write somewhere so I don't have to think as hard...  You could very likely make relative paths work.

The documentation says "the test's current directory (which is in the runfiles tree)", meaning the CWD for tests is guaranteed to be bazel-bin/package/target.runfiles. That directory tree has enough symlinks in it though that it's probably easier to just use absolute paths.

This doesn't solve your desire to log the output upon test failure though, which seems like a neat idea.

Not sure if this makes sense for what you're doing, but you could print the contents of the new golden file (possibly encoded in some way and with delimiters) from the test and then have a separate script that extracts it from the test log file.

 

 

Austin

On Thu, Feb 11, 2016 at 10:22 AM 'Eric Fellheimer' via bazel-discuss <bazel-...@googlegroups.com> wrote:

Hmm, what about passing in a flag to your test that instructs it to save the file? You could pass in --test_arg=--save_golden_file=<path> ?

To quote the test encyclopedia:

"The root directory observed by a test may or may not be the real root directory. /proc shall be mounted. All build tools shall be present at the absolute paths under /usr used by a local installation. Paths starting with /home may not be available. Tests should not access any such paths. /tmp and /export/hda3/tmp shall be writable, but tests should avoid using these paths. Tests must not assume that any constant path is available for their exclusive use. Tests must not assume that atimes are enabled for any mounted filesystem."

I'm not sure if the sandboxing precludes writing to the normal file system. Based on my understanding of Bazel I think this would work but I'd like to make sure that I'm doing this in a way that won't silently break in the future.

 

Sandboxing should preclude writing to the host.  The namespace is created with a lot of stuff bind mounted read-only.  If that isn't true, it would most likely be considered a bug.

The root filesystem when running under the sandbox is made up of bind-mounted (read-only) files and directories and a temporary directory Bazel deletes afterwards for the root. Everything which is saved from the sandbox has to be explicitly copied before it gets cleaned up, which basically means only explicit outputs (none for tests) and a fixed set of logs etc for tests.

I hope this helps!

Austin

--

You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CABsbf%3DEd5vDB%3D3qvkjmovZ%2BKX_oVQCW788JNT4z04XRv%3D%2BjTJg%40mail.gmail.com.

[Andrew Allen]

Inline

/** ~Andrew Z Allen */

On Thu, Feb 11, 2016 at 10:45 PM, Brian Silverman <bsilve...@gmail.com> wrote:

Hi,

I've got a bit more info about how some of this stuff works.

On Thu, Feb 11, 2016 at 11:19 PM, Austin Schuh <austin...@gmail.com> wrote:

Andrew,

On Thu, Feb 11, 2016 at 8:13 PM Andrew Allen <m...@andrewzallen.com> wrote:

Interesting advice.

/** ~Andrew Z Allen */

On Thu, Feb 11, 2016 at 12:44 PM, Austin Schuh <austin...@gmail.com> wrote:

I've been doing it by using bazel run when I want to update the test data instead of trying to get bazel test to update it.

So bazel run runs in the context of the real repo instead of in the sandbox? Or are you reaching into bazel-out (or wherever the files pop out) and copying them into your repo?

`bazel run` runs in the context of the host, after having built in the context of the sandbox.  The directory that it runs from seems to be the runfiles directory of the binary, FYI.  I use absolute paths when I want to write somewhere so I don't have to think as hard...  You could very likely make relative paths work.

The documentation says "the test's current directory (which is in the runfiles tree)", meaning the CWD for tests is guaranteed to be bazel-bin/package/target.runfiles. That directory tree has enough symlinks in it though that it's probably easier to just use absolute paths.

This doesn't solve your desire to log the output upon test failure though, which seems like a neat idea.

Not sure if this makes sense for what you're doing, but you could print the contents of the new golden file (possibly encoded in some way and with delimiters) from the test and then have a separate script that extracts it from the test log file.

I'm dealing with screenshots so printing them to test output would be pretty big. Do you know if it is possible to split the test output at all? Maybe I could write the image files to the test output folder instead and then just pretend that they are logs. If you squint hard enough, everything is a log, right?

 

 

Austin

On Thu, Feb 11, 2016 at 10:22 AM 'Eric Fellheimer' via bazel-discuss <bazel-...@googlegroups.com> wrote:

Hmm, what about passing in a flag to your test that instructs it to save the file? You could pass in --test_arg=--save_golden_file=<path> ?

To quote the test encyclopedia:

"The root directory observed by a test may or may not be the real root directory. /proc shall be mounted. All build tools shall be present at the absolute paths under /usr used by a local installation. Paths starting with /home may not be available. Tests should not access any such paths. /tmp and /export/hda3/tmp shall be writable, but tests should avoid using these paths. Tests must not assume that any constant path is available for their exclusive use. Tests must not assume that atimes are enabled for any mounted filesystem."

I'm not sure if the sandboxing precludes writing to the normal file system. Based on my understanding of Bazel I think this would work but I'd like to make sure that I'm doing this in a way that won't silently break in the future.

 

Sandboxing should preclude writing to the host.  The namespace is created with a lot of stuff bind mounted read-only.  If that isn't true, it would most likely be considered a bug.

Yeah, in my experimentation I was only able to get it to work by turning off sandboxing for the test.

The root filesystem when running under the sandbox is made up of bind-mounted (read-only) files and directories and a temporary directory Bazel deletes afterwards for the root. Everything which is saved from the sandbox has to be explicitly copied before it gets cleaned up, which basically means only explicit outputs (none for tests) and a fixed set of logs etc for tests.

A fixed set of logs you say.... do you know how I manipulate that set?

I hope this helps!

Very much! Thanks to both of you.

[Ming Zhao]

I think the recommended output location is $TEST_TMPDIR, which is
usually a sub-directory under bazel-$workspacename/_tmp.

> https://groups.google.com/d/msgid/bazel-discuss/CAAVj6hyO1z2W52i9Rm9wJ4LySDzPB%3DSEhken%2BMav2mDNHWeAvA%40mail.gmail.com.

[Brian Silverman]

Inline

On Fri, Feb 12, 2016 at 2:27 AM, Ming Zhao <ming...@gmail.com> wrote:

I think the recommended output location is $TEST_TMPDIR, which is
usually a sub-directory under bazel-$workspacename/_tmp.

Unfortunately, with sandboxing everything there gets deleted with the sandbox... 

Recently, it's the optionalOutputFiles argument to BaseSpawn's constructor which is called from StandaloneTestStrategy. If you want, adding another element to the set which currently only includes resolvedPaths.getXmlOutputPath().relativeTo(execRoot) should be easy. Before 39125b (Jan 18), it was different and harder to follow.

Now that I think about it, having a GOLDEN_OUTPUT_FILE environment variable like XML_OUTPUT_FILE (but which tools won't try to interpret as XML) might be generally useful enough to get accepted as a patch. Any thoughts from the Bazel team on that?

[sa...@carbon3d.com]

Sorry for resurrecting the ancient thread, but is there any update on this issue?

As far as I can tell, there's still no way to preserve test artifacts in bazel, other than running it outside the sandbox. Can we have TEST_UNDECLARED_OUTPUTS_DIR supported in bazel?

[Andrew Allen]

Resurrecting the thread.

It looks like with the submission of dd04adf Bazel is now supports the "undeclared outputs protocol". Is there any documentation on this protocol? If I put files in that directory during a test will it always pop up in a common place? Do I query the Bazel server via RPC to get this information?

/** ~Andrew Z Allen */

[Ben Vanik]

Re-resurrecting. I'd like to be able to get access to the undeclared outputs zip that is being generated:

https://github.com/bazelbuild/bazel/blob/7ed032e7da2245062040e20879400955fe775616/tools/test/test-setup.sh#L262

Is there a way to get this without building a custom test runner that does evil things?

[Ben Vanik]

Oh, nevermind, it looks like this was just added a few days ago and my install doesn't have it :)

https://github.com/bazelbuild/bazel/commit/36d2363f5a9cfd95c96c1af5303de2380e57d14e

FYI for others on the thread; this should be coming soon!