Machine parseable error messages

[Eric Dobson]

What is the recommended way for rules to export information about failures such that downstream tools can include them in UIs.

Example use case:

I ran bazel test //my:target, and one of the actions for //my:target fails because there is an unknown variable "usrname" in my/target.foo at line 7 column 10. It would also like to report that "username" is a valid variable and this is a possible misspelling. And thus wants to suggest an addition of an "e" character.

One way I have thought to do this is to have a separate file that my action produces //my:target.errors that is in a separate output group and have it write machine parseable data there in addition to human readable data on stdout.

I can then find all of these files and parse the data in them in downstream tools.

Is there any prior work on this, or does everything just try to parse the human readable output?

[László Csomor]

Hey, thanks for your email!

We recommend posting Bazel related questions on StackOverflow using the 'bazel' tag. We will be happy to answer it there, so that other users can also benefit. I forwarded your question to: https://stackoverflow.com/questions/44494363/machine-parseable-error-messages

Cheers,

Laszlo

--
László Csomor | Software Engineer | laszlo...@google.com

Google Germany GmbH | Erika-Mann-Str. 33 | 80636 München | Germany
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/9af0a3d3-daa8-4f5d-adc6-3ccd84b845f2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Eric Dobson]

I cannot reply there as it is not my post and I'm not a regular stack overflow user, and thus don't have the 'privilege' of participating in the discussion.

Should I be starting a new post so that the discussion can continue over on that one instead? 

Even if so to avoid round trip delays:

I recommend running the error checkers as extra actions.

These aren't special tools that are producing the errors, this is the compiler which is already using standard out to provide this information in a human readable form. What benefit does using an extra action have over a separate output of the same rule? Also why an extra action versus an aspect?

 

I don't think Bazel currently has hooks for custom error handlers like you describe. Please consider opening a feature request: https://github.com/bazelbuild/bazel/issues/new

I'm not asking for bazel to parse the machine readable format just that it passes it along to other tools. Thus I see no need for bazel to have any support for this (in particular because I want to control the schema of the machine parseable parts), but instead I'm asking about convention and possible prior art.

 

On Monday, June 12, 2017 at 1:07:52 AM UTC-7, László Csomor wrote:

Hey, thanks for your email!

We recommend posting Bazel related questions on StackOverflow using the 'bazel' tag. We will be happy to answer it there, so that other users can also benefit. I forwarded your question to: https://stackoverflow.com/questions/44494363/machine-parseable-error-messages

Cheers,

Laszlo

--
László Csomor | Software Engineer | laszlo...@google.com

Google Germany GmbH | Erika-Mann-Str. 33 | 80636 München | Germany
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle

On Sat, Jun 10, 2017 at 6:27 AM, Eric Dobson <eric.n...@gmail.com> wrote:

What is the recommended way for rules to export information about failures such that downstream tools can include them in UIs.

Example use case:

I ran bazel test //my:target, and one of the actions for //my:target fails because there is an unknown variable "usrname" in my/target.foo at line 7 column 10. It would also like to report that "username" is a valid variable and this is a possible misspelling. And thus wants to suggest an addition of an "e" character.

One way I have thought to do this is to have a separate file that my action produces //my:target.errors that is in a separate output group and have it write machine parseable data there in addition to human readable data on stdout.

I can then find all of these files and parse the data in them in downstream tools.

Is there any prior work on this, or does everything just try to parse the human readable output?

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

[László Csomor]

I cannot reply there as it is not my post and I'm not a regular stack overflow user, and thus don't have the 'privilege' of participating in the discussion.

Sorry about that, I didn't anticipate that problem.

Should I be starting a new post so that the discussion can continue over on that one instead? 

No, it's fine, the SO question links to this thread.

These aren't special tools that are producing the errors, this is the compiler which is already using standard out to provide this information in a human readable form. What benefit does using an extra action have over a separate output of the same rule?

That the separate output becomes part of the rule's or the action's signature, affecting everyone who uses that rule, not just you. That's fine if we're talking about a custom rule that you're implementing now, but it requires consideration if you want to do this for existing rules, e.g. cc_binary. But given that you control the schema, I assume you also control the compiler and the rule implementation too. Is that the case?

 

Also why an extra action versus an aspect?

Nothing in particular, I simply thought of extra actions first. I suppose you can achieve the same with aspects. 

Example use case:

I ran bazel test //my:target, and one of the actions for //my:target fails because there is an unknown variable "usrname" in my/target.foo at line 7 column 10. It would also like to report that "username" is a valid variable and this is a possible misspelling. And thus wants to suggest an addition of an "e" character.

This particular example is tricky -- test actions produce just a test.log (or one log per shard) and an exit code, I don't know of prior art where they'd produce additional outputs that subsequent actions consume. That's more the business of build actions.

You could create an error-checking action that depends on the compilation action, consuming the compiler-generated error report (or reports, from the transitive closure), and produces a dummy output that the downstream test action depends on. If the report file was empty, the dummy file is created, if not then it isn't and the report-consuming action can print the error message about the mistyped variable name or what have you, and the test won't even build.

 

I'm not asking for bazel to parse the machine readable format just that it passes it along to other tools. Thus I see no need for bazel to have any support for this (in particular because I want to control the schema of the machine parseable parts), but instead I'm asking about convention and possible prior art.

You can express the machine-parseable error file as just another compiler output, so it can be an output of the compilation action, as I just described. You can propagate that file to downstream rules using providers or aspects. Is this what you had in mind?

--
László Csomor | Software Engineer | laszlo...@google.com

Google Germany GmbH | Erika-Mann-Str. 33 | 80636 München | Germany
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/93c6cbfe-b23c-451c-90eb-4e426ee903f4%40googlegroups.com.

[Eric Dobson]

On Mon, Jun 12, 2017 at 8:18 AM, László Csomor <laszlo...@google.com> wrote:

I cannot reply there as it is not my post and I'm not a regular stack overflow user, and thus don't have the 'privilege' of participating in the discussion.

Sorry about that, I didn't anticipate that problem.

Should I be starting a new post so that the discussion can continue over on that one instead? 

No, it's fine, the SO question links to this thread.

These aren't special tools that are producing the errors, this is the compiler which is already using standard out to provide this information in a human readable form. What benefit does using an extra action have over a separate output of the same rule?

That the separate output becomes part of the rule's or the action's signature, affecting everyone who uses that rule, not just you. That's fine if we're talking about a custom rule that you're implementing now, but it requires consideration if you want to do this for existing rules, e.g. cc_binary. But given that you control the schema, I assume you also control the compiler and the rule implementation too. Is that the case?

 

Yep, in this case I control everything. Aspects seem like they would work well to add the behavior to existing rules, and I could see reasons to keep even the ones that could be baked in as aspects for consistency. One issue is that this requires running the compiler twice, but given that it is only the frontend it might be reasonable (my language doesn't have the c/c++ monstrosity of textual inclusion).

 

 

Also why an extra action versus an aspect?

Nothing in particular, I simply thought of extra actions first. I suppose you can achieve the same with aspects. 

Example use case:

I ran bazel test //my:target, and one of the actions for //my:target fails because there is an unknown variable "usrname" in my/target.foo at line 7 column 10. It would also like to report that "username" is a valid variable and this is a possible misspelling. And thus wants to suggest an addition of an "e" character.

This particular example is tricky -- test actions produce just a test.log (or one log per shard) and an exit code, I don't know of prior art where they'd produce additional outputs that subsequent actions consume. That's more the business of build actions.

Using bazel test was a bad example here, I care about the build step not the test step.

 

You could create an error-checking action that depends on the compilation action, consuming the compiler-generated error report (or reports, from the transitive closure), and produces a dummy output that the downstream test action depends on. If the report file was empty, the dummy file is created, if not then it isn't and the report-consuming action can print the error message about the mistyped variable name or what have you, and the test won't even build. 

 

I'm not asking for bazel to parse the machine readable format just that it passes it along to other tools. Thus I see no need for bazel to have any support for this (in particular because I want to control the schema of the machine parseable parts), but instead I'm asking about convention and possible prior art.

You can express the machine-parseable error file as just another compiler output, so it can be an output of the compilation action, as I just described. You can propagate that file to downstream rules using providers or aspects. Is this what you had in mind?

That was my plan. One issue that I have had so far with this is that there is no good way (that I know of) to figure out what output files were actually produced by a build. I'm hoping the Build Event Protocol will solve this, but it is currently lacking great documentation (https://github.com/bazelbuild/bazel/issues/2968).

Thanks for the help.

[László Csomor]

Sounds good! You're welcome.

--
László Csomor | Software Engineer | laszlo...@google.com

Google Germany GmbH | Erika-Mann-Str. 33 | 80636 München | Germany
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle