[llvm-dev] Automatic Regression Finding Tool
Yann Girsberger via llvm-dev
in the last few months Theodoros (theo...@inf.ethz.ch) and I have
created a tool which automatically finds regressions related to dead
code elimination.
We have collected more than 200 regressions with respect to trunk and
would like to report them.
However, as we do not wish to spam the GitHub issues, we would like to
ask how to best hand over these findings.
An example of how a current issue/report looks like can be found at
https://github.com/llvm/llvm-project/issues/52965 .
Any feedback on the report template is appreciated.
Yours sincerely,
Yann Girsberger
_______________________________________________
LLVM Developers mailing list
llvm...@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
David Spickett via llvm-dev
On the template I had a few ideas.
Could you put a dividing line between each pair of command/output? Or
perhaps just one after the "can not eliminate foo but..." line. Then
it's easier to see that each command and output is paired starting
from that point.
You have the output of -v, what is the use case for that? The most
obvious thing is the target. I primarily work on Arm so it is good to
see what the default target should be but I wonder if there's a better
way to do that. Not saying you have to remove the -v output, I just
wonder if an explicit --target option to clang would give the reader
the same info and help those of us not running on x86. (I'm not
suggesting you go back and redo 200 reports though, maybe for new
ones)
Having both assembly outputs is great. We (Linaro) just added this to
one of our regression loops and a lot of cases can be triaged with
just that.
However the phrasing is a bit imprecise on first read. "Started with"
I think, did the bisect start with this rev or did the problem start
there? After a few seconds you realise of course the latter is the
only thing that makes sense but it could be clearer. Maybe "Regression
started with".
(you can probably spend ages naming that bit. Some people prefer last
good/first bad, sometimes I want to put it the other way around as
before/after)
Would it be possible to attach the test case and the before/after as
an archive to the issue? (I found out recently github doesn't allow .c
files but an archive containing them is fine) Then it's quick to
download and diff locally if it can't be eyeballed.
Last thing, that I'm not sure is possible automatically. Tagging the
user who wrote the patch. Easy enough to do after the fact but perhaps
github has some way to map commit to github username.
Yann Girsberger via llvm-dev
thank you very much for your feedback!
I integrated it and other feedback into new reports.
(https://github.com/llvm/llvm-project/issues/53322)
Is there any interest from Linaro (or the ARM community in general) in
similar regressions for ARM targets?
If you have any further feedback on the reports, please let us know.
Best,
Yann
David Spickett via llvm-dev
Looks great.
> Is there any interest from Linaro (or the ARM community in general) in similar regressions for ARM targets?
We (Linaro) have been doing benchmarking related to code size and
speed using SPEC but nothing like this where you've got different
generated (I assume) examples each time. I've added Maxim on CC who
set up our benchmarking infrastructure and better knows what we test.
Certainly sounds like a good addition and it would be interesting to
find out whether it's detailed enough to catch target specific issues.
Is your approach detailed anywhere? (I think I saw mention of a paper)