Intelligent test report

[Kohsuke Kawaguchi]

Hi, fellow Jenkins folks,

I'm here with what I hope to be a win-win offer, which might sound shady :-), but hear me out!

At Launchable, we are trying to use data from tests to make devs more productive. As a part of that effort, we are actively building what I call "intelligent test session report". The easiest way to think of it is the equivalent of the test report page like this, except it's more intelligent:

• When one issue breaks multiple tests, they get grouped together (by LLM)
• Failures get summarized by LLM to help you quickly get to the most important failure
• Historical information -- new vs ongoing failures
• Failures annotated by attributes of tests, such as "flaky test", "never failing test", etc.

Admittedly, this is still an early product, as there are tons we can/want to do, but I believe it's already usable, and I'm looking for people who are willing to kick tires, and give us suggestions & feedback to improve them. So I naturally thought of the Jenkins project, where I know we have no shortage of devs who have opinions about tools they use!

Would it be OK if I submit the necessary change in the CI pipeline & others to get this feature activated for Jenkins core? This is of course free.

Just to give you some idea of what this would look like, here's one test report. 56 tests failed in this run, but we were able to group them into 3 distinct issues, and the summary makes it clear that two of them are some Spring issues, and the last one is sendgrid (library?) versioning problem.

--

Kohsuke Kawaguchi

[Mark Waite]

Thanks so much for the offer!  I'd love to see the pull request proposing the addition.

We discussed it in board meeting earlier this week and all those in attendance at the meeting agreed that this is a good thing to explore further.

Looking forward to the pull request(s).

Mark Waite