New Feature: Realtime test results for JUnit

[oliver gondža]

Hi,

I have started working on a feature that would allow users to look at
JUnit Test Results while the build is still running. The Test Result page
is updated with new results as they are produced by underlying build until
they are turned into final Test Results as we know them. As it is an
interesting feature and it is considerably easier to do in core than in a
plugin, I will try to get this pulled into core soon.

Our users found such a feature extremely useful as they work with build
running for hours. This was actually a part of the reason why wildfly team
switched from Jenkins CI to TeamCity (for the entire discussion see [1]).

What do you think about such a feature in core?

[1] http://lists.jboss.org/pipermail/wildfly-dev/2013-July/000413.html

--
oliver

[Larry Shatzer, Jr.]

How does this differ from https://wiki.jenkins-ci.org/display/JENKINS/Test+In+Progress+Plugin?

--
oliver

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[ogondza]

Thanks, never heard about the plugin before. This seems to require assistance from the project being built, right?

I simply rerun JUnitParser to collect report files regardless where did they come from.

Perhaps I'm wrong. I do not have enough time try it out.

--

oliver

[Larry Shatzer, Jr.]

I've not used the plugin, since it does require some extra setup on the project side, such as extra jar, and tests setup with a specific runner, or something like that. You might want to contact the plugin maintainer for more help or information.

-- Larry

--

oliver

--

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

[Kevin Fleming (BLOOMBERG/ 731 LEXIN)]

Re-executing the JUnitParser is going to be very expensive for builds with a large number of unit tests. When I've considered this sort of thing in the past, I was pretty certain that it would require 'help' from the project itself, so that the parser can be made aware of newly-available test results. Otherwise, you're basically polling, which we all know is evil :-)

[Roman]

Hi,
Could you please describe more how do you do it?
I use junit results for non-java tests and and want to see realtime
results feature but Test+In+Progress+Plugin oriented on java events.

Thanks,
Roman

> --
> You received this message because you are subscribed to the Google
> Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to jenkinsci-de...@googlegroups.com.

[Kohsuke Kawaguchi]

I think this is great! It's certainly a long awaited feature.

I agree with Kevin that this would likely eventually require some
changes in JUnitParser for efficiency. But I think it's good thing to do.

What do you think about spending an office hour on topic so that we can
hash out challenges and see if we can rally some people around this effort?

Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/
Try Jenkins Enterprise, our professional version of Jenkins

[ogondza]

Thanks for the feedback. I guess we agreed that it would be an interesting feature for core provided we can implement it in a reasonably efficient way.

@kpfleming: We can do better than re-runing JUnitResultArchiver. I have modified the parser to update result object incrementally in order _not_ to parse the same report files over and over again. My draft invokes parsing JIT (but it is blocking the UI thread which probably could be eliminated using ajax) so there is no parsing as long as no one wants to see the results. Possible improvement we might be interested in is spinning a thread on slave to monitor new report files and push updates to master when there is something new (no polling on master, no polling over network). I did not find a way to eliminate polling altogether preserving all the flexibility JUnitResultArchiver provides, though.

Testing on remote slave is actually faster than I expected. Refreshing test results takes ~5 seconds when building jenkins.

@Roman: This filesystem based approach will cover your use case as well.

Here is what I have: https://github.com/jenkinsci/jenkins/pull/904

[Jesse Glick]

On Mon, Aug 12, 2013 at 6:58 AM, ogondza <ogo...@gmail.com> wrote:
> it would be an interesting feature for core

Definitely it would, though for something this complex I cannot help
but ask if we should not first make a real effort to split JUnit
reporting out of core into a proper plugin.

> spinning a thread on slave to monitor new report files

Or use java.nio.file.WatchService when available. Most practical when
the report wildcard contains just a single wildcard in the final path
segment, i.e. ‘some-dir/TEST-*.xml’ can be easily monitored by
watching some-dir; in the general case (e.g. ‘**/TEST-*.xml’) you
might need to watch the whole workspace, which is inefficient using
the current API.

[Roman]

It will be great.
Could I help in achieving this?
Thanks,
Roman

>
> Here is what I have: https://github.com/jenkinsci/jenkins/pull/904
>

[ogondza]

I am releasing this plugin to deliver the feature early. In the meantime, someone might find it useful.

https://wiki.jenkins-ci.org/display/JENKINS/JUnit+Realtime+Test+Reporter+Plugin

[ogondza]

Hi, it's been a while but finally I have something that seems to work for freestyle, maven and matrix projects. I am working on report aggregation so it is sill a work in progress. But the most important part is done: asynchronous test result updating for individual builds.

tl;dr: Every build type provides it's implementation of updater thread that is started early in build and terminated after the build is finished. The thread poll the filesystem, parse new reports found and send it to master if there are any. Reports are merged on master.

This feature is on by default for maven jobs but it needs to be triggered explicitly ("Report test result in realtime") for freestyle and matrix jobs. This inconsistency might be in line with the philosophy of corresponding job types, I am not sure, but it was the easiest thing to do.

Any feedback welcome. https://github.com/jenkinsci/jenkins/pull/904

--

oliver

[ogondza]

I consider this pull request feature complete. If interested, try it out:

https://jenkins.ci.cloudbees.com/job/core/job/jenkins_main_trunk/1247/artifact/war/target/jenkins.war

--

oliver