[JVM] Cucumber scenario execution time reported in Jenkins CI

[C0deAttack]

I'm not sure if this is Cucumber-JVM or Jenkins related but I thought I would start here first.

We have our Cucumber scenarios executing in our Jenkins CI job. There seems to be a problem with the duration reported for each scenario in the Test Results table of the Jenkins job.

With duration values of "1 day 12 hr", "7hr 52 min" etc something is clearly incorrect.

Has anyone come across this?

[aslak hellesoy]

IIRC the JSON report reports execution time in nanoseconds (and not
milliseconds).

"1 day 12 hr" is 129600000000000 nanos is 129600000000 micros 129600000 millis.

How long did your your "1 day 12 hr" build _actually_ take?
If it was sub-second it's an indication that jenkins interprets the
nanosecond value as millisenconds.
If it was approx. 2 minutes it's an indication that it's interpreted
as microseconds (more likely).

The reason I picked nanosecond (over the more common millisecond unit)
is that the JVM is able to report duration at this level and I thought
it might be useful. Maybe not.

We have two options to fix the JSON formatter to avoid this kind of
unit confusion:

1) Rename the duration field to duration_nanos
2) Add an additional field called "unit" which can take the values ns, ms or µs.

Thoughts?

Aslak

> -- Rules --
>
> 1) Please prefix the subject with [Ruby], [JVM] or [JS].
> 2) Please use interleaved answers
> http://en.wikipedia.org/wiki/Posting_style#Interleaved_style
> 3) If you have a question, don't reply to an existing message. Start a new
> topic instead.
>
> You received this message because you are subscribed to the Google Groups
> Cukes group. To post to this group, send email to cu...@googlegroups.com. To
> unsubscribe from this group, send email to
> cukes+un...@googlegroups.com. For more options, visit this group at
> https://groups.google.com/d/forum/cukes?hl=en

[Kingsley Hendrickse]

Yes its nanoseconds - in the cucumber-reports jenkins plugin a few people have commented that the report it generates reports 0ms for the duration. In the jenkins plugin I convert the nanos to a jodatime object like this:

public static String formatDuration(Long duration) {
        PeriodFormatter formatter = new PeriodFormatterBuilder()
                .appendDays()
                .appendSuffix(" day", " days")
                .appendSeparator(" and ")
                .appendHours()
                .appendSuffix(" hour", " hours")
                .appendSeparator(" and ")
                .appendMinutes()
                .appendSuffix(" min", " mins")
                .appendSeparator(" and ")
                .appendSeconds()
                .appendSuffix(" sec", " secs")
                .appendSeparator(" and ")
                .appendMillis()
                .appendSuffix(" ms", " ms")
                .toFormatter();
        return formatter.print(new Period(0, duration / 1000000));


    }

so if you are using my jenkins plugin from here:  https://github.com/masterthought/jenkins-cucumber-jvm-reports-plugin-java

Then maybe I'm not doing the right thing with that nano duration.

I also noticed on regular cucumber the json that gets output seems broken for scenario outlines - there is not result section for scenario outlines.

If you are using a different jenkins cucumber plugin - then ignore what I just said !!

--K

[C0deAttack]

A little more context, sorry I should have probably mentioned this in the original post.

We're executing the Cucumber scenarios from the command line, and choosing the report format to be JUnit. The Jenkins build publishes the report.

The Build time was reported at "56mins" - which sounds about right.

But the duration for the whole test suite is reported as "1 mo 0 days". I'm waiting on one the build engineers to send me the JUnit Report XML so I can see what is in there.

What you've said above about the JSON formatter, would that still apply for the JUnit one?

[aslak hellesoy]

The JUnit formatter divides reports durations in milliseconds (integer
string in the XML). I think this might be the culprit. I wonder if it
should report them as seconds (double string in the XML).

Can someone verify this?

Aslak

[C0deAttack]

I got the XML from our build guy.

The value is in seconds. The report has "time=17454" which Jenkins reports as "4 hr 50 min".

I also just saw this post: https://groups.google.com/forum/?fromgroups#!topic/cukes/s0SBdi31uG0 I didn't find a bug report on GitHub so I will create one now.

[C0deAttack]

Issue ticket: https://github.com/cucumber/cucumber-jvm/issues/347