cucumber-json-to-junit command line tool

[aslak hellesoy]

Folks,

Both Cucumber-Ruby and Cucumber-JVM have junit formatters that output
a JUnit XML file. Cucumber-JS can't yet, but has a ticket for it.

The junit formatters are quite buggy, and there are several open
tickets. Rather than trying to fix them all I'm proposing an
alternative solution:

1) Delete the junit formatters
2) Implement a command line tool (cucumber-json-to-junit) that can
translate a cucumber JSON report to JUnit XML.

The JSON formatter is actually implemented in the gherkin library, and
it's consistent across implementations.

cucumber-json-to-junit would read JSON from STDIN and write an XML to STDOUT:

# This can be run by a build script after running cucumber
cucumber-json-to-junit < cucumber-report.json > cucumber-report.xml

It can be implemented in any language (Java, Ruby, JavaScript, Go, C, ...)
The most important is that it is easy to install and run. It will be
used by the people using Cucumber-Ruby, Cucumber-Js and Cucumber-JVM,
so ideally it would be a cross-platform self-contained executable.

What are your thoughts on this? Any takers?

Aslak

[Petter Måhlén]

I could potentially spend a little time on this, but as I mentioned in
one ticket, I'm not sure that I think the option of only having
command-line support is perfect. The reason is that for Maven and Java
users (of which I think there are a lot), the fact Cucumber-JUnit
automatically generates junit-formatted output means that you don't
have to do anything beyond including cucumber-junit as a test
dependency in your build to get the test results integrated into
reports, etc. If JUnit XML is only generated after you run a command
line script, then every Maven user would have to add specific
configuration to their build files, which goes against the grain of
Maven's convention over configuration. And even without Maven, it
doesn't feel right to run via JUnit and not generate JUnit output.

I'd say that it might be a good idea to create a correct
implementation of JUnit output in Cucumber-JUnit, and then wrap that
in a command line tool that can be used by others. It seems like that
might be the best of both worlds. It might be better to create a
separate support library rather than have the code directly in
Cucumber-JUnit.

[aslak hellesoy]

On Thu, May 17, 2012 at 2:34 PM, Petter Måhlén <petter...@gmail.com> wrote:
> I could potentially spend a little time on this, but as I mentioned in
> one ticket, I'm not sure that I think the option of only having
> command-line support is perfect. The reason is that for Maven and Java
> users (of which I think there are a lot), the fact Cucumber-JUnit
> automatically generates junit-formatted output

It's not quite like that. You're mixing up two different JUnit thingies here.

All the Cucumber-JUnit module does is to let you _run_ Cucumber from
JUnit via a @RunWith(Cucumber.class) annotation.
Using Cucumber-JUnit doesn't automatically give you a JUnit XML
report. You have to specify format="junit:some/dir" in your annotation
to tell Cucumber where to write the report.

If you don't specify the junit output format and you're running via
Maven, you'll still get a JUnit XML report, but this is generated by
Maven and not Cucumber. And it doesn't contain all the details [#171]

The JUnit XML report generation (JUnitFormatter) is implemented in the
Cucumber-Core module. It can be used with any of the backends -
Cucumber-Java, Cucumber-Jython etc.
For example, a project that doesn't have Cucumber-JUnit or JUnit on
the classpath can still generate a JUnit XML report by passing the
--format junit command line option.

[#171] https://github.com/cucumber/cucumber-jvm/issues/171

> means that you don't
> have to do anything beyond including cucumber-junit as a test
> dependency in your build to get the test results integrated into
> reports, etc. If JUnit XML is only generated after you run a command
> line script, then every Maven user would have to add specific
> configuration to their build files, which goes against the grain of
> Maven's convention over configuration. And even without Maven, it
> doesn't feel right to run via JUnit and not generate JUnit output.
>

This can be easily solved.

If we create a command line tool to convert JSON to XML it would be
trivial to wrap it in a junit formatter that would simply use the JSON
formatter to generate a temporary JSON report (in memory) and spawn a
child process in the end feeding that JSON to the command line tool.
Transparent for the user.

Aslak

> -- There are two rules:
>
> 1) Please prefix the subject with [Ruby], [JVM] or [JS]. This allows people to filter messages.
> 2) Please use interleaved answers http://en.wikipedia.org/wiki/Posting_style#Interleaved_style
>
> You received this message because you are subscribed to the Google Groups Cukes group. To post to this group, send email to cu...@googlegroups.com. To unsubscribe from this group, send email to cukes+un...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/cukes?hl=en

[chris young]

On May 17, 3:54 pm, aslak hellesoy <aslak.helle...@gmail.com> wrote:

> On Thu, May 17, 2012 at 2:34 PM, Petter Måhlén <pettermah...@gmail.com> wrote:
> > I could potentially spend a little time on this, but as I mentioned in
> > one ticket, I'm not sure that I think the option of only having
> > command-line support is perfect. The reason is that for Maven and Java
> > users (of which I think there are a lot), the fact Cucumber-JUnit
> > automatically generates junit-formatted output

I'm using Gradle to build and test a project that includes Cucumber.js
features.

Gradle is being run from Jenkins and I'm looking for a way to get the
Cucumber.js results into Jenkins.

If Cucumber.js could output its results as JSON then it would just
take a little bit of mustache.js to turn it into JUnit XML which
Jenkins could pick up.

I really like the idea of using JSON as the output format and then
transforming it to XML / HTML etc downstream. It's a very UNIX way of
thinking about the problem - i.e. lots of little tools strung together.