junitxml parsing with bamboo and specs2-three

[Alissa Pajer]

Hi,

Upon upgrading to specs2-three, bamboo is no longer able to parse junitxml. Here is the output from the logs:

```
Starting task 'Test Results Parser' of type 'com.atlassian.bamboo.plugins.testresultparser:task.testresultparser.junit'
Parsing test results under /home/myDir
Failed to parse test result file "core/target/test-reports/MySpec1.xml"
Failed to parse test result file "core/target/test-reports/MySpec2.xml"
... // etc
Failing task since test cases were expected but none were found.
Finished task 'Test Results Parser' with result: Failed
```

Did anything change in specs2-three regarding junitxml? In our build.sbt, we're including `testOptions in Test += Tests.Argument("junitxml", "console", "showtimes")` like we had before. And the bamboo plugin is set to look in `**/test-reports/*.xml`. Unfortunately, the logs didn't spit out anything else relevant :/

This isn't necessarily a specs2-three issue, but since this happened for the first time in the PR that upgraded to specs2-three (and made no other changes), I'm thinking that perhaps something changed in specs2-three.

Has anyone else seen this? Ideas for what to try? Thanks.

[etorreborre]

Hi Alissa,

Sorry that's my fault I forgot to port over the Specfication for the JUnit xml output and, as a result, it was not tested and completely off.

You can now try 3.0-20150306133021-ac0f819 which should fix that.

I will publish an official 3.1 next week with the most recent fixes but in the meantime you should be able to rely on those timestamped-git-versioned releases.

Eric

[Alissa Pajer]

Eric, thanks! I'll grab that new version and let you know how it goes.

--
You received this message because you are subscribed to a topic in the Google Groups "specs2-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/specs2-users/eis7NtFT1gc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to specs2-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alissa Pajer]

Ok, bamboo has given the thumbs up! This fixed it, thanks so much for the lightning-fast turnaround!

[etorreborre]

Great, please don't hesitate to ask if you find any other issues or oddities.

This version hasn't be battle-tested at all so I expect that there will be some. I also think that I'll be in a better position to fix them now :-)

Eric.

To unsubscribe from this group and all its topics, send an email to specs2-users+unsubscribe@googlegroups.com.

[Alissa Pajer]

Hi Eric,

Great thanks! :)

So we just had our first test fail in bamboo, and the xml parsing isn't working quite right. Below is the output from the logs. A unit test failed and the junitxml parsing succeeded, so according to the logs, all looks good. But here's the problem:

One way we use the junitxml parsing is to display (in the bamboo web interface) exactly which test has failed. But now instead of showing which test failed, in the bamboo Branch Summary page there is just the message: "No failed tests found, a possible compilation error occurred."

So it seems that the parsing has succeeded (i.e. not errored) but perhaps the xml wasn't parsed correctly, since bamboo wasn't able to read the parsed output in order to display the test.

```
Finished task 'core test' with result: Failed

Starting task 'Test Results Parser' of type 'com.atlassian.bamboo.plugins.testresultparser:task.testresultparser.junit'

Parsing test results under /home/...
Finished task 'Test Results Parser' with result: Success
```

I know this isn't a whole lot of information, so if there's more you need from me (sample xml, screenshots, etc) just let me know :)

Thanks,

Alissa

To unsubscribe from this group and all its topics, send an email to specs2-users...@googlegroups.com.

[etorreborre]

Yes I would like to analyze the xml and understand what's wrong. 

I have no idea on how Bamboo parses that xml and what could be incorrect there.

Probably the best thing would be to create a small failing specification with just one failing test and we could compare the junit xml output between 2.4.17 and 3.0.

Eric.

[etorreborre]

I caught something obvious which my tests didn't, there's no failure message in case of a failure.

The reason is that my specification for the JUnitXmlPrinter uses a fully executed spec while the printer was initialising itself with a non executed one.

This is now fixed in 3.0-20150307124924-d4acb61.

Eric.

[Alissa Pajer]

Ok, so I just finished running the two versions of my simple failing test on bamboo, one with specs2-two and one with specs2-three.

(This is with the release you provided yesterday, not the one you just made :))

The two xml files are quite different. They are attached. The specs2-two file showed one test failure in bamboo, while the specs2-three file showed "No tests found".

I'll try this again running against your new jar and let you know how that works.

Here's the test:

```
object AlissaSpec extends Specification {
  "my spec" should {
    "fail" in {
      true mustEqual false
    }
  }
}
```

To unsubscribe from this group and all its topics, send an email to specs2-users...@googlegroups.com.

[Alissa Pajer]

Oh geez, I was just responding to these is gmail, I had no idea the entire previous conversation was being included. Google should know better... :) Sorry! :)

[Alissa Pajer]

Ok cool, I just noticed that the xml contains the version used for specs2, so we can confirm exactly which xml goes with which specs2 release.

[Alissa Pajer]

Eric, your latest jar worked!!!  I attached the xml for reference. Woohoo! I now have "1 New Failures" displayed in bamboo. :)

That solves all the issues I know of, thank you!

[Alissa Pajer]

Hi Eric,

I just used the lastest version 3.0-20150307124924-d4acb61 on my master branch, and found another issue.

What happened is this: a unit test failed, and but then the parser fails to parse the failed spec. It has no problem parsing the specs that passed.

I attached two xml files. The xml file that succeds is the one that uses 2.4.15, the one that fails using 3.0-20150307124924-d4acb61.

I inspected them for notable differences:

(1) the specs2-two file has <?xml version='1.0' encoding='utf-8'?> as the first line, while the specs2-three file omits that.

(2) the specs2-two file has <system-out></system-out> near the end, while the specs2-three file has <system-out><![CDATA[]]></system-out>

(3) the specs2-two (line 11) file has &lt;init&gt; while the specs2-three (line 69) file has <init>.

Since I've seen this latest specs2 version parse a(nother) failed test with differences (1) and (2), my best guess is that (3) is causing this.

```
Finished task 'core test' with result: Failed

Starting task 'Tests result parser' of type 'com.atlassian.bamboo.plugins.testresultparser:task.testresultparser.junit'

Parsing test results under /home/...

Failed to parse test result file "MySpec.xml"
Finished task 'Tests result parser' with result: Success
```

Thanks!
Alissa

[Alissa Pajer]

I'm working up a fix for this now.

[Alissa Pajer]

Ok, PRs out for escaping four characters: < > " &

https://github.com/etorreborre/specs2/pull/339/commits

This still doesn't add the nice xml declaration, though. I played with this for a bit, just adding it manually, but I think the scalaz-stream file processing may have been eating up the header that I tried to add, but I didn't look too deeply into it. This would be nice to have, but it *shouldn't* be needed for correctness on the bamboo side of things.

We should be able to use scala.xml to nicely add a header, but that's a bit more work to incorporate into the existing file processing and writing workflow :)

[etorreborre]

Thanks again for your PR.

> This still doesn't add the nice xml declaration, though

What was the problem with just adding it at the beginning like this?

This seems to work fine for me.

Eric.

[etorreborre]

I published 3.0-20150307191330-fd443c8 if you want to have a go at it.

[Alissa Pajer]

Aha! I figured out what happened previously when I added the xml declaration and it "disappeared"! It's the reason you refactored `def print` into `def print` and `def printString` :)

scala.xml.XML.loadString eats the declaration! So I was checking for the presence of the declaration after running scala.xml.XML.loadString rather than before, like your code does.

[Alissa Pajer]

Tested the new release. Realized that my < and > characters were in the body of <failure></failure>, and that body was not being escaped.

PR out: https://github.com/etorreborre/specs2/pull/340

[Alissa Pajer]

Update: Release version 3.0.1 has fixed all known issues with junitxml parsing. Yay!