[JIRA] (JENKINS-58430) Publishing two fitnesse xml files with same test name confuses the results

0 views
Skip to first unread message

JohannMoro@gmail.com (JIRA)

unread,
Jul 10, 2019, 11:41:01 AM7/10/19
to jenkinsc...@googlegroups.com
Johann Moro created an issue
 
Jenkins / Bug JENKINS-58430
Publishing two fitnesse xml files with same test name confuses the results
Issue Type: Bug Bug
Assignee: Unassigned
Components: fitnesse-plugin
Created: 2019-07-10 15:40
Labels: plugin
Priority: Major Major
Reporter: Johann Moro
  • In one pipeline job, run the same FitNesse tests and generate 2 xml files. So for instance:
    • FitNesse tests on Linux generates fitnesse_linux.xml
    • FitNesse tests on Windows generates fitnesse_windows.xml
      The way the tests are generated is as follows (same for windows)
      java -jar ./fitnesse-standalone.jar -c "MySuite?suite&format=xml&includehtml" -b fitnesse_windows.xml
      
  • Now in the pipeline job, the tests are published in this way:
    step([$class: 'FitnesseResultsRecorder', fitnessePathToXmlResultsIn: 'fitnesse_*.xml'])
  • Then let's assume that MySuite.TestOne fails on Linux, when I go to see the Details of the failure from the FitNesseReport link in Jenkins, while TestOne shows as failed, the html details I can see from there actually shows the details of the windows run.
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

JohannMoro@gmail.com (JIRA)

unread,
Jul 11, 2019, 5:21:02 AM7/11/19
to jenkinsc...@googlegroups.com
Johann Moro updated an issue
Change By: Johann Moro
Attachment: fitnesse_linux.xml
Attachment: fitnesse_windows.xml

JohannMoro@gmail.com (JIRA)

unread,
Jul 11, 2019, 5:27:03 AM7/11/19
to jenkinsc...@googlegroups.com

JohannMoro@gmail.com (JIRA)

unread,
Jul 11, 2019, 5:27:03 AM7/11/19
to jenkinsc...@googlegroups.com
Johann Moro updated an issue
* In one pipeline job, run the same FitNesse tests and generate 2 xml files. So for instance:
** FitNesse tests on Linux generates fitnesse_linux.xml
** FitNesse tests on Windows generates fitnesse_windows.xml


The way the tests are generated is as follows (same for windows)
{code:java}

java -jar ./fitnesse-standalone.jar -c "MySuite?suite&format=xml&includehtml" -b fitnesse_windows.xml
{code}

* Now in the pipeline job, the tests are published in this way:
{code:java}

step([$class: 'FitnesseResultsRecorder', fitnessePathToXmlResultsIn: 'fitnesse_*.xml']){code}

* Then let's assume that MySuite.TestOne fails on Linux, when I go to see the Details of the failure from the FitNesseReport link in Jenkins, while TestOne shows as failed, the html details I can see from there actually shows the details of the windows run.

----
To reproduce this issue, use the following pipeline. Run it once and them manually copy the two xml files attached into the workspace. Run it again and it should publish the FitNesse results.
{code}
pipeline {
    agent {
        label 'your-agent'
    }
    stages {
        stage ('Repro') {
            steps {
                echo 'Make sure the files are in the Workspace'
            }
        }
    }
    post {
        always {

            step([$class: 'FitnesseResultsRecorder', fitnessePathToXmlResultsIn: 'fitnesse_*.xml'])
        }
    }
}
{code}

Once the results are published, you can see that the fitnesse_windows.xml has been successful while fitnesse_linux.xml has failed. You can see this in the FitNesse report that this is represented correctly:

JohannMoro@gmail.com (JIRA)

unread,
Jul 11, 2019, 5:30:02 AM7/11/19
to jenkinsc...@googlegroups.com
!FitNesseReport.png!
!Details-of-test-not-representing-failure.png.png|thumbnail!
But if I click on the fitnesse_linux.xml and then on the Details link for test _BuysideEnterprise.Ims.MySuite.TestOne_ then it actually shows me the details of the run in fitnesse_windows.xml instead of the linux one. It looks like the details have been overridden.

Therefore, while I see that the test has failed, I cannot see why as it is showing me the incorrect run details:
!Details-of-test-not-representing-failure.png!

JohannMoro@gmail.com (JIRA)

unread,
Jul 11, 2019, 5:30:03 AM7/11/19
to jenkinsc...@googlegroups.com
Johann Moro updated an issue
Change By: Johann Moro
Attachment: Details-of-test-not-representing-failure.png

JohannMoro@gmail.com (JIRA)

unread,
Jul 11, 2019, 5:32:02 AM7/11/19
to jenkinsc...@googlegroups.com
Johann Moro updated an issue
* In one pipeline job, run the same FitNesse tests and generate 2 xml files. So for instance:
** FitNesse tests on Linux generates fitnesse_linux.xml
** FitNesse tests on Windows generates fitnesse_windows.xml
The way the tests are generated is as follows (same for windows)
{code:java}
java -jar ./fitnesse-standalone.jar -c "MySuite?suite&format=xml&includehtml" -b fitnesse_windows.xml
{code}

* Now in the pipeline job, the tests are published in this way:
{code:java}
step([$class: 'FitnesseResultsRecorder', fitnessePathToXmlResultsIn: 'fitnesse_*.xml']){code}

* Then let's assume that MySuite.TestOne fails on Linux, when I go to see the Details of the failure from the FitNesseReport link in Jenkins, while TestOne shows as failed, the html details I can see from there actually shows the details of the windows run.

----
To reproduce this issue, use the following pipeline. Run it once and them manually copy the two xml files attached into the workspace. Run it again and it should publish the FitNesse results. My agent is a linux agent if that makes a difference.
{code}
pipeline {
    agent {
        label 'your-agent'
    }
    stages {
        stage ('Repro') {
            steps {
                echo 'Make sure the files are in the Workspace'
            }
        }
    }
    post {
        always {
            step([$class: 'FitnesseResultsRecorder', fitnessePathToXmlResultsIn: 'fitnesse_*.xml'])
        }
    }
}
{code}

Once the results are published, you can see that the fitnesse_windows.xml has been successful while fitnesse_linux.xml has failed. You can see this in the FitNesse report that this is represented correctly:
!FitNesseReport.png!
!Details-of-test-not-representing-failure.png.png|thumbnail!
But if I click on the fitnesse_linux.xml and then on the Details link for test _BuysideEnterprise.Ims.MySuite.TestOne_ then it actually shows me the details of the run in fitnesse_windows.xml instead of the linux one. It looks like the details have been overridden.

Therefore, while I see that the test has failed, I cannot see why as it is showing me the incorrect run details:
!Details-of-test-not-representing-failure.png!

jaruzafa@gmail.com (JIRA)

unread,
Aug 13, 2019, 6:37:03 PM8/13/19
to jenkinsc...@googlegroups.com
jaruzafa started work on Bug JENKINS-58430
 
Change By: jaruzafa
Status: Open In Progress

JohannMoro@gmail.com (JIRA)

unread,
Aug 16, 2019, 10:23:01 AM8/16/19
to jenkinsc...@googlegroups.com
Johann Moro commented on Bug JENKINS-58430
 
Re: Publishing two fitnesse xml files with same test name confuses the results

Note that this issue probably needs to be reproduced on Linux because JENKINS-58923 will prevent you from reproducing this on Windows.

jaruzafa@gmail.com (JIRA)

unread,
Aug 16, 2019, 6:00:03 PM8/16/19
to jenkinsc...@googlegroups.com
Change By: jaruzafa
Status: In Progress Fixed but Unreleased
Resolution: Fixed

jaruzafa@gmail.com (JIRA)

unread,
Sep 19, 2019, 3:29:02 PM9/19/19
to jenkinsc...@googlegroups.com
 

Fixed on fitnesse-plugin 1.30

Change By: jaruzafa
Status: Fixed but Unreleased Resolved
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages