Question about the Performance plugin usage with JUnit

474 views
Skip to first unread message

Balazs Attila-Mihaly (Cd-MaN)

unread,
May 10, 2011, 3:17:20 AM5/10/11
to jenkins...@googlegroups.com, man...@apache.org, ver...@dev.java.net, ae...@dev.java.net
Hello everybody,

I'm trying to use the performance plugin (https://wiki.jenkins-ci.org/display/JENKINS/Performance+Plugin) to log and keep historical stats about some integration tests which are written as JUnit tests. However I can't get it to work. The setup:

- Ubuntu 32 bit, OpenJDK 1.6.0_22
- Jenkins 1.410 from the Jenkins debian repo
- Performance plugin 1.6 installed trough the web interface
- Standard maven project (created with archetype:quickstart) with one unit test
- For the build I've configured the performance plugin with one junit entry and set the "Report files" directory to "target/surefire-reports"

However, for any run I see just zeros in my reports (and max/min long for max/min). It seems to see the individual report files, since it lists them (like "Performance Breakdown by URI: TEST-com.blogspot.hypefree.AppTest.xml"), but doesn't seem to collect the actual timing information from it. I've done a quick debug and it seems to me that the reports are parsed ok during the build, however when the report is displayed, the JMeter report parser is hardcoded, rather than selecting the appropriate parser based on the configuration (PerformanceReportMap, line 62-63).

Did anyone manage to use the Performance plugin with JUnit tests? If so, what am I doing wrong?

Best regards and thank you for your time,
Attila Balazs

David Karlsen

unread,
May 10, 2011, 5:46:03 AM5/10/11
to jenkins...@googlegroups.com
Got the same problem

2011/5/10 Balazs Attila-Mihaly (Cd-MaN) <x_at_...@yahoo.com>



--
--
David J. M. Karlsen - http://www.linkedin.com/in/davidkarlsen

Balazs Attila-Mihaly (Cd-MaN)

unread,
May 10, 2011, 10:21:10 AM5/10/11
to Manuel Carrasco Moñino, jenkins...@googlegroups.com, ver...@dev.java.net, ae...@dev.java.net
Thank you for the reply.    

Those were my initial thoughts also. I've double checked and the "JUnit" type is selected in the build configuration section (under "Publish Performance test result report"). I've also compared my .xml report files with the ones from the unit test (in fact I've created a unit-test which drops the same file present in the performance plugin unit-test in the folder parsed by the plugin and got the same result).

I've also checked the logs and it says that the files were parsed successfully (except the .txt file which I'm not interested in):

Performance: Parsing JMeter report file TEST-com.blogspot.hypefree.AppTest.xml
Performance: Parsing JMeter report file TEST-JUnitResults.xml
Performance: Parsing JMeter report file com.blogspot.hypefree.AppTest.txt
Performance: Failed to parse /var/lib/jenkins/.jenkins/jobs/testY/builds/2011-05-09_17-37-38/performance-reports/com.blogspot.hypefree.AppTest.txt: Content is not allowed in prolog.

I suspect that the problem is around the following lines of the PerformanceReportMap class (this is from the constructor, around lines 62-63):

    if (files != null) {
      addAll(new JMeterParser("").parse(buildAction.getBuild(),
          Arrays.asList(files), listener));
    }

I think that JMeterParser shouldn't be hardcoded here, but rather the appropriate parser should be identified based on the configuration (JUnitParser in this case), however I don't have the necessary expertise with the codebase to implement a fix.

Could you please confirm that this is indeed the issue? If so, I'm happy to log an issue on JIRA, but I don't want to pollute it if the problem is between the chair and keyboard ;-)

Best regards,
Attila Balazs

From: Manuel Carrasco Moñino <man...@apache.org>
To: Balazs Attila-Mihaly (Cd-MaN) <x_at_...@yahoo.com>
Cc: "jenkins...@googlegroups.com" <jenkins...@googlegroups.com>; "ver...@dev.java.net" <ver...@dev.java.net>; "ae...@dev.java.net" <ae...@dev.java.net>
Sent: Tuesday, 10 May 2011, 14:37
Subject: Re: Question about the Performance plugin usage with JUnit

When configuring the plugin for the project you have to select junit parser, did you?, also after running the build you should see a report with the files parsed, did you see that?

May be it is an issue in the junit parser, take a look to your .xml report files and compare then with the files in the test/resources  folder to figure out what is happening.

If after this you identify a bug do either send a patch or open a jira issue.

Cheers
- Manolo

Balazs Attila-Mihaly (Cd-MaN)

unread,
May 11, 2011, 10:19:27 AM5/11/11
to jenkins...@googlegroups.com, Manuel Carrasco Moñino, ver...@dev.java.net, ae...@dev.java.net
Hello.

I did a test and harcoded JUnitParser instead of JMeterParser in PerformanceReportMap, so that it looks like the following (around lines 62-63):

    if (files != null) {
      addAll(new JUnitParser("").parse(buildAction.getBuild(),
          Arrays.asList(files), listener));
    }

And now it works as expected (I can see reports for individual builds and also graphs across builds with trending), however a better fix would be needed which selects the parser based on the configuration. I also create an issue for this problem: https://issues.jenkins-ci.org/browse/JENKINS-9655

Please let me know if there is anything else I can help with to get this issue fixed.

Best regards,
Attila Balazs

From: Balazs Attila-Mihaly (Cd-MaN) <x_at_...@yahoo.com>
To: Manuel Carrasco Moñino <man...@apache.org>
Sent: Tuesday, 10 May 2011, 17:21

Balazs Attila-Mihaly (Cd-MaN)

unread,
May 12, 2011, 10:33:49 AM5/12/11
to jenkins...@googlegroups.com, Manuel Carrasco Moñino, ver...@dev.java.net, ae...@dev.java.net
Hello again,

After more debugging it seems to me that I've found the issue: each parser should put its performance reports in its subdirectory (ie. performance-reports/JUnit, performance-reports/JMeter, etc), with the files in performance-reports directory being interpreted as JMeter (for backward compatibility). However this doesn't seem to be fully implemented in at least two ways:

- the copying of files doesn't respect this pattern (rather it dumps everything into performance-reports)
- when listing the performance reports (ie PerformanceProjectAction.getPerformanceReportList), it doesn't search in subdirectories.

I attached a patch to the Jira issue (https://issues.jenkins-ci.org/browse/JENKINS-9655) which fixes both of these issues, and additionally it changes the subdirectory name from being based on the class name on a string, because the class name can contain special characters (like "$") which might not be well liked by some filesystems. I also found an older issue (https://issues.jenkins-ci.org/browse/JENKINS-5571) which seems to describe the same problem (I linked it to the newer issue).

Please let me know if the diagnosis is correct and if there is anything I can do to further the application of the patch.

Best regards,
Attila Balazs

From: Balazs Attila-Mihaly (Cd-MaN) <x_at_...@yahoo.com>

Subject: Re: Question about the Performance plugin usage with JUnit

Hello.

I did a test and harcoded JUnitParser instead of JMeterParser in PerformanceReportMap, so that it looks like the following (around lines 62-63):

    if (files != null) {

      addAll(new JUnitParser("").parse(buildAction.getBuild(),
          Arrays.asList(files), listener));
    }

And now it works as expected (I can see reports for individual builds and also graphs across builds with trending), however a better fix would be needed which selects the parser based on the configuration. I also create an issue for this problem: https://issues.jenkins-ci.org/browse/JENKINS-9655

Please let me know if there is anything else I can help with to get this issue fixed.

Best regards,
Attila Balazs

From: Balazs Attila-Mihaly (Cd-MaN) <x_at_...@yahoo.com>
To: Manuel Carrasco Moñino <man...@apache.org>
Sent: Tuesday, 10 May 2011, 17:21
Reply all
Reply to author
Forward
0 new messages