How to run multiple Jenkins Unit tests in parallel?

[Kirill]

Hi noble Jenkins developers,

I am wondering how to make my unit tests (well, they're integration ones in reality, aren't they), based on Jenkins Unit test harness, faster.

I have like 35 tests, and it takes approximately 9 minutes to execute them. I want to reduce this time. I use the default Surefire settings, and if I try to play around with forking options like advised on Surefire page https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html, I run into different problems.

For example, if I set configuration

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.17</version>
    <configuration>
        <systemProperties>
            <property>
                <name>hudson.udp</name>
                <value>33849</value>
            </property>
        </systemProperties>
        <reuseForks>true</reuseForks>
        <forkCount>5</forkCount>
    </configuration>
</plugin>

I get errors like

Caused by: java.nio.file.FileSystemException: C:\Users\me\AppData\Local\Temp\hudson4898380288667948066test\jobs\test0\builds\2015-11-12_14-44-25\log: The process cannot access the file because it is being used by another process.
        at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
        at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
        at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
        at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)

If I use more sophisticated configuration like

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.17</version>
    <configuration>
        <systemProperties>
            <property>
                <name>hudson.udp</name>
                <value>33849</value>
            </property>
        </systemProperties>
        <reuseForks>true</reuseForks>
        <forkCount>2</forkCount>
        <parallel>classes</parallel>
        <threadCount>5</threadCount>
    </configuration>
</plugin>

I get errors like:

testJobSetup(com.myproject.BuildStepConfigurationTest)  Time elapsed: 0.158 sec  <<< ERROR!
java.lang.IllegalStateException: second instance
....

SEVERE: Timer task hudson.model.Queue$MaintainTask@a4da332 failed
java.lang.NullPointerException
        at hudson.model.Queue.maintain(Queue.java:994)
        at hudson.model.Queue$MaintainTask.doRun(Queue.java:2033)
....
testJobSetup2(com.myproject.BuildStepConfigurationTest)  Time elapsed: 0.201 sec  <<< ERROR!
java.lang.NullPointerException: null
        at hudson.ProxyConfiguration.getXmlFile(ProxyConfiguration.java:202)
        at hudson.ProxyConfiguration.load(ProxyConfiguration.java:206)
        at jenkins.model.Jenkins.<init>(Jenkins.java:782)
        at hudson.model.Hudson.<init>(Hudson.java:82)
....

If I run tests with default Surefire settings, it takes about 9 minutes, but at least the tests run fine.

Are there any guidelines about how to make unit tests run as fast as possible? How to tune Surefire, etc.

Thanks in advance!

Regards,
Kirill.

[Jesse Glick]

On Thu, Nov 12, 2015 at 10:07 AM, Kirill <yam...@gmail.com> wrote:
> I have like 35 tests, and it takes approximately 9 minutes to execute them.
> I want to reduce this time. I use the default Surefire settings, and if I
> try to play around with forking options like advised on Surefire page

The default plugin parent POM already configures all this stuff
(correctly AFAIK), and sets concurrency=2 by default. You can raise
that if you need on the command line:

mvn -Dconcurrency=10 test

[Mark Waite]

I'd attempted various experiments with parallel execution of tests for the git client plugin (particularly the credentials tests which tend to be slow and spend much of their time transferring data from external git repositories).  I never found settings that improved performance over the defaults and still allowed the tests to execute successfully.

I tried -Dconcurrency=10 today and found no difference in wall clock time compared to running without that argument.  Does that argument depend on a newer version of Jenkins, or some other more recent developments?  The git client plugin is intentionally based on an older Jenkins version, so it doesn't see the latest Jenkins core changes.

Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr29nsteY65wnSpqXH4Yj%2BxBf1ETGmEk0Hd73AqUDDaFMg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks!

Mark Waite

[Jesse Glick]

On Thu, Nov 12, 2015 at 8:50 PM, Mark Waite <mark.ea...@gmail.com> wrote:
> I tried -Dconcurrency=10 today and found no difference in wall clock time
> compared to running without that argument. Does that argument depend on a
> newer version of Jenkins

AFAIK this is available in 1.535+ parent POMs. Assuming you write
`JenkinsRule` tests rather than unit tests, you should see ten tests
start before any start to complete.

Wall clock time is highly sensitive to your machine: number of
processors, free RAM, HDD vs. SSD, etc. If you are patient you can try
something like

mvn clean test-compile; for c in 10 9 8 7 6 5 4 3 2 1; do echo
"concurrency=$c"; time mvn surefire:test -Dconcurrency=$c
-Dmaven.test.failure.ignore &>-; done

[Mark Waite]

I think my issue may be that I'm using a parameterized unit test that is a JenkinsRule test.  I suspect the parallel may not operate with tests with parameters defined at run time and using a non-default RunWith.

I generally prefer to only use JenkinsRule tests when I can't avoid them, because their setup and teardown time is longer.  Does the parallel also work for tests which don't use JenkinsRule?

Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr3aEOtZK%2B_M3Cxm%3DJ0gMt1Oc%3DWqP0GZp0YtqRQOJ1Eapw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks!

Mark Waite

[Jesse Glick]

Parallelization operates at the granularity of suites, not cases.