Random failure in SonarQube plugin tests (using JenkinsRule)

[Julien HENRY]

Hi guys,

This is an issue I'm facing for a long time now, without having managed to find the root cause.

Some of our plugin tests are randomly failing. Our test suite is using the Jenkins test harness, and I can't manage to find the root cause.

Only a few tests among all the test suite are affected.

For example this simple one:

https://github.com/SonarSource/sonar-scanner-jenkins/blob/master/src/test/java/hudson/plugins/sonar/MigrationTest.java

hudson.plugins.sonar.MigrationTest.testShouldMigrateDatabasePasswords(hudson.plugins.sonar.MigrationTest)
 Run 1: MigrationTest.testShouldMigrateDatabasePasswords:56 NullPointer
 Run 2: MigrationTest.testShouldMigrateDatabasePasswords:56 NullPointer
 Run 3: MigrationTest.testShouldMigrateDatabasePasswords:56 NullPointer
 Run 4: MigrationTest.testShouldMigrateDatabasePasswords:56 NullPointer
 Run 5: MigrationTest.testShouldMigrateDatabasePasswords:56 NullPointer

It look like the plugin under test is sometimes not properly loaded in the Jenkins instance.

I can even reproduce by running this test alone:

mvn clean verify -Dtest=MigrationTest

After a few runs, I get:

Running hudson.plugins.sonar.MigrationTest
Feb 27, 2017 12:16:29 PM org.eclipse.jetty.util.log.Log initialized
INFO: Logging initialized @355ms
Exploding /home/julien/.m2/repository/org/jenkins-ci/main/jenkins-war/2.32.2/jenkins-war-2.32.2-war-for-test.jar into /home/julien/Prog/Projects/jenkins-sonar-plugin/target/jenkins-for-test
Feb 27, 2017 12:16:31 PM org.eclipse.jetty.server.Server doStart
INFO: jetty-9.2.12.v20150709
Feb 27, 2017 12:16:31 PM org.eclipse.jetty.webapp.StandardDescriptorProcessor visitServlet
INFO: NO JSP Support for /jenkins, did not find org.eclipse.jetty.jsp.JettyJspServlet
Feb 27, 2017 12:16:31 PM org.eclipse.jetty.server.handler.ContextHandler doStart
INFO: Started o.e.j.w.WebAppContext@3a3e78f{/jenkins,file:/home/julien/Prog/Projects/jenkins-sonar-plugin/target/jenkins-for-test/,AVAILABLE}{/home/julien/Prog/Projects/jenkins-sonar-plugin/target/jenkins-for-test}
Feb 27, 2017 12:16:31 PM org.eclipse.jetty.server.AbstractConnector doStart
INFO: Started ServerConnector@7ca33c24{HTTP/1.1}{localhost:39063}
Feb 27, 2017 12:16:31 PM org.eclipse.jetty.server.Server doStart
INFO: Started @1903ms
Feb 27, 2017 12:16:31 PM org.jvnet.hudson.test.JenkinsRule createWebServer
INFO: Running on http://localhost:39063/jenkins/
Loading $JENKINS_HOME from /home/julien/Prog/Projects/jenkins-sonar-plugin/target/test-classes/hudson/plugins/sonar/MigrationTest
Feb 27, 2017 12:16:31 PM jenkins.InitReactorRunner$1 onAttained
INFO: Started initialization
Feb 27, 2017 12:16:35 PM jenkins.InitReactorRunner$1 onAttained
INFO: Listed all plugins
Feb 27, 2017 12:16:35 PM jenkins.bouncycastle.api.SecurityProviderInitializer addSecurityProvider
INFO: Initializing Bouncy Castle security provider.
Feb 27, 2017 12:16:35 PM jenkins.bouncycastle.api.SecurityProviderInitializer addSecurityProvider
INFO: Bouncy Castle security provider initialized.
Feb 27, 2017 12:16:35 PM jenkins.InitReactorRunner$1 onTaskFailed
SEVERE: Failed Initializing plugin sonar
java.lang.NullPointerException
       at hudson.plugins.sonar.SonarPlugin.postInitialize(SonarPlugin.java:48)
       at hudson.PluginManager$2$1$2.run(PluginManager.java:542)
       at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
       at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
       at jenkins.model.Jenkins$7.runTask(Jenkins.java:1085)
       at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
       at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       at java.lang.Thread.run(Thread.java:745)

Any help would be much appreciated.

Julien