How does Jenkins test harness run the tests?

[Kirill]

Hi,

I use Jenkins test harness (jenkins-test-harness) v 1.551 to test my Jenkins plugin.
I'm wondering if there are any guidelines or "best practices" for writing the tests - I haven't found anything better than https://wiki.jenkins-ci.org/display/JENKINS/Unit+Test.

My problem is that tests hang or get interrupted occasionally. Sometimes the tests pass (more often), sometimes they fail with thread interruption or they hang.

Few questions:
1) how does Jenkins JUnit rule organize tests if I run tests via Surefire Maven plugin without any custom settings (so it's 1 JVM and threads are reused)?
2) is Jenkins instance shared by all the tests? I have a few tests that delete all the projects and then create the project with the same name - MY_PROJECT on set up (annotated by @Before). However, sometimes tests fail complaining that MY_PROJECT already exists - but quite rarely, probably 1 in 10 times. It shouldn't have happened if tests would have different Jenkins instances. But if tests would share Jenkins, it would happen every time, not occasionally. I'm stunned.
2) how many threads are started by Jenkins rule? Probably only one; when I run the test, I am able to open the Jenkins instance in browser only if I put Thread.sleep() in test code. Is this correct?
3) how can I find out which test hangs and where?
4) why does Jenkins interrupt tests sometimes?
5) is jenkins-test-harness a good thing to use for integration tests, or there are better alternatives? Is https://github.com/jenkinsci/acceptance-test-harness a newer and better alternative?

I volunteer for documentation update if I understand how this thing works under the hood :)

Regards,
Kirill.

[oliver gondža]

Hi, see my replies inline.

The jenkins is running only for the time of the test so if you want to
open it in actual browser and look around you will probably need to pause
the test thread. Of course you can use WebDriver to simulate browser like
interaction in automated test.

> 3) how can I find out which test hangs and where?

Inspect the state of running threads using jstack.

> 4) why does Jenkins interrupt tests sometimes?

Sounds like it times out, see `JenkinsRule#timeout`. What is the exact
exception?

> 5) is jenkins-test-harness a good thing to use for integration tests, or
> there are better alternatives? Is
> https://github.com/jenkinsci/acceptance-test-harness a newer and better
> alternative?

I do not dare to say it replaces jenkins-test-harness. I provides more
end-to-end approach to testing. Test and Jenkins run in different JVM,
selenium + actual browser is used to do most of the interactions, you have
better control over the version of Jenkins and plugins under test, etc.

--
oliver

[Jesse Glick]

On Thu, Feb 19, 2015 at 11:36 AM, Kirill <yam...@gmail.com> wrote:
> how does Jenkins JUnit rule organize tests if I run tests via Surefire
> Maven plugin without any custom settings (so it's 1 JVM and threads are
> reused)?

Depending on the parent POM version, there may be tests run in
parallel, but never >1 per JVM.

> is Jenkins instance shared by all the tests?

No. Each test case gets its own Jenkins.instance, $JENKINS_HOME, HTTP port, etc.

(Unless you make JenkinsRule a @ClassRule rather than a @Rule. This is
rarely done, but it is an option if you are concerned about overhead
and none of the test cases within this suite actually change any
settings or create any items, but merely rely on Jenkins services
being loaded.)

> I have a few tests that delete all the projects

No need to delete projects unless they were created earlier in the
same test case (@Test method).

> then create the project with the same name -
> MY_PROJECT on set up (annotated by @Before). However, sometimes tests fail
> complaining that MY_PROJECT already exists - but quite rarely, probably 1 in
> 10 times.

I have never seen such a thing happen. Perhaps your test is doing
something unusual, or your plugin has an odd flaw.

[Kirill]

Thanks a million, Jesse!

Your suggestion about JenkinsRule being a @ClassRule rather than a @Rule was a shot in the bulls eye!
I changed @ClassRule to @Rule and it works fine!

Regards,
Kirill.

пятница, 20 февраля 2015 г., 20:40:23 UTC пользователь Jesse Glick написал:

[Jesse Glick]

On Tue, Mar 17, 2015 at 11:52 AM, Kirill <yam...@gmail.com> wrote:
> I changed @ClassRule to @Rule and it works fine!

I added a tip to https://wiki.jenkins-ci.org/display/JENKINS/Unit+Test
for the next person.