Does PIT work with jmockit ?

[paul_...@mentor.com]

Hi,
I've just started to look at mutation testing to see if it could be
used to improve the quality of our testing. Some of our newer tests
use jmockit as a mock testing framework.
Now I've been able to get PIT to work with our non-jmockit test cases
and the results look promising, but any test that uses a mock object
fails with a null pointer exception at the point where a method is
called on the mock. This implies that the mock framework is not
getting initialized.
Has anyone else tried to use jmockit, did it work for them?

I know that jmockit won't work with Jumble, from personal experience
and new groups, but I had hoped PIT would work.

[henry]

Hi Paul,

Unfortunately PIT does not currently work with JMockit. Of the mocking
frameworks it's been tested with (JMock, EasyMock, Mockito, PowerMock
and JMockit) it's the only one with known issues.

This will be looked at, but I don't think it is likely be fixed in the
next couple of releases.

If you're not too heavily invested in JMockit you could consider
converting the tests to Powermock which works nicely with PIT, and
provides pretty similar capabilities to JMockit. The only other
alternative at the moment is to exclude the JMockit tests using a
excludedClasses glob.

Henry

[henry]

I've just taken a second look at this and it may be possible to enable
JMockit support a little more easily than I'd originally thought.

More investigation is required, but I'll see if this can be fixed for
the next release.

Henry

[henry]

I've just uploaded a 0.24 snapshot to the sonatype repository
(https://oss.sonatype.org/content/repositories/snapshots/org/pitest/)
with JMockit support.

Feedback would be helpful from both JMockit and non-JMockit users of any
issues encountered with it or, just as importantly, trouble free runs.

Tests are in place as part of the build to confirm that JMockit works,
but its API is quite large and only a fraction of it is covered by the
tests. In theory if one part of the API works then all of it will, but
practice may be different.

In order to support JMockit I've ripped out a layer of overly complex
code that detected and split up JUnit tests, so things now work a little
differently behind the scenes compared to previous releases.

There's a change of behaviour in the snapshot that will be fixed before
the 0.24 release goes out - previously PIT treated tests with
BeforeClass or AfterClass annotations as atomic units. The snapshot does
not do this and will split out individual tests wrapped in the before
and after methods.

Henry

[Abbott, Paul]

That would be great if it was possible. We are not using jMocket in a large number of tests, but there is a push to write more unit tests which are likely to use JMocket. So going forward this is likely to become a bigger issue.

BTW: Has any tried to use PIT with UISpec4J, we are also starting to use this in our tests.

Paul.

[henry]

I'm afraid I don't know if anyone has tried PIT with UISpec4J, but I
don't see any particular reason to expect issues.

I've just uploaded a new snapshot to the sonatype repository that
restores the BeforeClass and AfterClass behaviour.

If you can confirm that this resolves your issues with JMockit this will
become a release candidate for 0.24.

Thanks

Henry

[Abbott, Paul]

Hi,
I've just downloaded pitest-0.24-20111128.225554-3.jar and rerun one of our JMockit tests and it runs without error. Currently PIT is the only mutation testing tool I've found that works with this mock framework.

As for UISpec4J, I asked as I'm getting one of our UISpec4J tests failing, deep down in the Swing code (repaint method I think it was) and at the moment I'm not sure if it's a problem due to the test being run with PIT or our code.

[henry]

Hi Paul,

Glad to hear JMockit is working with PIT on a real codebase - assuming
no-one reports any issues with the snapshot I'll push out an official
0.24 release early next week.

Unless I've missed one I think we're now confirmed to work with all the
major mocking frameworks.

If you do track down the UISpec4J problem to an issue with PIT could you
raise a defect with a small project that demonstrates the issue?

Thanks

Henry

[henry]

On Tue, 2011-11-29 at 11:06 +0000, Abbott, Paul wrote:

> Currently PIT is the only mutation testing tool I've found that works with this mock framework.

Out of interest, apart from PIT and Jumble have you tried any other
mutation testing tools?

[Abbott, Paul]

Javalanche, which looked promising but it would not work with JMockit. The failure was more cryptic, in that it didn't fail during initialisation (if I remember correctly) but later on when the mock was used.

Paul.

-----Original Message-----
From: pitu...@googlegroups.com [mailto:pitu...@googlegroups.com] On Behalf Of henry
Sent: 29 November 2011 11:13 PM
To: pitu...@googlegroups.com
Subject: RE: Does PIT work with jmockit ?

[Abbott, Paul]

I guess PIT does not work with cygwin? I get the following for a simple test case;

$ java -cp ".;pitest-0.24-3.jar;junit.jar" org.pitest.mutationtest.MutationCoverageReport -reportDir report -inScopeClasses Mover* -sourceDirs . -targetClasses Mover*
12:53:37 PIT >> INFO : Dependency analysis reduced number of potential tests by 0
stderr : java.lang.NoClassDefFoundError: org/pitest/coverage/execute/CoverageSlave
stderr :
Caused by: java.lang.ClassNotFoundException: org.pitest.coverage.execute.CoverageSlave
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Sourc
stderr : e)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
Could not find the main class: org.pitest.coverage.execute.CoverageSlave. Program
stderr : will exit.
Exception in thread "main"

And then the process hangs.....

I do have a workaround, I can set the CLASSPATH variable globally on the machine and the slave process runs.

Paul.

[henry]

Hi Paul,

I know that at least one person has been using PIT with cygwin without
issue with the 0.23 release. I can't imagine that any of the changes in
the 0.24 snapshot will have impacted cygwin, but you could try with 0.23
just in case?

Can you let me know how you get on. I don't have easy access to a
windows box to try and reproduce this.

Cheers

Henry

[Abbott, Paul]

Mmmmm,
You are right, I can get the test case to work with 23 and 24, but an hour ago it failed. I need to find out what I did differently, either way it's my problem and not yours.

Paul.

-----Original Message-----
From: pitu...@googlegroups.com [mailto:pitu...@googlegroups.com] On Behalf Of henry
Sent: 02 December 2011 1:44 PM
To: pitu...@googlegroups.com

[Abbott, Paul]

I think there may be a problem with UISpec4J. I'm getting some of our tests hanging during the mutation phase of testing. I've run a number of different combinations but I can't pin it down to one specific operation.

It appears to be the MutationTestSlave that is not completing. I have looked at the stack traces with jstack but nothing appears to be running (all threads are RUNNABLE or WAITING). I've reproduced the problem on 2 machines (windows and Linux) both using a Sun/Oracle 1.6 JVMs. Finally I downloaded the address book example from the UISpec4J web site and it can be used to demonstrates the problem.

If the slave process is killed, testing continues to the next set of mutations (another slave process is created) and may complete, with the large test set it may hang a second or third time.

I can supply the UISpec4J example if it would.

Paul.

-----Original Message-----
From: pitu...@googlegroups.com [mailto:pitu...@googlegroups.com] On Behalf Of henry

Sent: 28 November 2011 11:06 PM
To: pitu...@googlegroups.com

[henry]

Hi Paul,

Can you raise a defect and attach the example? I'll take a look at this
as soon as I get chance.

Thanks

Henry