Wrong coverage with powermock

[Richard Sassoon]

Hi all,

I'm having a hard time making PIT work correctly when using PowerMock. It seems that too much byte code manipulation causes strange coverage issues.

In the above example my tests contain calls to PowerMockito.whenNew operator for the internal collaborator returning a MatchConverter mock, and a mockStatic for the static MatchSearchDao.findMatchById method where Match is returned as a mock.

The converter.convert is correctly stubbed. The match.setChanged is correctly verified. And all the unit tests pass without problems checking the corresponding value for matchChanged as the return.

I tried then adding a different method implementation and got this:

Now using an injected internal converter mock, the coverage goes green. But for an injected match mock still the match.SetChanged verification was not covered, so I comment it. And the weirdest part was the return line not being covered!? How is that possible?

I have the corresponding tests for true/false expectations. And the canConvertMatch is correctly checked. 

I'm running PIT 1.1.11, powermock 1.6.6 and mockito 1.10.19.

I would appreciate any feedback as I really like PIT and am trying to get it running in my company, but with the wrong coverage in such cases it gets difficult... A lot of legacy code which makes extensive use of the more advanced features of powermock.

Thanks!

Best regards,

Richard

[Henry Coles]

Hi Richard,

It's a long time since I implemented powermock support for PIT and I'm afraid I've forgotten the details of what powermock does behind the scenes.

If you can put together a minimal project that reproduces the issue I can take a look and see what's going on.

Henry

--
You received this message because you are subscribed to the Google Groups "PIT Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pitusers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Richard Sassoon]

Hi Henry,

That would be awesome. Please take a look at https://github.com/rsassoon/pit_powermock_example

Let me know if there is anything I can help with.

Thanks a lot!

Cheers,

Richard

--
You received this message because you are subscribed to a topic in the Google Groups "PIT Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pitusers/bKsfZqiBbos/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pitusers+unsubscribe@googlegroups.com.

[Richard Sassoon]

Hi Henry,

Any news?

Thanks!

Cheers,

Richard

[henry]

Sorry, not had chance to look yet - only just back from Tokyo.

On Sunday, 19 March 2017 19:32:10 UTC, Richard Sassoon wrote:

Hi Henry,

Any news?

Thanks!

Cheers,

Richard

On Tue, Mar 7, 2017 at 8:43 PM, Richard Sassoon <> wrote:

Hi Henry,

That would be awesome. Please take a look at https://github.com/rsassoon/pit_powermock_example

Let me know if there is anything I can help with.

Thanks a lot!

Cheers,

Richard

On Tue, Mar 7, 2017 at 4:02 PM, 'Henry Coles' via PIT Users <> wrote:

Hi Richard,

It's a long time since I implemented powermock support for PIT and I'm afraid I've forgotten the details of what powermock does behind the scenes.

If you can put together a minimal project that reproduces the issue I can take a look and see what's going on.

Henry

On 6 March 2017 at 19:57, Richard Sassoon <> wrote:

Hi all,

I'm having a hard time making PIT work correctly when using PowerMock. It seems that too much byte code manipulation causes strange coverage issues.

In the above example my tests contain calls to PowerMockito.whenNew operator for the internal collaborator returning a MatchConverter mock, and a mockStatic for the static MatchSearchDao.findMatchById method where Match is returned as a mock.

The converter.convert is correctly stubbed. The match.setChanged is correctly verified. And all the unit tests pass without problems checking the corresponding value for matchChanged as the return.

I tried then adding a different method implementation and got this:

Now using an injected internal converter mock, the coverage goes green. But for an injected match mock still the match.SetChanged verification was not covered, so I comment it. And the weirdest part was the return line not being covered!? How is that possible?

I have the corresponding tests for true/false expectations. And the canConvertMatch is correctly checked. 

I'm running PIT 1.1.11, powermock 1.6.6 and mockito 1.10.19.

I would appreciate any feedback as I really like PIT and am trying to get it running in my company, but with the wrong coverage in such cases it gets difficult... A lot of legacy code which makes extensive use of the more advanced features of powermock.

Thanks!

Best regards,

Richard

--
You received this message because you are subscribed to the Google Groups "PIT Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pitusers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[henry]

Just taken a look at this now.

The issue seems to be that whatever modifications powermock makes to the class (I've not dug deep enough to understand what they are yet) are being made based on the un-instrumented bytecode of the original class during the coverage stage.

When a mutant is active PIT does some tricks to ensure that tools (such as powermock) that are based on javassist see the mutated version of the bytecode - but the same tricks are not applied at the coverage stage, so at least some of the probes that PIT inserts are being removed.

I've tried a quickly hacked in change that seems to fix the issue - but a bit of care and thought will be required to make the change for real.

Thanks for reporting this. Unless this is the result of a recent change on the powermock side it seems likely that PIT has been under reporting line and mutation coverage for codebases using powermock for some time.

Henry

[Richard Sassoon]

Hei Henry,

Thanks a lot for looking into it. I appreciate the feedback. I realize that it might take a bit more time to come up with the proper solution, but at least you already have a good idea about it.

I was setting up a unit testing workshop at my company and during the exercises/demo preparation I realized that there was something weird. I wanna point out to my colleagues that line coverage is not enough, and that mutation testing provides much more information, and also includes line coverage hehe...  Now I know that this will be handled.

Thanks again! Let me know if I can help with something.

Cheers,

Richard

[Richard Sassoon]

Hei Henri,

any news regarding this bug?

Thanks!

Cheers,

Richard

Virus-free. www.avg.com

[Richard Sassoon]

Ah, just saw it has been released! I haven't looked into it for a long time. 

Thanks!