Cannot get to work mutationCoverage with @Testcontainers - example attached

[Jessica Aguado]

Hello!

I have created an issue.... but maybe someone in this group can help me. Does anyone have Pit mutation coverage working with @Testcontainers?

Here the issue: https://github.com/hcoles/pitest/issues/827

Thanks in advance and have a nice day for everyone!

Jessi

[Stefan]

Hi Jessi,

couldn't dive very deep into the details but there seems to be a difference in startup when pitest tries to gather coverage information. @Testcontainers seem to create new/more containers when invoked by pitest:

With surefire only one pg container created:

~/workspaces/pitTestcontainers [master]
21:22 $ mvn clean install | grep -i 'Starting container'
21:23:04.990 [main] DEBUG 🐳 [postgres:latest] - Starting container: postgres:latest
21:23:04.991 [main] DEBUG 🐳 [postgres:latest] - Starting container: postgres:latest
21:23:05.087 [main] INFO 🐳 [postgres:latest] - Starting container with ID: 4c26858119b56cfe87f23302428c5d7c306f2e999d8119dff8b43af391e310e0
--------- DYNAMIC PROPERTY REGISTRYjdbc:postgresql://localhost:32814/test?loggerLevel=OFF (true)

with pitest it looks good in the beginning but when pitest starts the tests a second time for gathering coverage information a new container is created but your @DynamicPropertySource isn't updated thus pointing to the first container/port that has already been stopped and replaced.

~/workspaces/pitTestcontainers [master]
21:23 $ mvn org.pitest:pitest-maven:mutationCoverage 2>&1 | grep 'Starting container'
21:23:32 PIT >> FINE : MINION : 21:23:32.027 [main] DEBUG 🐳 [postgres:latest] - Starting container: postgres:latest
21:23:32 PIT >> FINE : MINION : 21:23:32.028 [main] DEBUG 🐳 [postgres:latest] - Starting container: postgres:latest
21:23:32 PIT >> FINE : MINION : 21:23:32.120 [main] INFO 🐳 [postgres:latest] - Starting container with ID: ac65925c8236d49b23ec70984b1ade0fdd799b0f7c97cd5e454e7eb890ef56e4
21:23:36 PIT >> FINE : MINION : 2020-11-30 21:23:36.487  INFO 25383 --- [           main] 🐳 [postgres:latest]                     : Starting container with ID: f8e35d2b5b597bb2c6ed18eb87b25d9d072f5788e4d77703859177bcccc56b0a
21:24:08 PIT >> FINE : MINION : 2020-11-30 21:24:08.388  INFO 25383 --- [           main] 🐳 [postgres:latest]                     : Starting container with ID: 2d1fdc9fbaeb4ca8dea3dd853346456e1456dc67e9189fe814657529aae720a8

Not sure what's causing this difference but I regret I cannot have a deeper look into the mechanics.

Regards

Stefan

Am 30.11.20 um 12:15 schrieb Jessica Aguado:

--
You received this message because you are subscribed to the Google Groups "PIT Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pitusers+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pitusers/26c11ab9-382b-4529-89a2-c312a43e0085n%40googlegroups.com.

[Jessica Aguado]

Hi! That's right. That is what what I have managed to find out, but I don't understand why. I will continue investigating. I hope that someone has experience on this and can help.

Thank you very much! 

[Jessica Aguado]

Hi (again)! 

If anyone faces the same problem.... The solution is to add @DirtiesContext to the test class (https://github.com/jaguado-arima/pitTestcontainers/blob/master/src/test/java/com/example/pitTestcontainers/pets/PetServiceWorkingWithPitestTest.java) this way the database properties are set every time a container is created.

Nevertheless when using the singleton container approach there is no need for "tricks". So as using singleton container would be best for many cases, I think that this is what I will user (https://github.com/jaguado-arima/pitTestcontainers/blob/master/src/test/java/com/example/pitTestcontainers/pets/PetService1SingletonContainerWorkingWithPitestTest.java)

Thanks!

[Stefan]

Hi Jessi,

managed to give it another look today. Your solutions helped me to understand what's going on. The problem is Spring's test context as far as I can tell: Spring caches all application contexts loaded in a static variable and reused them among different tests and test classes (see TestContextManager). Unfortunately Spring does not cleanup after all(!) tests have been run assuming the JVM will be terminated after tests have been executed. This might hold for surefire or IDE starting those tests. But pitest reuses the JVM instance for different test suite executions. Thus the cached application contexts stay there.

On the other hand @Testcontainers integrates with the JUnit Jupiter Engine and uses ClosableResource hook to cleanup containers afterwards.

I've created a JUnit extension cleaning up application contexts on shutdown. See SpringBootCleanup class based on your sample project.

@DirtiesContext will have the same effect here and I didn't notice any differences in performance regarding this simple example. But if you're relying on application context caching to reduce overall testing time for a huge test suite adding @DirtiesContext to all tests for the sake of mutation testing will slow down test execution (Springs context cache has been implemented for a reason...).

I think it'll be best to open an issue in spring framework repository with a feature request for this behavior.

Regards

Stefan

Am 01.12.20 um 14:37 schrieb Jessica Aguado:

To view this discussion on the web visit https://groups.google.com/d/msgid/pitusers/0675042f-d596-40cf-864c-8609633158a1n%40googlegroups.com.

[Jessica Aguado]

Hi! 

Yes, I read that the properties where cached in the context... and I tried the "trick" ;) You're right: doing a "trick" is not the best solution. The fact is that I usually use the singleton container strategy, without any tricks it works, so I let it be.

I've seen the issue you've opened, so I'll keep an eye on it ;) Thank you very very much for your time.

Jessi