Big PIT execution time difference between two modules
202 views
Skip to first unread message
Lammaer B
Nov 17, 2020, 11:24:56 AM11/17/20
to PIT Users
Hi,
I have two modules (service and webapp) with PIT tests. While according to the PIT reports they run the same amount of integration tests (~4K) the service module test execution time is MUCH longer than the webapp module (7 minutes vs 1.5).
There are 5 Minion exited abnormally due to TIMED_OUT in the service module (I know those are time consuming), but even if I reduce it to 0, the execution time difference still huge.
Any glaringly obvious in the detailed PIT results? Any advice how could I find out which unit test might be the culprit? (the verbose logging didn't help me in that regard)
Service module:
run mutation analysis 6 minutes and 57 seconds,
>> Generated 2783 mutations Killed 2192 (79%)
>> Ran 4647 tests (1.67 tests per mutation)
Webapp module
> run mutation analysis : 1 minutes and 19 seconds
>> Generated 381 mutations Killed 339 (89%)
>> Ran 4089 tests (10.73 tests per mutation)
****** SERVICE MODULE DETAILED RESULT (SLOW) --------------------------------------------------------------
================================================================================
- Mutators
================================================================================
> org.pitest.mutationtest.engine.gregor.mutators.RemoveConditionalMutator_ORDER_ELSE
>> Generated 7 Killed 5 (71%)
> KILLED 5 SURVIVED 1 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 1
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.ConditionalsBoundaryMutator
>> Generated 7 Killed 4 (57%)
> KILLED 4 SURVIVED 2 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 1
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.IncrementsMutator
>> Generated 5 Killed 5 (100%)
> KILLED 5 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.VoidMethodCallMutator
>> Generated 195 Killed 146 (75%)
> KILLED 146 SURVIVED 27 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 22
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.RemoveConditionalMutator_EQUAL_ELSE
>> Generated 224 Killed 181 (81%)
> KILLED 181 SURVIVED 22 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 21
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.ReturnValsMutator
>> Generated 372 Killed 271 (73%)
> KILLED 270 SURVIVED 10 TIMED_OUT 1 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 91
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.RemoveConditionalMutator_ORDER_IF
>> Generated 7 Killed 5 (71%)
> KILLED 4 SURVIVED 1 TIMED_OUT 1 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 1
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.MathMutator
>> Generated 5 Killed 5 (100%)
> KILLED 4 SURVIVED 0 TIMED_OUT 1 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.ConstructorCallMutator
>> Generated 75 Killed 56 (75%)
> KILLED 56 SURVIVED 3 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 16
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.NegateConditionalsMutator
>> Generated 233 Killed 198 (85%)
> KILLED 198 SURVIVED 12 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 23
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.NonVoidMethodCallMutator
>> Generated 1427 Killed 1191 (83%)
> KILLED 1187 SURVIVED 120 TIMED_OUT 2 NON_VIABLE 2
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 116
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.RemoveConditionalMutator_EQUAL_IF
>> Generated 226 Killed 125 (55%)
> KILLED 125 SURVIVED 79 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 22
--------------------------------------------------------------------------------
================================================================================
- Timings
================================================================================
> scan classpath : < 1 second
> coverage and dependency analysis : 11 seconds
> build mutation tests : 1 seconds
> run mutation analysis : 6 minutes and 57 seconds
--------------------------------------------------------------------------------
> Total : 7 minutes and 11 seconds
--------------------------------------------------------------------------------
================================================================================
- Statistics
================================================================================
>> Generated 2783 mutations Killed 2192 (79%)
>> Ran 4647 tests (1.67 tests per mutation)
***** WEBAPP MODULE DETAILED RESULT (FAST) -----------------------------------------------
2:01:29 PM PIT >> INFO : Completed in 84 seconds
================================================================================
- Mutators
================================================================================
> org.pitest.mutationtest.engine.gregor.mutators.RemoveConditionalMutator_ORDER_ELSE
>> Generated 1 Killed 1 (100%)
> KILLED 1 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.ConditionalsBoundaryMutator
>> Generated 1 Killed 1 (100%)
> KILLED 1 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.VoidMethodCallMutator
>> Generated 74 Killed 72 (97%)
> KILLED 72 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 2
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.RemoveConditionalMutator_EQUAL_ELSE
>> Generated 22 Killed 21 (95%)
> KILLED 21 SURVIVED 1 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.ReturnValsMutator
>> Generated 73 Killed 70 (96%)
> KILLED 70 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 3
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.RemoveConditionalMutator_ORDER_IF
>> Generated 1 Killed 1 (100%)
> KILLED 1 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.MathMutator
>> Generated 2 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 2
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.ConstructorCallMutator
>> Generated 1 Killed 1 (100%)
> KILLED 1 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.NegateConditionalsMutator
>> Generated 23 Killed 23 (100%)
> KILLED 23 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.NonVoidMethodCallMutator
>> Generated 161 Killed 140 (87%)
> KILLED 140 SURVIVED 6 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 15
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.RemoveConditionalMutator_EQUAL_IF
>> Generated 22 Killed 9 (41%)
> KILLED 9 SURVIVED 13 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0
--------------------------------------------------------------------------------
================================================================================
- Timings
================================================================================
> scan classpath : < 1 second
> coverage and dependency analysis : 5 seconds
> build mutation tests : < 1 second
> run mutation analysis : 1 minutes and 19 seconds
--------------------------------------------------------------------------------
> Total : 1 minutes and 24 seconds
--------------------------------------------------------------------------------
================================================================================
- Statistics
================================================================================
>> Generated 381 mutations Killed 339 (89%)
>> Ran 4089 tests (10.73 tests per mutation)
Henry Coles
Nov 17, 2020, 11:36:46 AM11/17/20
to pitu...@googlegroups.com
Your service module has 2783 mutants, the web one only 339, so it is not surprising that the service module takes longer to analyse.
Depending on how much code is in the two modules, and the sort of code it is (dense logic vs glue code moving data about) the difference in the number of mutants might be entirely expected.
You can speed repeated builds up by using the history option, or use the maven version control integration to limit the analysis to changed code.
Alternatively you could choose to use a more limited set of mutation operators, or exclude certain portions of the codebase.
Henry
--
You received this message because you are subscribed to the Google Groups "PIT Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pitusers+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pitusers/39afcfd8-36c3-473d-b34e-9beb23f14a16n%40googlegroups.com.
Message has been deleted
Message has been deleted
Lammaer B
Nov 17, 2020, 12:37:04 PM11/17/20
to PIT Users
Thanks.
Just being curious - why the number of mutants have more impact on the execution time than the number of tests? At the end, both modules had the same number of tests executed, that is why I thought something wrong with the unit tests themselves....
Regarding your advice: on what basis should I remove some mutation operators? All of them seems to be useful :)
Just being curious - why the number of mutants have more impact on the execution time than the number of tests? At the end, both modules had the same number of tests executed, that is why I thought something wrong with the unit tests themselves....
Regarding your advice: on what basis should I remove some mutation operators? All of them seems to be useful :)
Henry Coles
Nov 17, 2020, 12:56:21 PM11/17/20
to pitu...@googlegroups.com
There are no simple rules, performance depends on many details such as :-
* How high the percentage of surviving mutants is
* How fast any effective killing tests are
* How many tests are run before a mutant is killed
* How fast those tests are
* How many timeouts are encountered
* How many mutants there are
* How they are distributed
* Other factors I've forgotten
But one of the most expensive things pitest must do is launch jvms, and it must currently do this at least once for each class (as otherwise any poisoning of state in that jvm could affect the results of mutants from other classes).
So, if you have 1000 mutants, all in the same class, it might run faster than if you had 100 mutants in 100 different classes. Or it might not, depending on the other factors.
There are no simple answers.
On Tue, 17 Nov 2020 at 17:32, Lammaer B <lamma...@gmail.com> wrote:
Thanks.
Just being curious - why the number of mutants have more impact on the execution time than the number of tests? At the end, both modules had the same number of tests executed, that is why I thought something wrong with the unit tests themselves....
On Tuesday, 17 November 2020 at 17:36:46 UTC+1 henry wrote:
To view this discussion on the web visit https://groups.google.com/d/msgid/pitusers/a1fb5005-5e70-4b2d-9059-f9bad07f2e22n%40googlegroups.com.
Lammaer B
Nov 18, 2020, 2:09:55 AM11/18/20
to PIT Users
Thanks again :)
What are the ms values next to the tests on the bottom of of the PIT test reports under the "Tests examined" section?
I have some relatively slow unit tests when they executed by surefire but in the PIT report I see much smaller ms values
Actual example from the log:
If they are not actual execution times during the mutation tests, can I find such data somewhere?
(the given class where these tests are run by PIT has 124 mutations)
What are the ms values next to the tests on the bottom of of the PIT test reports under the "Tests examined" section?
I have some relatively slow unit tests when they executed by surefire but in the PIT report I see much smaller ms values
Actual example from the log:
Unit test execution (surefire):
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.878 s - ServiceTest
ms values in PIT test report:
ms values in PIT test report:
Tests examined
ServiceTest.testException() (27 ms)
ServiceTest.testGetter() (56 ms)
ServiceTest.testExceptionNew() (14 ms)
ServiceTest.testExceptionOld() (61 ms)
ServiceTest.testSetter() (15 ms)
ServiceTest.testRecommendation() (130 ms)
If they are not actual execution times during the mutation tests, can I find such data somewhere?
(the given class where these tests are run by PIT has 124 mutations)
Reply all
Reply to author
Forward
0 new messages