Only mutate covered code

[kutscha]

Pit has a feature to detect, if something is covered.

Can this be used to only mutate classes where some code is covered?

[Henry Coles]

There is nothing built in that does this, but it shouldn't be hard to implement. Why would it be useful?

Henry

On Mon, 26 Jul 2021 at 11:10, kutscha <jonask...@gmail.com> wrote:

Pit has a feature to detect, if something is covered.

Can this be used to only mutate classes where some code is covered?

--
You received this message because you are subscribed to the Google Groups "PIT Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pitusers+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pitusers/60702eea-7a42-4a99-9961-8e085fff4043n%40googlegroups.com.

[kutscha]

I am trying to use mutation testing for a company and the scenario where it would be most useful would be during test development. Here we only want to test the newly created tests and would like not to have to specify specific target classes.

Currently, such tests take at least 1 hour and 20 minutes as a lot of mutants are created in untested areas.

The goal would be tests that run for a maximum of 30 seconds.

[Henry Coles]

Avoiding classes without coverage would have little effect on performance as analysis of mutants without coverage has near zero cost. It sounds like you want something slightly different though - to only mutate the classes that a recently modified/added test covers? Is that correct?

You might be interested in some of the work we are doing at CDG. We've achieved some impressive improvements to general pitest performance, and also added git integration, so analysis is driven by the code that has changed.

https://pitest.groupcdg.com/

Working the other way, so only code that is covered by changed tests is mutated would be a natural extension. We'd be happy to discuss if you'd be interested in this feature.

Henry

To view this discussion on the web visit https://groups.google.com/d/msgid/pitusers/28cf76c8-fdd3-448c-8161-acc84f681d60n%40googlegroups.com.

[kutscha]

If what your saying is correct, I don't understand what is taking so
long or the near zero is not enough in our case.
We used the default mutators.
These are the timings running pit on one class (Mind it was PIT 1.4.11):
> scan classpath : < 1 second
> coverage and dependency analysis : 5 seconds
> build mutation tests : 19 seconds
> run mutation analysis : 1 hours, 24 minutes and 25 seconds
--------------------------------------------------------------------------------
> Total : 1 hours, 24 minutes and 50 seconds

We don't want to integrate pitest with our git, because we have other
systems for that, but thank you for the offer.


Am Di., 27. Juli 2021 um 17:46 Uhr schrieb 'Henry Coles' via PIT Users
<pitu...@googlegroups.com>:

> You received this message because you are subscribed to a topic in the Google Groups "PIT Users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/pitusers/ZBB8NErf5LE/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to pitusers+u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/pitusers/CALGd%2BSRq-Dd%2BUwohyHNS1bjyPJ4KDF8FDZe2QzQ-Snaexa0cjw%40mail.gmail.com.

[Henry Coles]

An hour is insanely long for a single class. Can you provide more of the output? How many mutants were analysed? How many survived? How many had no coverage? How many timeouts were encountered etc etc.

A lot of different factors influence the performance of mutation testing, so it is hard to predict times from simple things like number of lines of code. The following all decrease performance.

* Tests that perform IO (network, file, disk, access dbs etc). These can hang, are generally much slower even when they don't, and can give unpredictable results when run in different orders.

* Code that is covered by tests, but poorly tested. All tests that exercise this code must be executed to determine that the mutants survive. They are therefore more expensive than uncovered mutants (where no tests are run), or mutants that are killed.

* Large classes. Pitest currently doesn't run mutants in the same class in parallel, so analysing thousands of mutants in the same class will be single threaded if only 1 class is analysed.

* Slow tests. A test that takes 2 seconds to run might not normally be a problem, but if that test covers a lot of code pitest might need to run it thousands of times (see 2nd point).

* Imperative code with lots of loop counters. These are prone to generating infinite loops, forcing pitest to time the mutant out. Each time out costs seconds.

1.4.11 is of course also very old. It would be worth re-running with the latest version, there have been numerous performance improvements since 1.4.11 was released.

From the timings you provided, the 19 seconds to build the mutation tests is the bit that includes the time taken to analyse mutants with no test coverage. This is very high (normally it is <1 sec), but your 1 hour 24 minutes is not being spent on uncovered mutants.

To view this discussion on the web visit https://groups.google.com/d/msgid/pitusers/CALpF80RoYA%3DT-qFA0%2BSvtE2M27QLG1BW7o0WAcnQsdhnpi1tAw%40mail.gmail.com.

[kutscha]

I forgot to mention a big part, sorry. I want to use Pitclipse to run PIT, because Eclipse is a center point in the development process.
This is why I stated using 1.4.11, the newer versions of PIT are integrated, but not released.

I ran a similar test as above-mentioned.
Stats:

> scan classpath : < 1 second
> coverage and dependency analysis : 5 seconds

> build mutation tests : 16 seconds
> run mutation analysis : 55 minutes and 34 seconds
--------------------------------------------------------------------------------
> Total : 55 minutes and 53 seconds

Survived mutants: 446

Killed: 2.671

No Coverage: 867.650

Pit Test Coverage Report says:
1070 Classes

It is surely a problem also related to pitclipse, but I thought PIT had a solution.
One key thing I think is that the generating of the HTML files takes up some significant time too? Which would go back to my orignal question to not look at uncovered mutations in the first place.