I am using PIT mutation testing (1.6.7) and a maven build system. There is information on which test killed the mutants. But, what I need is whether the mutants were killed by JUnit assertion violation or by any implicit checks or run-time system? Is it possible by PIT?
2 Answers
If by 'implict checks' you mean a compile time error, then this is not possible under PIT. Unlike some earlier systems it will not produce mutants that are not viable.
The only way in which a mutant can be killed is if an uncaught exception is thrown while running a test. PIT does not distinguish between AssertionFailedErrors and other types of exception, so it is not possible to tell if the test failed or errored.
2 Comments
middle-mood
Can I ask another question? I found that during across several tests, the set of generated mutants are not the same. How to keep the generated mutants the same across all runs? Please note that, I am modifying my test suite, and running mutation testing, so I want the set to be the same, so that I can do some valid comparisons across the runs.
henry
Pitest will always generate the same mutants when given the same input. If this is not what you're seeing then something strange is going on.
I implemented such an extension that differentiates between assertion failures and implicitly thrown exceptions a couple of years ago. Though, the code was not merged back to Pitest.
You can still find the branch here: https://github.com/hcoles/pitest/compare/master...nrainer:pitest:features/categorizeTestFailure
To get meaningful results, you will need to generate a full mutation matrix (for each mutation, executing all tests that cover it, without stopping when the first test case kills the mutation). This means that the mutation "kill type" depends on mutation and test case.
