Mutation testing

Coverage says a line ran. Mutation testing says the line mattered: change it slightly, and see whether any test notices. A mutant that survives is either an untested behaviour or a line that does not need to exist — both worth knowing.

test-generator-mutate --lib lib --tests t --json mutation.json --min-score 85

Operators

OperatorMutatesCatches
BooleanNegationA boolean expression or a returned boolean, including return $x; → return !$x;Tests that check truthiness at all, rather than only that a call succeeded.
ConditionalInversionif/unless/while/until conditions, negated in placeBranches where only one side is exercised.
NumericBoundaryA numeric literal, nudged to the value either side of itOff-by-one handling and inclusive-versus-exclusive comparisons.
ReturnUndefA return expression, replaced by undefCallers that never check for a failed or empty result.

Four operators is a deliberate choice. They are the ones whose survivors usually mean something, and whose mutants rarely fail for structural reasons. A mutant that does not compile, or that changes a signature, is noise in a report a human has to read.

How a mutant is applied

The mutated source is written into the project's own lib/ file, the tests run, and the original is restored — always, including when prove crashes. A workspace copy is used only for the syntax check.

This is not the obvious design, and the reason is worth knowing if you modify it. Many CPAN modules put use lib "$Bin/../lib" near the top of a test file. use lib unshifts onto @INC at compile time, which happens after PERL5LIB has been applied — so a harness pointed at a mutated workspace copy still loads the original library, every mutant survives, and the reported kill rate is 0%. Writing to lib/ means every resolution path sees the mutant.

If the process is killed mid-run

The restore happens in the same process that writes the mutant. Kill it between the two and the file is left mutated. git status lib/ will show it; git restore lib/ puts it back. Running mutation testing only on a clean tree makes that obvious.

Per-mutant runs use prove -l -r. The baseline run uses prove -Mblib -l -r, deliberately, so the baseline sees what the full suite sees while the per-mutant runs are not shadowed by the original code in blib/lib.

Reading the result

TermMeaning
KilledA test failed with the mutant applied. The behaviour is covered.
SurvivedEvery test passed. Nothing asserts this behaviour.
SkippedThe line is inside a MUTANT_SKIP block, or the file was excluded.
DifficultyHIGH, MEDIUM or LOW — how hard the survivor is likely to be to kill, used to order the work and to decide what to generate.
ScoreKilled as a percentage of mutants that were actually run.

Fast mode

mutation_level => 'full' is the default and tries every mutant. 'fast' deduplicates first: several mutants on one line usually produce the same behaviour, and one good test kills all of them. The fast list is a subset of the full one, so it is the right mode for a pull request and the full mode for a nightly run.

Skipping a line

Some code must not be mutated, because a mutant does not fail the suite — it hangs it. Mark the region with two anchored comments:

## MUTANT_SKIP_BEGIN
return 1 if $n <= 1;
## MUTANT_SKIP_END

The two cases this exists for:

Markers are matched only as whole comment lines. A mismatched or nested pair is fatal rather than silently ignored, because a skip that stops halfway is how a run turns into an afternoon.

The feedback loop

A survivor is a statement that a test is missing. The pipeline tries to write that test:

mutation survivor
    │
    ▼
SchemaExtractor reads the enclosing sub
    │
    ▼
schema augmented with the mutant's boundary value
    │
    ▼
written to t/conf/
    │
    ▼
t/fuzz.t picks it up and runs it
    │
    ▼
mutation killed on the next run

Three flags drive it, and they are additive:

FlagProduces
--generate_mutant_tests=DIR t/mutant_YYYYMMDD_HHMMSS.t — TODO stubs for HIGH and MEDIUM survivors, with the suggested boundary value, the enclosing subroutine and environment hints. LOW survivors get a comment only. Several mutants on one line collapse into one stub, because one good test kills them all.
--generate_test=mutant For each NUM_BOUNDARY survivor, extract the enclosing routine's schema and write a runnable t/conf/mutant_MODULE_FUNCTION_*.yml, with the boundary value and one value either side added. Falls back to a TODO stub when the schema cannot be parsed, the enclosing sub cannot be found, or confidence is very_low/none.
--generate_fuzz Copies every existing schema in t/conf/ that has matching survivors, merges their boundary values in, and writes t/conf/mutant_fuzz_*.yml. The originals are never modified, and files that already start with mutant_fuzz_ are skipped so augmentation cannot cascade.
test-generator-index \
    --generate_mutant_tests=t \
    --generate_test=mutant \
    --generate_fuzz

The shipped dashboard.yml workflow runs all three and commits the new files, so a repository accumulates tests that were provoked by its own mutation results. That is the closest thing here to a test suite that improves without being written.

Feeding survivors straight into a harness

The same report can be handed to the harness generator, with no intermediate schema file:

test-generator-mutate --lib lib --tests t --json mutation.json
fuzz-harness-generator --mutation-results mutation.json -o t/fuzz.t schema.yml

Each surviving numeric-boundary mutant in the schema's module is traced to the comparison it mutated, and that comparison becomes a min or max on the matching input field, so the harness generates values either side of the boundary the suite missed. A constraint already in the schema is never replaced, and survivors that do not describe a numeric comparison are listed for review, since they need a hand-written test. From Perl, pass mutation_results => $file to generate().

A survivor is not always a bug

Sometimes the mutant is equivalent: no input can distinguish it from the original. Equivalent mutants are unavoidable in mutation testing, and a kill rate of 100% is usually a sign that the source has been shaped around the operators rather than that the tests are complete. Use the score as a trend, not a target.

← Fuzzing Dashboard →