Fuzzing and property tests
A generated harness is not a black box. It is a Test::Most file you can read, with a corpus you can extend, random generation you can seed, and an optional property-test layer for invariants that example-based tests cannot reach.
What is in a generated harness
- The static corpus — every case from
cases,yaml_cases, POD examples and the generated boundary list. - Boundary cases — three per declared constraint, the outer one expected to die.
- The fuzz loop —
iterationscases (30 by default) built from the declared types, with edge-case pools mixed in. - Input validation — each argument spec is re-expressed as a
Params::Validate::Strictschema, so a case that should die is checked for dying for the right reason. - Output validation — the return value goes through
Return::Set, which is what makes "returns an integer" a testable claim. - Side-effect checks — after every call the harness checks that the caller's globals (
$_,$/,$,,$;), the working directory,%ENVand the warn and die handlers are unchanged, that no file was left behind and no file descriptor leaked, and that an accessor did not damage its object. - Documentation checks — a constraint that came only from POD is probed from outside as well. A routine that rejects a value inside a documented bound, or accepts one outside a documented bound, member list or pattern, is recorded, and all such findings are reported at the end of the run. They fail the run only when the harness was generated with
--strict-documented. - Housekeeping —
srandwhen a seed is set, an alarm-based timeout, and cleanup of files a routine writes into the working directory.
# what a run looks like
$ fuzz-harness-generator -r t/conf/validate.yml
Generated: /tmp/S7Ql2m.t
Result: PASS
Files=1, Tests=118, 0 wallclock secs
If it fails, the TAP output names the case: the parameter, the value,
whether it was expected to live or die, and the seed and input that were
sent, so the case can be replayed. Turn the failing value into an
edge_cases entry so it is retested forever, not just until
the next random seed.
Turning probes on and off
The config section controls the default
probes: NUL bytes, undef, empty values, non-ASCII strings,
deduplication, the timeout, and the security-string set. All of them are
on by default except the last two, because a routine that accepts
undef is unusual — but not rare enough to be a safe
assumption. The side-effect checks have switches too:
check_files, check_fds and
check_accessors, all on by default.
config:
test_undef: no
test_nuls: no
timeout: 30
Corpora
Random inputs are worth keeping. The coverage-guided fuzzer records the inputs that reached a branch nobody had reached before, plus every input that made the code die, in a corpus file. That corpus can be replayed:
# fuzz as schemas are extracted, keeping the corpus
extract-schemas --fuzz --corpus-dir schemas/corpus lib/My/Module.pm
# turn a corpus into a deterministic regression test
fuzz-harness-generator --replay-corpus schemas/corpus/translate.json \
-o t/fuzz_replay.t
A replay harness runs the recorded inputs in a fixed order. It is the right thing to commit: no randomness, no coverage tooling, and a failure that reproduces on every machine.
Coverage-guided fuzzing
When the program is started under Devel::Cover
(perl -MDevel::Cover or cover -test), the fuzzer
does not just generate — it measures. Each iteration is scored by which branches it
executed; an input that adds coverage is kept and becomes a seed for
mutation, so later iterations are variations on inputs that already got
deep into the code rather than fresh random values.
| Method | Does |
|---|---|
run() | Fuzzes, and returns a report with total_iterations, interesting_inputs, corpus_size, branches_covered, bugs_found and the bug inputs. |
minimize_corpus() | Greedy set-cover: repeatedly keeps the input that covers the most still-uncovered branches, until every branch is covered. Returns the size before and after, and the branch count. |
save_corpus() / load_corpus() | Persist and restore across processes. Coverage data cannot travel with it, so loaded entries are deduplicated by fingerprint instead. |
Two invariants make the minimiser safe to run in CI. Bug inputs are always retained, whether or not they add coverage — they are the most valuable thing the fuzzer produces. And minimisation is idempotent: a second pass finds nothing left to remove.
Without Devel::Cover the fuzzer still runs; it simply has no branch
feedback, so it degrades to random generation with corpus retention. The
same is true when Devel::Cover is installed but the program was started
without it: the report then shows branches_covered as 0.
Corpus files are UTF-8; files written by 0.47 and earlier still load.
Property-based tests
Example-based tests check the cases you thought of. Property tests check that a statement holds for a thousand generated inputs. Turn them on in the schema:
config:
properties:
enable: true
trials: 1000
Properties come from two places. The generator derives some automatically from each transform's output spec — range bounds, type preservation, definedness, exact values — and you can list more by name, or write them as Perl. See transforms and properties for the full syntax and the list of built-in names.
Generated property tests need Test::LectroTest and Test::LectroTest::Compat. Both are prerequisites of the distribution since 0.48, so installing it installs them; a generated file that cannot load them skips its property section with a diagnostic instead of failing.
ok 123 - negative property holds (1000 trials)
ok 124 - positive property holds (1000 trials)
On failure LectroTest shrinks the counterexample to a small one and prints the inputs that broke the property. Those belong in the static corpus.
Properties worth asserting
The properties that pay for themselves are the ones an example-based test cannot express:
- Idempotence — normalising twice changes nothing.
- Round trips —
decode(encode($x)) eq $x. - Order — output of a sort is non-decreasing, and is a permutation of the input.
- Invariants across inputs — a total is never less than any of its parts; a length never grows.
Keeping fuzz runs finite
Fuzzing multiplies: every schema is a harness, every harness is a
prove run, and property tests add their trials on top. Two
practical rules keep make test quick:
- Keep the aggressive schemas in
t/conf/and the extended-test guard on the runner, so a normal test run does the cheap part and CI does the rest. - Set
iterationslow (or0, with properties only) for a schema whose random generation is slow, and rely on the corpus for depth.
A fuzz test that never fails is still doing work: it is sampling the space the schema describes. The failure rate should be near zero, and when it is not, the value is in what the failures tell you — not in raising the iteration count until they stop.