Limitations
This is a test generator, not an oracle. It knows only what the schema says, and the schema is written by a person who may be wrong — or by a heuristic that may be. The failures worth worrying about are the ones it does not report.
What generated tests cannot know
- Intent. A schema says "a string of 1 to 64 characters". It cannot say that the string must be a valid shipping reference, or that two customers must not share one.
- Anything about state. A generated harness constructs
an object when
new:is present, but it does not set up a database, a fixture or a sequence of calls. Behaviour that only appears after three operations stays untested. - Most side effects. The harness checks the return
value, whether the call died, and a fixed list of things a call must leave
alone:
$_,$/,$,and$;, the working directory,%ENV, the warn and die handlers, stray files, open file descriptors and an accessor's object. It does not assert what was written to a file, sent over a socket, or stored in any other global. - Concurrency and timing. Tests run one call at a time; races are out of scope.
These are properties of black-box testing, not defects. The response is to keep writing ordinary unit tests next to the generated ones, and to use mutation testing to find out which of them the suite does not really check.
Extraction is a heuristic
- Types are inferred from how a parameter is used. A parameter that is only ever compared for equality looks like a string, whatever it really is.
- Constraints are inferred from comparisons near the parameter. A bound enforced in another function, or by a database, is invisible.
- Inheritance is not followed: a constructor defined in a parent class
leaves the schema with an unenrichable
new:key. - Heuristics can also be too eager. A validation regex in the body can
be read as an enum; a documented type can conflict with the code. The
confidence level says how much evidence there was, and
--strict-pod fatalturns a documented-type disagreement into a hard error.
The confidence levels are the part of extraction to read first.
Mutation testing costs time — and can hang
- No per-mutant timeout. The per-mutant
proverun is unbounded, so a mutant that causes infinite recursion or a wait on a process stalls the whole run. Skip markers are the workaround; a watchdog around the runner is the other. - Four operators. Absence of survivors does not mean absence of untested code — only that these four transformations are caught.
- Equivalent mutants. Some mutants cannot be distinguished from the original by any test. They count as survivors and are not a defect.
- In-place writes. Mutants are applied to the real
lib/file and restored afterwards. If the process is killed in between, the file stays mutated;git restore lib/fixes it. - A baseline is required. If the suite does not pass before mutation begins, every mutant is reported as killed for the wrong reason.
The dashboard is expensive
- The suite runs twice — once under Devel::Cover, once under the LCSAJ debugger. On a large module this dominates CI time.
- Coverage-guided fuzzing needs the program to be started under
Devel::Cover; having it installed is not enough. Without it, fuzzing still runs but gets no branch feedback and no corpus minimisation benefit. - Property tests need
Test::LectroTestandTest::LectroTest::Compat; a generated file skips its property section when they are absent. - The CPAN Testers section makes HTTP requests unless
NO_NETWORK_TESTINGis set. With network access and a slow API, a run can wait minutes per report. - The trend line's y-axis is the arithmetic mean of per-file percentages, while the table's Total row is Devel::Cover's weighted aggregate. They differ by a fraction of a percent, always, by construction.
Platform and portability
- Perl 5.44.0. Two of the distribution's own test files abort after every subtest has passed, because of a bug in the Perl core (Perl/perl5#24933), fixed in 5.45.5. The failure is not in this module.
- Perl 5.036 or later. Not negotiable; the code does not compile on older perls.
- Windows. The
close_stdinoption is ignored andgenerate-test-dashboardis a bash script. The module itself and all the Perl commands work. BSD::Resourceis used only to cap the signature-evaluation subprocess, and is not available everywhere. Its absence is handled, not fatal.- Generated harnesses assume a POSIX-ish shell when they shell out; the test suite that ships with the distribution is verified on Linux, macOS and Windows CI, but a harness for code that invokes system commands inherits that code's portability problems.
Things the schema format does not express
- Element types the extractor cannot see. A hand-written schema can
describe what is inside a container with a nested
schema, but an extractedhashreforarrayrefusually has only its size constrained. - Relationships between parameters, such as "max must be greater than min". Each parameter is checked against its own spec.
- Ordering constraints between calls.
The distribution tests itself
Two integration tests run every module in lib/ through the
extractor and the generator:
EXTENDED_TESTING=1 TEST_JOBS=4 prove -Ilib t
They are the reason a change to extraction is expensive to get wrong, and the reason a failure is usually easy to locate — the end-of-run table names every routine and why it was skipped.
A generated test asserts what the schema says. When one fails, decide which of the two is wrong before changing either. Weakening the schema until the suite is green is the failure mode this tool makes easy, and it is the one to watch for in review.