Extracting a schema from Perl
Writing a schema by hand means reading the source anyway. So the
distribution can read it for you: extract-schemas parses a
.pm file with PPI, infers one schema per routine, and rates
its own confidence in each one.
extract-schemas lib/My/Module.pm --output-dir schemas --verbose
The result is one YAML file per routine. They are ordinary schemas: edit them, keep the ones that are right, delete the rest. Extraction is a way to skip the typing, not a statement that the inference is correct.
What it reads
| Source | What is inferred |
|---|---|
Formal =head3 Input / =head4 Input blocks |
Authoritative. Both positional ([ {type => 'string'}, ... ]) and named ({ name => {type => 'string'} }) formats are understood, along with isa. Params from these blocks are never overridden by heuristics. |
Formal =head4 Output blocks |
The return type, again taking priority over inference. |
| Signatures and argument unpacking | Modern sub f($x, $y) signatures, my ($x, $y) = @_, direct $_[0] indexing, shift, and Params::Get named extraction. |
| Validation libraries | Params::Validate::Strict schema hashes, and Type::Params signature_for declarations. |
| Code heuristics | What a parameter is compared with, folded into arithmetic, checked with looks_like_number, matched against a regex, or used to index a hash. Constraints such as >= 0 become min; length(...) <= 255 becomes max. A length comparison is read according to the guard it sits in, so croak ... if length($t) < 5 and croak ... unless length($t) >= 5 both give min: 5. A parameter tested with -d gets semantic: directory, one tested only with -e or -f gets semantic: existing_file, and one used only in a truthiness test is optional. |
| POD | Parameter names and types from =item entries, plus runnable examples, which become static corpus cases. |
| Structure | Accessors (getter, setter, getset), including those that guard on defined($arg) rather than on @_; wrappers that forward @_ to another method, which inherit that method's input schema; object parameters and the methods they are documented to need (can), constructor arguments from new, Perl 5.38 class syntax, and Moose before/after/around modifiers. |
The best signal is a spec block
A formal input block beats every heuristic, because it is the author stating the contract rather than the extractor guessing at it. Writing one is the single most effective way to get a useful schema:
=head2 validate
=head4 input
{ score => { type => 'number', min => 0, max => 100 },
label => { type => 'string', optional => 1 } }
=head4 output
{ type => 'boolean' }
=cut
sub validate {
my %args = @_;
...
}
Two details worth knowing. A parameter listed in a spec block
without a type is treated as deliberately
unconstrained: the extractor will not fill in a type from heuristics, and
the harness will not generate wrong-type cases that are expected to die.
More generally, wrong-type cases are only generated where the routine's
own validator enforces the type (_source: validator in the
schema); a type taken from POD or inferred from code is not a promise to
reject a reference.
And a type may be written either as a quoted string
(type => 'object') or as a Params::Validate
constant (type => OBJECT).
A bound in a spec block is compared with the bounds inferred from the
code. A formal min => 5 next to an inferred
max => 4 is reported as a contradiction instead of being
merged into a range with no valid value. Provenance is also kept per
constraint, so a documented minimum and a code-enforced maximum on the
same field are judged separately by the harness.
The YAML is written with sorted keys and sorted analysis notes, and a
matches pattern as a plain string, so repeated runs over
the same module are byte-identical and any YAML reader can load the
result.
Confidence
Every extracted schema carries a confidence level, and the level decides what the rest of the toolkit does with it:
| Level | Typical cause | Consequence |
|---|---|---|
high | A formal spec block, or a validation-library schema plus agreeing code. | Used for test generation. |
medium | Validation library or POD and code agree, without a formal block. | Used for test generation. |
low | Only heuristics, or POD and code conflict. | Used, but worth reading before trusting. |
very_low | Very little type information; parameters found only by name. | Falls back to a TODO stub in the mutation pipeline. |
none | Nothing to go on. | Falls back to a TODO stub. |
--strict-pod controls what happens when POD and code
disagree: off ignores the conflict, warn reports
it, fatal refuses to extract. fatal is the right
setting in CI for a project that documents its parameters properly; it
turns documentation drift into a build failure.
Invocants are not parameters
$self, $class, $pkg,
$proto and $klass are recognised as the
receiver and excluded from the input schema, in every argument-unpacking
style. Object methods are emitted with a new: key, which
tells the generator to construct an instance first — and tells the
mutation pipeline that the routine needs an object it cannot build, so it
becomes a TODO stub rather than a broken schema.
A class whose constructor refuses to build one — an abstract base class,
or a constructor that croaks over an argument the schema does not describe
— gets new: ~ for its instance methods and
_STATUS: DIES for new(), so its routines are
skipped rather than reported as failures.
Fuzzing during extraction
--fuzz runs each extracted schema through the
coverage-guided fuzzer as it is produced, so a corpus accumulates next to
the schemas. --fuzz-iters sets how hard to try,
--corpus-dir says where to keep the corpus, and
--minimize-corpus trims it to the smallest set that still
covers every discovered branch.
extract-schemas --fuzz --fuzz-iters 300 \
--corpus-dir t/corpus --minimize-corpus lib/My/Module.pm
How the fuzzer uses that coverage.
Debugging one routine
The distribution tests itself by extracting schemas from every module in
lib/ and proving the harnesses that come out
(t/self-fuzz.t, opt-in via EXTENDED_TESTING=1).
When one routine fails there, isolate it with --function:
extract-schemas --function bugs --output-dir /tmp/dbg \
lib/App/Test/Generator/CoverageGuidedFuzzer.pm
fuzz-harness-generator -r /tmp/dbg/bugs.yml
The flag extracts everything, then keeps only bugs.yml and
limits the fuzz run and the summary table to that one routine. If the name
does not exist, it prints the routine names it did find and exits 1 —
usually the fastest way to discover that the sub is a method or is named
differently from what you assumed.
Where it stops
- A constructor defined in a parent class is not followed.
@ISAis not traversed, so such a routine is emitted with an unenrichablenew:rather than constructor arguments inferred from the base class. - Comments are read as code. A comment inside a sub
that quotes an argument-unpacking line is taken for a real declaration,
and under
--strict-pod=fatalthat is an error. Describe such idioms in words. - Cross-package subs are skipped.
sub DB::DBand other qualified declarations are not fuzz targets. - Heuristic types can be wrong. A parameter used only as
a hash key looks like a string. A parameter that is never compared,
concatenated or validated looks like a string too — because there is
nothing else to go on.
--verboseshows which evidence was used. - Business rules are invisible. Nothing in the source says a discount may not exceed the order total. Only a spec block or a static case can express that.
Extraction is a very good first draft and an unreliable final one. The
loop that works is: extract, run the harness, fix the schema where the
harness is wrong, keep it in t/conf/. From then on it is
hand-written code that happens to have been typed by a program.