Extracting a schema from Perl

Writing a schema by hand means reading the source anyway. So the distribution can read it for you: extract-schemas parses a .pm file with PPI, infers one schema per routine, and rates its own confidence in each one.

extract-schemas lib/My/Module.pm --output-dir schemas --verbose

The result is one YAML file per routine. They are ordinary schemas: edit them, keep the ones that are right, delete the rest. Extraction is a way to skip the typing, not a statement that the inference is correct.

What it reads

SourceWhat is inferred
Formal =head3 Input / =head4 Input blocks Authoritative. Both positional ([ {type => 'string'}, ... ]) and named ({ name => {type => 'string'} }) formats are understood, along with isa. Params from these blocks are never overridden by heuristics.
Formal =head4 Output blocks The return type, again taking priority over inference.
Signatures and argument unpacking Modern sub f($x, $y) signatures, my ($x, $y) = @_, direct $_[0] indexing, shift, and Params::Get named extraction.
Validation libraries Params::Validate::Strict schema hashes, and Type::Params signature_for declarations.
Code heuristics What a parameter is compared with, folded into arithmetic, checked with looks_like_number, matched against a regex, or used to index a hash. Constraints such as >= 0 become min; length(...) <= 255 becomes max. A length comparison is read according to the guard it sits in, so croak ... if length($t) < 5 and croak ... unless length($t) >= 5 both give min: 5. A parameter tested with -d gets semantic: directory, one tested only with -e or -f gets semantic: existing_file, and one used only in a truthiness test is optional.
POD Parameter names and types from =item entries, plus runnable examples, which become static corpus cases.
Structure Accessors (getter, setter, getset), including those that guard on defined($arg) rather than on @_; wrappers that forward @_ to another method, which inherit that method's input schema; object parameters and the methods they are documented to need (can), constructor arguments from new, Perl 5.38 class syntax, and Moose before/after/around modifiers.

The best signal is a spec block

A formal input block beats every heuristic, because it is the author stating the contract rather than the extractor guessing at it. Writing one is the single most effective way to get a useful schema:

=head2 validate

=head4 input

{ score => { type => 'number', min => 0, max => 100 },
  label => { type => 'string', optional => 1 } }

=head4 output

{ type => 'boolean' }

=cut

sub validate {
	my %args = @_;
	...
}

Two details worth knowing. A parameter listed in a spec block without a type is treated as deliberately unconstrained: the extractor will not fill in a type from heuristics, and the harness will not generate wrong-type cases that are expected to die. More generally, wrong-type cases are only generated where the routine's own validator enforces the type (_source: validator in the schema); a type taken from POD or inferred from code is not a promise to reject a reference. And a type may be written either as a quoted string (type => 'object') or as a Params::Validate constant (type => OBJECT).

A bound in a spec block is compared with the bounds inferred from the code. A formal min => 5 next to an inferred max => 4 is reported as a contradiction instead of being merged into a range with no valid value. Provenance is also kept per constraint, so a documented minimum and a code-enforced maximum on the same field are judged separately by the harness.

The YAML is written with sorted keys and sorted analysis notes, and a matches pattern as a plain string, so repeated runs over the same module are byte-identical and any YAML reader can load the result.

Confidence

Every extracted schema carries a confidence level, and the level decides what the rest of the toolkit does with it:

LevelTypical causeConsequence
highA formal spec block, or a validation-library schema plus agreeing code.Used for test generation.
mediumValidation library or POD and code agree, without a formal block.Used for test generation.
lowOnly heuristics, or POD and code conflict.Used, but worth reading before trusting.
very_lowVery little type information; parameters found only by name.Falls back to a TODO stub in the mutation pipeline.
noneNothing to go on.Falls back to a TODO stub.

--strict-pod controls what happens when POD and code disagree: off ignores the conflict, warn reports it, fatal refuses to extract. fatal is the right setting in CI for a project that documents its parameters properly; it turns documentation drift into a build failure.

Invocants are not parameters

$self, $class, $pkg, $proto and $klass are recognised as the receiver and excluded from the input schema, in every argument-unpacking style. Object methods are emitted with a new: key, which tells the generator to construct an instance first — and tells the mutation pipeline that the routine needs an object it cannot build, so it becomes a TODO stub rather than a broken schema.

A class whose constructor refuses to build one — an abstract base class, or a constructor that croaks over an argument the schema does not describe — gets new: ~ for its instance methods and _STATUS: DIES for new(), so its routines are skipped rather than reported as failures.

Fuzzing during extraction

--fuzz runs each extracted schema through the coverage-guided fuzzer as it is produced, so a corpus accumulates next to the schemas. --fuzz-iters sets how hard to try, --corpus-dir says where to keep the corpus, and --minimize-corpus trims it to the smallest set that still covers every discovered branch.

extract-schemas --fuzz --fuzz-iters 300 \
    --corpus-dir t/corpus --minimize-corpus lib/My/Module.pm

How the fuzzer uses that coverage.

Debugging one routine

The distribution tests itself by extracting schemas from every module in lib/ and proving the harnesses that come out (t/self-fuzz.t, opt-in via EXTENDED_TESTING=1). When one routine fails there, isolate it with --function:

extract-schemas --function bugs --output-dir /tmp/dbg \
    lib/App/Test/Generator/CoverageGuidedFuzzer.pm
fuzz-harness-generator -r /tmp/dbg/bugs.yml

The flag extracts everything, then keeps only bugs.yml and limits the fuzz run and the summary table to that one routine. If the name does not exist, it prints the routine names it did find and exits 1 — usually the fastest way to discover that the sub is a method or is named differently from what you assumed.

Where it stops

Treat the output as a draft

Extraction is a very good first draft and an unreliable final one. The loop that works is: extract, run the harness, fix the schema where the harness is wrong, keep it in t/conf/. From then on it is hand-written code that happens to have been typed by a program.

← Commands Fuzzing →