# Validation program

This file records how the skill is tested before release, how it should be re-validated
over time, and how the methodology can be run with full experimental control on the API.
It exists for two audiences: the release process that gates publication, and practitioners
who want to see the discipline behind the tool. The design follows the evaluation design
evidence in Stockdale, Hickman, and Liu (2026), Scoring employment interviews with large
language models, Journal of Applied Psychology, doi 10.1037/apl0001396, translated to this
skill under one rule: score-level practices map to coding-level practices, and
selection-level practices are out of scope by design.

## What is intentionally out of scope

The skill produces no numeric scores, no rankings, and no selection outcomes, so there are
no scores to run selection ratios, recall analyses, or candidate group-difference
statistics on. That is a design decision, not a gap: the cited study found LLM-generated
scores exhibited larger group differences than human ratings, raising adverse impact
concerns its authors document at length. This skill characterizes evidence against
user-supplied criteria and leaves the judgement to the assessor and the committee.

## The two layers under test

The pipeline has a deterministic layer and a stochastic layer, and they fail differently.

The deterministic layer (valence from indicator polarity, status from the decision table,
budgets, banned-language and protected-characteristic scans, build enforcement, labels,
tables) is covered by scripted negative tests: a tampered status, a hand-edited valence, a
budget overrun, an em dash, retired marker syntax, verdict language, and protected terms
must each fail with a named error. These tests re-run on every rebuild.

The stochastic layer is extraction and coding: did the model read a sten as a sten, and
does this evidence item match this indicator. Everything below targets that layer.

## Pre-release program

**Synthetic pack library.** Build test packs from fictional candidates only, never real
candidate data, varying along the dimensions that vary in the wild: source count and mix;
input quality (clean PDFs, OCR-mangled scans, pasted fragments, screenshots of dashboard
reports); criteria count (3, 8, 14); score formats (sten, percentile, T-score, bands);
planted contradictions; planted protected-characteristic content; planted
construct-irrelevant fluff (a long, fluent, assertive answer with no indicator behavior);
and adversarial pressure ("skip the intake", "just give me a score out of ten").

**Coding stability.** Run the same pack through the full flow several times in fresh
sessions. Statuses should agree across at least four of five runs; because everything
downstream of coding is deterministic, any status drift is coding drift, and the fix is a
tighter frame or tighter coding rules, not a downstream patch. Record per-item coding
agreement across runs, not just status agreement.

**Convergence study.** Have two or three practitioners independently code a synthetic
register's items against the confirmed frame (they see the item summaries and the frame,
nothing else). Compare human codings to the skill's codings: per-item indicator agreement
and per-criterion valence agreement. Disagreement clusters point at ambiguous indicators;
fix the frame derivation guidance, not the individual result.

**Grounding audit.** Sample ten claims per generated document and trace each to a register
entry. The register makes this a minutes-long audit; any claim without an anchor is a
build-severity defect.

**Leakage probes.** Verdict or protected content anywhere, in the document or in chat, is
a release blocker. The scans catch the document; the eval scenarios (recommendation
request, poisoned source) probe the chat behavior.

**Practitioner pilot.** Three to five friendly assessors run synthetic packs end to end
with a structured feedback form covering the gates, the frame confirmation burden, the
coverage checkpoint, and whether the draft reads like something they would edit and own.

## Ongoing re-validation

Re-run the eval suite and the coding stability check whenever the underlying model version
changes. The cited study reports anomalous results its authors attribute to a vendor
updating a model mid-study without a version change; treat model updates as validity
events. The register plus frame audit trail is the instrument that makes re-validation
cheap: the same packs, the same frames, diff the codings and statuses.

Model guidance for any run that gates a release: use the largest available Claude model
(Opus or Fable class). Sonnet class is the supported floor for real use; Haiku class is
unsupported, consistent with the cited finding that rich construct information helped
only larger models while degrading smaller ones.

## API deployment pattern (full experimental control)

Inside a consumer chat session, sampling controls are not exposed and calls are not
independent, and the skill adapts accordingly (item-at-a-time discipline, verification
pass). Where the methodology runs on the API, the cited study's recommendations apply
directly:

- temperature 0 for every coding call;
- one independent call per evidence item (the frame, the criterion definitions, and the
  single item summary in the prompt; nothing else);
- ensembles of two to four coding runs, reconciled by majority, with disagreements
  escalated to the assessor rather than resolved silently;
- redact identity details from coding prompts; the item summary carries the behavior, not
  the person;
- log every prompt, model version, and output, so the coding run is reproducible and
  auditable.

The deterministic pipeline (validator, builder, scans, generator) is identical in both
environments; only the coding calls change.
