Accessibility audit scope is an input, not an inference from checks
A conformance claim can report a page as unchecked only if it knows the page exists. If the list of pages under assessment gets inferred from the checks you recorded, then a page nobody ever opened is not a gap in the claim. It is absent from the claim, and the report comes back quiet about it.
That is the one defect in an evidence-based audit that looks like success. A failure is loud, because somebody wrote down what they ran into. A partially covered criterion is loud too, as long as the uncovered part is named. But a route that no tester ever loaded produces no record at all, and a system that derives its scope from the records it holds cannot tell a page that passed from a page that was never opened. We ran into this while building the evidence layer behind an accessibility claim, a11ytrail, and the fix is not a heuristic. It is moving one list from output to input.
The convenient default is exactly where a page disappears#
assess takes the scopes under assessment from its options, and falls back to the evidence when you do not give them. The fallback is this getter on Evidence:
/** The scopes that appear anywhere in the evidence. */
get scopes(): readonly string[] {
return [...new Set(this.#checks.map((check) => check.scope))].sort();
}That is the sorted set of every scope any check mentions. As a default for a quick look at a record you already trust it is reasonable, and as the denominator of a claim it is wrong. Under it, the pages under assessment are the pages somebody already touched. A route with no check contributes no scope, a scope that does not exist cannot be missing, so the criterion settles on home and checkout, and the account page is not part of the assessment in any sense you could detect as a reader.
The degenerate case is handled (an empty scope list makes the claim incomplete with the reason that no evidence has been recorded), but the degenerate case is not the dangerous one. The dangerous one is twenty pages audited carefully and three never opened, where every number in the report is true about the twenty and says nothing about the three.
Naming the scopes turns the missing page into a fact#
So you pass the list in:
const claim = assess(evidence, {
level: "AA",
build: "2026.9.0",
scopes: ["home", "checkout", "account", "password-reset"],
});and each criterion is assessed against it:
const latest = evidence
.latestPerScope(criterion.id, build)
.filter((c) => scopes.includes(c.scope));
const covered = new Set(latest.map((check) => check.scope));
const unevaluatedScopes = scopes.filter((scope) => !covered.has(scope));Two things happen in those five lines. The named scopes with no check become unevaluatedScopes, a positive fact about absence that travels with the result. And checks whose scope is not on the list get dropped, because the list is the denominator in both directions. That second half is deliberate: a claim about part of a service should not be propped up by evidence from the part it excludes. It also means you have to argue the list rather than collect it. Whatever the scopes are, somebody decided them.
In practice the list comes from the same place as the release checklist: the route table, the flows a compliance team signs off, the screens each role actually sees. Across the compliance tooling, admin panels and multi-role apps we have built, the pages that escape an audit are rarely the front page. They are the password reset, the empty state, the error path, the screen only one role opens. None of those pull traffic through a test plan on their own, and none of them generate a check you could infer a scope from.
A crawler looks like a tempting source for the list, and it is the wrong one twice over: scope then depends on what the crawler could reach, and automated checking settles only seven of the fifty-five criteria at A and AA (the set listed in AUTOMATABLE), so the tool defining the scope would be unable to decide most of what the scope is for.
A gap ranks below a failure and above a pass#
Once the gap is visible, it needs a place in the ordering. The status chain in assessCriterion is explicit about it:
let status: CriterionStatus;
if (latest.length === 0) {
status = "not-evaluated";
} else if (outcomes.has("failed")) {
status =
outcomes.has("passed") || outcomes.has("not-applicable") || unevaluatedScopes.length > 0
? "partial"
: "failed";
} else if (outcomes.has("inconclusive")) {
status = "inconclusive";
} else if (unevaluatedScopes.length > 0) {
status = "not-evaluated";
} else if (outcomes.size === 1 && outcomes.has("not-applicable")) {
status = "not-applicable";
} else {
status = "satisfied";
}Read it as three placements for the same gap.
Beside a pass, the gap wins. Passes on the pages that were checked say nothing about the pages that were not, so the criterion stays not-evaluated, and assess collects not-evaluated and inconclusive results into claim.unevaluated, any one of which makes the whole claim incomplete. A criterion that holds on three pages out of four is not three quarters satisfied. It is unsettled, and the claim has to say so.
Beside a failure, the failure wins. The criterion is already not met, and promoting the gap would change nothing you need to act on, so the status is partial and it lands in claim.failed alongside the outright failures. The claim is partially conformant either way. What the gap must not do is disappear, which is why partlyCovered picks out results that have unevaluated scopes but are not not-evaluated, and adds a reason recording that the rest is unknown rather than passing.
And a gap is not a defect. Reporting it as one sends somebody to fix code that may be perfectly fine, when the real answer is that nobody looked. The test for it reads:
const result = claim.results.find((r) => r.criterion.id === "1.4.3")!;
assert.equal(result.status, "not-evaluated");
assert.deepEqual(result.unevaluatedScopes, ["checkout"]);
assert.equal(claim.failed.length, 0, "a coverage gap must not be reported as a failure");
assert.equal(claim.status, "incomplete");The statement repeats it in words#
The ordering only matters if it survives into the published document, so describe in statement.ts carries the scopes across verbatim:
return {
criterion: result.criterion.id,
name: result.criterion.name,
level: result.criterion.level,
failingScopes: failing.map((check) => check.scope),
uncheckedScopes: result.unevaluatedScopes,
reason: notes.length > 0 ? notes.join("; ") : undefined,
alternative: organisation.alternatives?.[result.criterion.id],
};failingScopes and uncheckedScopes are rendered as two different sentences. The first says what a person runs into, the second says the page was not checked at all and is therefore unknown rather than working. Folding the unchecked pages into the failing ones would overstate the defect, and leaving them out would understate the claim. And when the claim is incomplete rather than qualified, statement refuses outright with UnsupportedClaim, because there is no honest way to render "we have not finished looking" as one of the three statuses the statement model allows.
What it costs to run#
The scope list becomes a maintained artefact, and it will drift. A route added without a line in the list is as invisible as it was under the inferred default. The mechanism moves the decision somewhere a human can be held to it, which is the whole gain, but it does not make the decision for you. We treat the list the way we treat a migration: generated from the route table where that is possible, reviewed on the pull request that adds the route, and not edited to make a report look better.
Scope ids want to be stable, too. Because checks outside the list are filtered out, renaming a route turns a well-tested page into a total gap and strands its history. The id is a key, not a URL.
The honest cost is that declaring scope up front makes your first claim look worse than the inferred one did, usually much worse, and the gap list is probably the part of the report nobody wants in writing. That is the point. The inferred claim was the same claim with the unchecked pages deleted from the question.
The discipline generalises past accessibility. Any assessment that reports coverage has a denominator, and if you derive the denominator from the samples, the report can only describe the work that was done, not the work that was skipped.