Why a scorecard owes evidence only at the ends of the scale
Interview scorecards fail in two directions. Require a written note for every rating and the panel learns to produce filler, because most ratings are unremarkable and nobody has a paragraph in them. Require none and the two ratings that move a hiring decision most (the lowest anchor and the highest) stand on nothing but a click.
We took the middle path in sealedloop (https://github.com/shipmindlabs/sealedloop), our interview-loop engine, and made it a property of the rubric rather than a habit of the panel: a competency rated at either end of its scale must carry a note filed under that competency, at least as long as the rubric's declared minimum; a middle anchor passes as given; an explicit NOT_ASSESSED owes nothing.
The ends are where a rating is hardest to revisit#
A middle anchor is a position you can argue with. If one interviewer put systems design at sd-working and another at sd-strong, the debrief has somewhere to go: both of them watched the same hour and read the same anchor descriptions, and the gap is the conversation.
The ends work differently. The bottom anchor is usually what sinks a must-have competency, and the top anchor is what carries a candidate over a bar nobody else cleared. Those are the ratings that get questioned weeks later by someone who was not in the room, and they are the ones where recalling what was actually observed is hardest. If the engine is going to spend the panel's patience anywhere, it should spend it there.
The rubric already knows which competencies matter, since each one declares mustHave and a bar that must name one of its anchor levels, but the evidence rule does not key off importance. It keys off position, because position is what makes a rating brittle.
Where the end of a scale comes from#
Nothing sorts anchors at check time. createRubric builds them once, and buildAnchors refuses a competency whose anchors are not strictly ascending integer levels, so by construction the first anchor is the lowest and the last is the highest.
const rubric = createRubric({
role: 'Backend Engineer',
minEvidenceChars: 40,
competencies: [
{
id: 'systems-design',
mustHave: true,
bar: 3,
anchors: [
{ id: 'sd-novice', level: 1, label: 'Novice', description: 'Sketches one service and stops there.' },
{ id: 'sd-working', level: 2, label: 'Working', description: 'Weighs two designs and names the tradeoff.' },
{ id: 'sd-strong', level: 3, label: 'Strong', description: 'Designs for failure, load and rollback.' },
],
},
],
});edgeName then needs no arithmetic: NOT_ASSESSED is null, the last anchor id is top, the first is bottom, anything else is null and the competency is skipped. A competency with only two anchors has no middle at all, so both of its anchors are edges, and one of our tests pins exactly that, because it is the case a reader assumes is an oversight.
The check runs in rubric order and names what is missing#
export function checkSubmittable(
card: Scorecard,
resolve: TextResolver = (digest) => readDraftText(card, digest),
): void {
const minChars = card.rubric.minEvidenceChars;
for (const competency of card.rubric.competencies) {
const rating = card.ratings[competency.id];
if (rating === undefined) {
throw new SealedLoopError(
'MISSING_RATING',
`${competency.id} has no rating; an unrated competency must say ${NOT_ASSESSED}`,
);
}
const edge = edgeName(competency, rating);
if (edge === null) {
continue;
}
const filed = card.evidence
.filter((note) => note.competencyId === competency.id)
.map((note) => resolve(note.digest));
// backed: some filed read is PRESENT and its trimmed text is at least minChars code points,
// otherwise EVIDENCE_REQUIRED names the competency, the edge, the anchor id and minChars
}
}Two refusals, both earning their codes. MISSING_RATING fires when a competency has no entry in card.ratings, because leaving a competency out is not the same as saying NOT_ASSESSED: silence is an interviewer who ran out of time, and the engine will not let that read as a deliberate abstention. The same idea runs through the aggregation side, where anchorPosition(NOT_ASSESSED) is null rather than a number, so a competency nobody assessed can never be ranked below the lowest anchor.
EVIDENCE_REQUIRED fires when an edge rating has no long-enough note behind it, and the message names the competency, the edge, the anchor id and minChars. That matters more than it sounds. An interviewer who gets this back does not have to hunt for which of six competencies is short, or guess how much writing will clear it.
It measures effort, not substance#
The check counts characters. Notes arrive through addEvidence, which refuses a non-string with INVALID_EVIDENCE and normalizes to NFC before hashing, so what gets measured is a stable form of what was typed. Length is taken over text.trim() by countChars, spreading the string so the unit is code points rather than UTF-16 units, and a note filed under a different competency does not count toward this one.
test('a middle anchor needs no evidence and NOT_ASSESSED needs none either', () => {
const card = setRating(
setRating(draft(), 'systems-design', 'sd-working'),
'collaboration',
NOT_ASSESSED,
);
checkSubmittable(card);
});
test('whitespace-only evidence does not back an edge anchor', () => {
const card = addEvidence(
setRating(setRating(draft(), 'systems-design', 'sd-strong'), 'collaboration', NOT_ASSESSED),
'systems-design',
' \n\t ',
);
refusal(() => checkSubmittable(card), 'EVIDENCE_REQUIRED');
});
test('evidence filed under another competency does not count', () => {
const card = addEvidence(
setRating(setRating(draft(), 'systems-design', 'sd-strong'), 'collaboration', NOT_ASSESSED),
'collaboration',
NOTE,
);
refusal(() => checkSubmittable(card), 'EVIDENCE_REQUIRED');
});That is the whole of it. The engine cannot tell a real observation from forty characters of plausible noise, and we did not pretend otherwise. What a length floor buys you is the removal of the empty gesture (the top rating with an empty box, or a box holding a space), and it puts the demand in front of the interviewer while the interview is still fresh instead of at the debrief. Substance stays a human review: the notes are read out when the cards are revealed, next to the ratings they back.
One subtlety is worth lifting out of the signature. checkSubmittable takes a resolver rather than reading text itself, defaulting to readDraftText over the draft's own texts map, and the submit method of Loop runs the same check again with a resolver backed by the store. A read that comes back MISSING cannot back an edge rating, so the rule covers text the system can actually produce, not a digest that once had something behind it.
The bar rides inside the hash#
minEvidenceChars is not a service setting and not a line in a hiring handbook. createRubric takes input.minEvidenceChars or DEFAULT_MIN_EVIDENCE_CHARS, which is 80; requireCount refuses anything but a positive integer with INVALID_RUBRIC; and the value goes into the body that rubricHash is computed over.
rubric.rubricHash; // sha256 hex of the canonical form
Object.isFrozen(rubric); // true, all the way down
const revised = reviseRubric(rubric, { role: 'Backend Engineer II' });
revised.parent === rubric.rubricHash; // the old hash, kept as lineage
assertSameRubric([{ rubricHash: rubric.rubricHash }, { rubricHash: revised.rubricHash }]);
// throws SealedLoopError with code RUBRIC_MISMATCH, naming both hashesSo the bar is identified along with the role and the competencies. Move it from 80 to 40 and you do not get the same rubric with a looser rule, you get a different rubricHash, with reviseRubric keeping the old hash as parent so the lineage stays readable. A scorecard is stamped with the hash it was drafted from, and assertSameRubric refuses a set of cards resting on two hashes rather than averaging them.
That closes the obvious hole. A loop where one interviewer owed 80 code points and another owed 20 is not a panel disagreeing about a candidate; it is two different instruments, and it is cheaper to make the arithmetic impossible than to detect the mismatch after a decision.
What it costs to run#
Three costs, named plainly.
The panel writes only where the scale ends, which is the point, but it creates a quiet incentive: an interviewer in a hurry can avoid writing by rating one step in from the end. We think the trade is acceptable because the dodge is visible, since a must-have competency still has to clear its bar and a middle rating on a must-have is its own signal in the debrief, but anyone adopting this rule should expect that behaviour rather than discover it.
The threshold is a judgement call, not a finding. Eighty code points is the default because it is about one specific sentence. Set it per role and let the hash change announce it.
And the note has a lifecycle with real consequences. addEvidence keeps the digest on the card and the text beside it on the draft; submit moves the text into the candidate's vault and files a redacted card holding digests only, and the chain commits the digest of each note, never the text. Erasing a candidate leaves a tombstone and the chain still verifies, which also means the evidence that justified an edge rating can be gone while the rating remains. That is the right order of priorities for candidate data, and you are better off knowing it upfront than discovering it when someone asks about a decision from two years back.
The policy is one line: notes at the ends, nothing in the middle. It only holds because the rubric is hashed content rather than a document. Put the minimum anywhere else and it becomes a convention, which is another word for something each panel applies a little differently.