The comparison lab

Two decoders you can inspect

This is a synthetic symbolic OCR analogue. It does not read pixels. We corrupt body digits 1 → l and 0 → O, leaving the page suffix unchanged. The baseline repairs every lowercase l to 1. The candidate repairs O to 0, and can also remember a labeled document body from training.

The candidate uses the document ID as a lookup key. If it has a template, it copies that body and appends the observed page suffix. Otherwise it uses its generic substitution rule. A template is learned only from admitted training labels; target references are used only for scoring.

Both protocols admit six training pages with matched reference lengths. Disjoint training uses G1–L1. Sibling-exposed training uses A1–F1. The targets remain A2–F2 throughout. The actual training content changes; the contrast is a deliberately constructed mechanism, not a measured causal effect for a real learned model.

LAB-001 · Every string and rule is invented and shipped in the canonical model.

Predict, switch, inspect

Before switching, predict: will the candidate’s template shortcut still work when training contains no document A–F? Compare the candidate string for A2 in both states. Then look at E2: is the overall loser worse on every document?

With siblings admitted, the candidate reproduces all target bodies. With disjoint documents, it falls back to its generic rule. The baseline never uses training templates, so its predictions and score do not move. The scorer has faithfully measured two different information conditions.

The lesson is not that group splitting always lowers scores, or that all template use is wrong. It is that the test’s information condition must match the intended deployment question.

Open the full record

These records expose the observed string, reference, both predictions, edit counts and reference lengths. In the sibling-exposed protocol the candidate prediction is exactly the reference for every target; the baseline is unchanged. Nothing is inferred from a hidden model call.

To reproduce both protocols, run node lab/run.mjs from the project folder. The canonical model (local copy) is shared by the browser and the runnable lab. Its edit-distance routine and resampling checks are included. Now we can ask a different question: how much of a comparison depends on which documents happened to be sampled?