Write the sentence before the number

You replace an OCR system and the reported character error rate falls. Before celebrating, ask one question: what was new to the model when it produced those predictions? A fresh page from a familiar document and a fresh document from an unfamiliar archive are different tests.

For this guide, our question is concrete: which decoder makes fewer character errors on documents whose other pages were unavailable during training? Later we will add uncertainty and ask whether the errors matter. We will use six invented documents so that every prediction can be inspected.

A useful evaluation sentence names the population, the information available at prediction time, the independent unit, the score and the intended decision. For Arabic document extraction, you might specify new documents from a defined collection, scanned pages only, grouping by originating document or writer, a declared transcription policy, and a decision about whether to replace the current system. Those choices need evidence from your actual collection; this guide supplies no such evidence.

The evaluation sentence is our synthesis of the paper’s claim and information-boundary principles. Kapoor & Narayanan · CL-001

Make the comparison repeatable

Start with the system someone could use today: the current OCR pipeline, a simple decoder, or a published method you can rerun. Record its checkpoint, inputs, preprocessing, output conversion and failure policy. Compare both systems on the same eligible items, with the same references and scorer. Report different tuning budgets or extra training data when they are part of the comparison.

A number copied from another paper may provide context. It becomes a matched baseline only when the data version, split, scorer and relevant conditions align. An OCR timeout must not quietly disappear from one model’s denominator; define how missing outputs enter scoring before running the comparison.

“Candidate minus baseline” is incomplete. Add: on which items, under which information boundary, scored in which way?

Matched comparison checklist: an application of controlling sources of benchmark variation. Bouthillier et al. · CL-007

A short refresher: learning and measuring

A training loss supplies the numerical objective used to fit a model. An evaluation metric measures a chosen property of its predictions. They can coincide, but optimizing one does not automatically improve the other—or performance on new examples.

Use training data for fitting and development data for model choices, including thresholds, normalization choices and early stopping. Keep a final test set protected until the comparison procedure is settled. If you use its errors to choose the next model, those examples have entered development; a revised claim needs fresh protected evaluation.

Our tiny experiment is a teaching bench, not an untouched final test. We inspect it freely because we make no real model-selection claim from it. First, we need to agree on how its mistakes are counted.

Preprocessing and leakage · CL-006

Count edits, then divide

Take target page A2. Its reference is 110-2; the baseline produces 11O-2. One substitution changes the letter O into the digit 0. The five-character reference therefore gives an error rate of one fifth, or 20%.

Let $r$ be a reference string, $\hat r$ a predicted string, and $N=|r|$ the reference character count. Let $S,D,I$ count substitutions, deletions and insertions in a minimum-edit alignment. With unit cost per edit:

The denominator counts the reference, not the prediction. Insertions can make CER exceed 100%: reference a and prediction aaaa need three insertions, giving 300%. Our scorer rejects an empty reference because this simple ratio would divide by zero; a production scorer needs an explicit policy for such cases.

Definition is source-backed; examples are invented arithmetic. JiWER character scoring · CL-008

What is a “character” here?

Our lab counts Unicode code points, preserves punctuation and whitespace, and performs no normalization. The page suffix counts too. A visible mark can contain several code points, so this is a declared scoring convention, not a universal definition of a human-perceived character.

For Arabic OCR, decide whether diacritics, punctuation, spacing and letter distinctions must be preserved for the task. Unicode canonical normalization can make equivalent encodings comparable; deleting diacritics is a separate task policy. Silently changing either policy can change the score without changing a prediction.

Keep the raw outputs and apply the same frozen transformation to both systems and references. A predetermined normalization rule is different from learning a vocabulary, imputer or scaler from held-out data.

Unicode normalization · CL-009

Characters or documents?

Our six target pages contain 49 reference characters. In the disjoint-document protocol, the baseline makes 17 edits and the candidate makes 20. Adding edits first produces 34.69% versus 40.82% corpus CER. These are deterministic outputs of the invented fixture, not OCR benchmark results.

Let $e_d$ be total edits and $n_d$ reference characters for document $d$. With $G$ documents:

Corpus CER gives a longer document more influence. The document mean gives each document one vote after combining its pages. In this fixture the document means are 31.11% versus 41.39%. Neither weighting is automatically fairer: choose whether your decision is about a typical reference character or a typical document, and state how the sample represents that population.

Here each document contributes one target page. In a real collection with many pages per document, ten pages can improve measurement of one document without creating ten independent documents.

The weighting relationship follows directly from the definitions. Numbers come from LAB-001. We use document means for the paired uncertainty example.

A new page can carry an old answer

Every invented document has two pages with the same body and a different page suffix. A1 is 110-1; A2 is 110-2. They are different strings. Yet seeing the labeled first page can disclose almost the entire answer to the second.

If deployment means unseen documents, hold all members of an evaluation document out of training. If deployment means unseen writers, organizations or templates, document grouping alone may still share information that your claim requires to be new. The right unit follows the question and the collection’s provenance.

Previous pages may be legitimate inputs in a product that explicitly supplies them at prediction time. Then evaluate that product under that information policy. Our lab labels sibling exposure inappropriate because its question requires new documents.

Grouped cross-validation · CL-003

Check identity, similarity and origin

An exact file hash can catch identical bytes across partitions. It will miss a re-encoded image, a different crop, or a new scan of the same page. Text equality can catch some repeated content; near-match retrieval can surface more candidates. Review those candidates alongside document IDs, source records and collection dates.

For OCR, inspect whether a “different” PDF is a second copy of the same report or whether a preprocessed crop came from a held-out original. Define duplicate policy before final evaluation and keep related records together when the deployment question requires it. Record what the search could not detect.

Near-duplicate research on image benchmarks motivates this audit, but it does not validate a universal OCR detector. No matches found is not proof of independence.

Barz & Denzler · CL-004

The boundary also surrounds preprocessing

A clean file split can still leak. Suppose feature selection uses all labels before the split, or an imputer estimates its values from all records. Held-out information has already shaped the transformation, even if the final estimator sees only training rows.

1. Choose the split units and assign partitions.
2. Fit learned transformations and the model using only the training part of that fold.
3. Apply those fitted transformations to development examples; choose settings there.
4. Freeze the procedure, then evaluate on the protected test set.

When cross-validating, repeat fitting inside each training fold. A pipeline helps preserve that order, but it cannot choose valid group IDs or notice that a feature was unavailable at deployment time. Those are design decisions.

Preprocessing and leakage · CL-005

Kapoor and Narayanan’s leakage taxonomy is broader than duplicate files: it includes broken separation, illegitimate features, and test data that do not support the claimed distribution. We will now isolate one transparent shortcut.

Kapoor & Narayanan · CL-002

Two decoders you can inspect

This is a synthetic symbolic OCR analogue. It does not read pixels. We corrupt body digits 1 → l and 0 → O, leaving the page suffix unchanged. The baseline repairs every lowercase l to 1. The candidate repairs O to 0, and can also remember a labeled document body from training.

The candidate uses the document ID as a lookup key. If it has a template, it copies that body and appends the observed page suffix. Otherwise it uses its generic substitution rule. A template is learned only from admitted training labels; target references are used only for scoring.

Both protocols admit six training pages with matched reference lengths. Disjoint training uses G1–L1. Sibling-exposed training uses A1–F1. The targets remain A2–F2 throughout. The actual training content changes; the contrast is a deliberately constructed mechanism, not a measured causal effect for a real learned model.

LAB-001 · Every string and rule is invented and shipped in the canonical model.

Predict, switch, inspect

Before switching, predict: will the candidate’s template shortcut still work when training contains no document A–F? Compare the candidate string for A2 in both states. Then look at E2: is the overall loser worse on every document?

With siblings admitted, the candidate reproduces all target bodies. With disjoint documents, it falls back to its generic rule. The baseline never uses training templates, so its predictions and score do not move. The scorer has faithfully measured two different information conditions.

The lesson is not that group splitting always lowers scores, or that all template use is wrong. It is that the test’s information condition must match the intended deployment question.

Open the full record

These records expose the observed string, reference, both predictions, edit counts and reference lengths. In the sibling-exposed protocol the candidate prediction is exactly the reference for every target; the baseline is unchanged. Nothing is inferred from a hidden model call.

To reproduce both protocols, run node lab/run.mjs from the project folder. The canonical model (local copy) is shared by the browser and the runnable lab. Its edit-distance routine and resampling checks are included. Now we can ask a different question: how much of a comparison depends on which documents happened to be sampled?

Keep the pair together

Use the disjoint-document output. For document $d$, let $c_d$ be the candidate’s CER and $b_d$ the baseline’s. Define the paired difference:

A negative difference favors the candidate because fewer errors are better. Multiply by 100 to express percentage points, not relative percent improvement. Pairing means each comparison uses the same document and reference, rather than comparing one model’s easy examples with another’s hard ones.

The mean is +10.28 percentage points: the candidate is worse on this synthetic document mean. E2 strongly favors the candidate, C2 ties, and the remaining four favor the baseline. A single average conceals that structure.

Paired differences and their mean are algebraic derivations from LAB-001. This is descriptive evidence about the fixture only.

Resample the evidence you actually have

Imagine sampling document IDs with replacement. A draw such as A, A, C, D, E, F repeats A’s entire candidate–baseline pair twice and omits B. Recompute the mean difference. Repeating this process shows how the empirical comparison changes when the observed documents receive different sampling weights.

Our tiny lab enumerates every ordered six-draw sample: $6^6=46{,}656$. It takes the 2.5th and 97.5th percentiles of their means, using a declared linear-interpolation convention. The resulting illustrative 95% percentile bootstrap interval is −22.50 to +35.28 percentage points.

This interval has no calibrated coverage claim for real OCR: the documents were invented, there are only six, and the resampling model assumes independent, representative document contributions. Exact enumeration removes Monte Carlo noise. It does not make population coverage exact, create new documents or repair contamination.

In a real multi-page sample, retain the document’s pages and both models’ predictions together when the document is the independent unit. If documents share writers or sites, a broader or hierarchical design may be needed. A bootstrap that treats correlated pages as independent asks a different sampling question.

The paired construction is source-backed. The tiny exhaustive enumerator and quantile convention are declared local choices. SciPy bootstrap documentation · CL-012 CL-013

Two ways a result can move

Sample uncertainty: the same fitted models could receive a different set of deployment documents. Our fixed-prediction resampling illustrates that question.

Training variability: a training procedure could produce different fitted models because of initialization, data order, augmentation or other randomness. Rerunning seeds on the same split investigates those choices conditional on that data. It does not supply new test documents.

A real evaluation may need both. Preserve the run-by-document record and describe the analysis rather than pooling every seed–page result as an independent observation. This lab has no stochastic training, so it measures no seed variability.

Bouthillier et al. · CL-014

Inconclusive does not mean equal

An interval spanning zero can be consistent with both improvement and harm under its assumptions. The toy interval is especially limited; we do not turn it into a significance test. More generally, failing to detect a difference is not evidence that two systems are equivalent.

Equivalence asks whether differences are small enough to ignore for a declared purpose. That requires a justified margin set before inspecting the result and an appropriate analysis. We do not implement an equivalence test here.

Lakens on equivalence · CL-016

Statistical significance also says little by itself about practical value. A tiny consistent change may not justify deployment cost; an uncertain estimate may still include consequential harm. Report the effect, uncertainty, error types and decision context together.

ASA statement · CL-015

The average points you toward questions

Return to E2. Its body contains many zeros. The candidate repairs those O-shaped corruptions, while the baseline leaves them. F2 contains many ones and reverses that advantage. The error pattern follows the decoders’ rules; the overall winner depends partly on the document mix.

For real Arabic OCR, inspect substitutions, missing lines, reading order, diacritics, tables and critical numeric fields. Predefine useful slices where possible and report each slice’s size. A low overall CER does not establish that account numbers, units or totals are reliable. An error in one important field can matter differently from an error in prose; that importance needs a stated task policy.

Use development errors to revise the model. If final-test errors reveal a new direction, treat that as a new development cycle and preserve the original final result as historical evidence. Repeatedly looking at the same test is not a fresh confirmation.

The OCR error checklist is a practical application; it is not a claim about observed user-model failures. Preprocessing and leakage · CL-006

Write a claim someone can challenge

A defensible record names the deployment question; collection and split provenance; group and duplicate policy; baseline and candidate versions; fitted preprocessing boundary; fixed scorer and denominators; full predictions; aggregate and paired effects; uncertainty method; and important failures.

In our fixture the claim is narrow: with six disjoint training documents and the declared symbolic rules, the baseline has lower corpus and document-mean CER on A2–F2. Allowing sibling templates changes the candidate’s predictions and reverses that ranking. This proves the stated arithmetic and mechanism. It does not select an OCR system for anyone.

For a real project, finish with the decision the evidence supports—and the evidence still needed. “No detected difference,” “better on this sample,” “equivalent within a margin” and “useful enough to deploy” are distinct conclusions.

Transfer: windows from a disturbance

A power-system fault classifier sees overlapping waveform windows. One disturbance appears at several feeders. A proposed benchmark randomly splits windows, fits a scaler on the full dataset, balances the test set to 50% faults, and reports five training seeds. Deployment will see future disturbances on feeders absent from training.

Choose the evaluation design. Which records must stay together? Which time and feeder boundaries matter? When is the scaler fitted? Which baseline and metrics would make the comparison useful? What uncertainty do five seeds leave unresolved?

Time-aware validation · CL-018

The next useful question is precise: what new, independent evidence could change this decision? That question keeps the model comparison open to correction.