Better at what?

Write the sentence before the number

You replace an OCR system and the reported character error rate falls. Before celebrating, ask one question: what was new to the model when it produced those predictions? A fresh page from a familiar document and a fresh document from an unfamiliar archive are different tests.

For this guide, our question is concrete: which decoder makes fewer character errors on documents whose other pages were unavailable during training? Later we will add uncertainty and ask whether the errors matter. We will use six invented documents so that every prediction can be inspected.

A useful evaluation sentence names the population, the information available at prediction time, the independent unit, the score and the intended decision. For Arabic document extraction, you might specify new documents from a defined collection, scanned pages only, grouping by originating document or writer, a declared transcription policy, and a decision about whether to replace the current system. Those choices need evidence from your actual collection; this guide supplies no such evidence.

The evaluation sentence is our synthesis of the paper’s claim and information-boundary principles. Kapoor & Narayanan · CL-001

Make the comparison repeatable

Start with the system someone could use today: the current OCR pipeline, a simple decoder, or a published method you can rerun. Record its checkpoint, inputs, preprocessing, output conversion and failure policy. Compare both systems on the same eligible items, with the same references and scorer. Report different tuning budgets or extra training data when they are part of the comparison.

A number copied from another paper may provide context. It becomes a matched baseline only when the data version, split, scorer and relevant conditions align. An OCR timeout must not quietly disappear from one model’s denominator; define how missing outputs enter scoring before running the comparison.

“Candidate minus baseline” is incomplete. Add: on which items, under which information boundary, scored in which way?

Matched comparison checklist: an application of controlling sources of benchmark variation. Bouthillier et al. · CL-007

A short refresher: learning and measuring

A training loss supplies the numerical objective used to fit a model. An evaluation metric measures a chosen property of its predictions. They can coincide, but optimizing one does not automatically improve the other—or performance on new examples.

Use training data for fitting and development data for model choices, including thresholds, normalization choices and early stopping. Keep a final test set protected until the comparison procedure is settled. If you use its errors to choose the next model, those examples have entered development; a revised claim needs fresh protected evaluation.

Our tiny experiment is a teaching bench, not an untouched final test. We inspect it freely because we make no real model-selection claim from it. First, we need to agree on how its mistakes are counted.

Preprocessing and leakage · CL-006