How uncertain is the comparison?

Keep the pair together

Use the disjoint-document output. For document $d$, let $c_d$ be the candidate’s CER and $b_d$ the baseline’s. Define the paired difference:

A negative difference favors the candidate because fewer errors are better. Multiply by 100 to express percentage points, not relative percent improvement. Pairing means each comparison uses the same document and reference, rather than comparing one model’s easy examples with another’s hard ones.

The mean is +10.28 percentage points: the candidate is worse on this synthetic document mean. E2 strongly favors the candidate, C2 ties, and the remaining four favor the baseline. A single average conceals that structure.

Paired differences and their mean are algebraic derivations from LAB-001. This is descriptive evidence about the fixture only.

Resample the evidence you actually have

Imagine sampling document IDs with replacement. A draw such as A, A, C, D, E, F repeats A’s entire candidate–baseline pair twice and omits B. Recompute the mean difference. Repeating this process shows how the empirical comparison changes when the observed documents receive different sampling weights.

Our tiny lab enumerates every ordered six-draw sample: $6^6=46{,}656$. It takes the 2.5th and 97.5th percentiles of their means, using a declared linear-interpolation convention. The resulting illustrative 95% percentile bootstrap interval is −22.50 to +35.28 percentage points.

This interval has no calibrated coverage claim for real OCR: the documents were invented, there are only six, and the resampling model assumes independent, representative document contributions. Exact enumeration removes Monte Carlo noise. It does not make population coverage exact, create new documents or repair contamination.

In a real multi-page sample, retain the document’s pages and both models’ predictions together when the document is the independent unit. If documents share writers or sites, a broader or hierarchical design may be needed. A bootstrap that treats correlated pages as independent asks a different sampling question.

The paired construction is source-backed. The tiny exhaustive enumerator and quantile convention are declared local choices. SciPy bootstrap documentation · CL-012 CL-013

Two ways a result can move

Sample uncertainty: the same fitted models could receive a different set of deployment documents. Our fixed-prediction resampling illustrates that question.

Training variability: a training procedure could produce different fitted models because of initialization, data order, augmentation or other randomness. Rerunning seeds on the same split investigates those choices conditional on that data. It does not supply new test documents.

A real evaluation may need both. Preserve the run-by-document record and describe the analysis rather than pooling every seed–page result as an independent observation. This lab has no stochastic training, so it measures no seed variability.

Bouthillier et al. · CL-014

Inconclusive does not mean equal

An interval spanning zero can be consistent with both improvement and harm under its assumptions. The toy interval is especially limited; we do not turn it into a significance test. More generally, failing to detect a difference is not evidence that two systems are equivalent.

Equivalence asks whether differences are small enough to ignore for a declared purpose. That requires a justified margin set before inspecting the result and an appropriate analysis. We do not implement an equivalence test here.

Lakens on equivalence · CL-016

Statistical significance also says little by itself about practical value. A tiny consistent change may not justify deployment cost; an uncertain estimate may still include consequential harm. Report the effect, uncertainty, error types and decision context together.

ASA statement · CL-015