From score to decision

Inspect what failed, preserve what was tested, and narrow the claim.

The average points you toward questions

Return to E2. Its body contains many zeros. The candidate repairs those O-shaped corruptions, while the baseline leaves them. F2 contains many ones and reverses that advantage. The error pattern follows the decoders’ rules; the overall winner depends partly on the document mix.

For real Arabic OCR, inspect substitutions, missing lines, reading order, diacritics, tables and critical numeric fields. Predefine useful slices where possible and report each slice’s size. A low overall CER does not establish that account numbers, units or totals are reliable. An error in one important field can matter differently from an error in prose; that importance needs a stated task policy.

Use development errors to revise the model. If final-test errors reveal a new direction, treat that as a new development cycle and preserve the original final result as historical evidence. Repeatedly looking at the same test is not a fresh confirmation.

The OCR error checklist is a practical application; it is not a claim about observed user-model failures. Preprocessing and leakage · CL-006

Write a claim someone can challenge

A defensible record names the deployment question; collection and split provenance; group and duplicate policy; baseline and candidate versions; fitted preprocessing boundary; fixed scorer and denominators; full predictions; aggregate and paired effects; uncertainty method; and important failures.

In our fixture the claim is narrow: with six disjoint training documents and the declared symbolic rules, the baseline has lower corpus and document-mean CER on A2–F2. Allowing sibling templates changes the candidate’s predictions and reverses that ranking. This proves the stated arithmetic and mechanism. It does not select an OCR system for anyone.

For a real project, finish with the decision the evidence supports—and the evidence still needed. “No detected difference,” “better on this sample,” “equivalent within a margin” and “useful enough to deploy” are distinct conclusions.

Transfer: windows from a disturbance

A power-system fault classifier sees overlapping waveform windows. One disturbance appears at several feeders. A proposed benchmark randomly splits windows, fits a scaler on the full dataset, balances the test set to 50% faults, and reports five training seeds. Deployment will see future disturbances on feeders absent from training.

Choose the evaluation design. Which records must stay together? Which time and feeder boundaries matter? When is the scaler fitted? Which baseline and metrics would make the comparison useful? What uncertainty do five seeds leave unresolved?

Work through the changed case

Trace each window back to its disturbance and feeder. Windows from the same disturbance must not leak across training and evaluation when the claim is about new disturbances. An unseen-feeder split alone may still share the same event through another feeder. Future deployment also requires a time boundary; a group label does not enforce chronology.

Construct partitions that honor the joint event, time and feeder claim, and report if the available data cannot support it. Fit the scaler within training only. Check that every feature exists at the actual prediction time. Choose an existing detector or a simple stated rule as a baseline and give both models the same eligible event record.

Report missed faults and false alarms with denominators and a threshold fixed on development data. If the evaluation is artificially balanced, its predictive-value interpretation may not match deployment prevalence. Keep event-level outcomes visible so overlapping windows do not dominate the apparent sample size. The exact metric and acceptable tradeoff need a domain decision; no threshold is prescribed here.

Five seeds describe some training variability on the chosen dataset. They leave uncertainty about new events, feeders and deployment conditions. Group-aware uncertainty can only reuse the evidence present; it cannot manufacture missing disturbance types or repair a contaminated split.

This is a design exercise inferred from grouping, temporal validation and information-availability principles. It is not a tested power-system result or operating recommendation.

Time-aware validation · CL-018

The next useful question is precise: what new, independent evidence could change this decision? That question keeps the model comparison open to correction.