Who crossed the split?

A new page can carry an old answer

Every invented document has two pages with the same body and a different page suffix. A1 is 110-1; A2 is 110-2. They are different strings. Yet seeing the labeled first page can disclose almost the entire answer to the second.

If deployment means unseen documents, hold all members of an evaluation document out of training. If deployment means unseen writers, organizations or templates, document grouping alone may still share information that your claim requires to be new. The right unit follows the question and the collection’s provenance.

Previous pages may be legitimate inputs in a product that explicitly supplies them at prediction time. Then evaluate that product under that information policy. Our lab labels sibling exposure inappropriate because its question requires new documents.

Grouped cross-validation · CL-003

Check identity, similarity and origin

An exact file hash can catch identical bytes across partitions. It will miss a re-encoded image, a different crop, or a new scan of the same page. Text equality can catch some repeated content; near-match retrieval can surface more candidates. Review those candidates alongside document IDs, source records and collection dates.

For OCR, inspect whether a “different” PDF is a second copy of the same report or whether a preprocessed crop came from a held-out original. Define duplicate policy before final evaluation and keep related records together when the deployment question requires it. Record what the search could not detect.

Near-duplicate research on image benchmarks motivates this audit, but it does not validate a universal OCR detector. No matches found is not proof of independence.

Barz & Denzler · CL-004

The boundary also surrounds preprocessing

A clean file split can still leak. Suppose feature selection uses all labels before the split, or an imputer estimates its values from all records. Held-out information has already shaped the transformation, even if the final estimator sees only training rows.

1. Choose the split units and assign partitions.
2. Fit learned transformations and the model using only the training part of that fold.
3. Apply those fitted transformations to development examples; choose settings there.
4. Freeze the procedure, then evaluate on the protected test set.

When cross-validating, repeat fitting inside each training fold. A pipeline helps preserve that order, but it cannot choose valid group IDs or notice that a feature was unavailable at deployment time. Those are design decisions.

Preprocessing and leakage · CL-005

Kapoor and Narayanan’s leakage taxonomy is broader than duplicate files: it includes broken separation, illegitimate features, and test data that do not support the claimed distribution. We will now isolate one transparent shortcut.

Kapoor & Narayanan · CL-002