What the score counts

The denominator and the text policy are part of the result.

Count edits, then divide

Take target page A2. Its reference is 110-2; the baseline produces 11O-2. One substitution changes the letter O into the digit 0. The five-character reference therefore gives an error rate of one fifth, or 20%.

Let $r$ be a reference string, $\hat r$ a predicted string, and $N=|r|$ the reference character count. Let $S,D,I$ count substitutions, deletions and insertions in a minimum-edit alignment. With unit cost per edit:

\operatorname{CER}(r,\hat r)=\frac{S+D+I}{N}

The denominator counts the reference, not the prediction. Insertions can make CER exceed 100%: reference a and prediction aaaa need three insertions, giving 300%. Our scorer rejects an empty reference because this simple ratio would divide by zero; a production scorer needs an explicit policy for such cases.

Definition is source-backed; examples are invented arithmetic. JiWER character scoring · CL-008

What is a “character” here?

Our lab counts Unicode code points, preserves punctuation and whitespace, and performs no normalization. The page suffix counts too. A visible mark can contain several code points, so this is a declared scoring convention, not a universal definition of a human-perceived character.

For Arabic OCR, decide whether diacritics, punctuation, spacing and letter distinctions must be preserved for the task. Unicode canonical normalization can make equivalent encodings comparable; deleting diacritics is a separate task policy. Silently changing either policy can change the score without changing a prediction.

Keep the raw outputs and apply the same frozen transformation to both systems and references. A predetermined normalization rule is different from learning a vocabulary, imputer or scaler from held-out data.

Unicode normalization · CL-009

Characters or documents?

Our six target pages contain 49 reference characters. In the disjoint-document protocol, the baseline makes 17 edits and the candidate makes 20. Adding edits first produces 34.69% versus 40.82% corpus CER. These are deterministic outputs of the invented fixture, not OCR benchmark results.

Let $e_d$ be total edits and $n_d$ reference characters for document $d$ . With $G$ documents:

\operatorname{CER}_{\mathrm{corpus}}=\frac{\sum_d e_d}{\sum_d n_d}

\operatorname{CER}_{\mathrm{doc}}=\frac{1}{G}\sum_d \frac{e_d}{n_d}

Corpus CER gives a longer document more influence. The document mean gives each document one vote after combining its pages. In this fixture the document means are 31.11% versus 41.39%. Neither weighting is automatically fairer: choose whether your decision is about a typical reference character or a typical document, and state how the sample represents that population.

Here each document contributes one target page. In a real collection with many pages per document, ten pages can improve measurement of one document without creating ten independent documents.

The weighting relationship follows directly from the definitions. Numbers come from LAB-001. We use document means for the paired uncertainty example.