Case file · Policy Detective

Evidence board

The file before interpretation

The fixed prompt is x. The behavior row at x is [0.75, 0.25]. The recorded uniform draw is 0.8. Do not use the teacher or loss name to infer the sampled token.

OPD-TOY-001 · invented teaching probabilities; vocabulary [A, B]
Prefix	Student A	Student B	Teacher A	Teacher B	Trace role
`x`	0.75	0.25	0.50	0.50	fixed start · included
`xB`	0.60	0.40	0.90	0.10	student-visited · included
`xA`	0.30	0.70	0.80	0.20	fixed teacher comparison · excluded

fixed prompt xstudent-v0 draw u=.8sample Bvisit xB

Derived draw from a toy fixture. The provenance rule is source-backed. OPD-CL-001 OPD-CL-006 DETECTIVE-CL-013 DETECTIVE-CL-014 DETECTIVE-CL-015 DETECTIVE-CL-016 DETECTIVE-CL-017

Branching diagnosis

Which clue can answer the provenance question?

The choice changes the evidence you reveal, not the frozen trace.

DETECTIVE-EXP-001 · challenge / reveal + trace replay

Choose the evidence before reading the ruling

Awaiting a diagnostic

No evidence record is open.

Commit to the first field you would inspect before reading the ruling.

TRACE-FIG-001 · deterministic replay

Reconstruct the order

The cursor never recomputes or edits the trace. It selects one event from the generated NDJSON record.

Trace event 1 of 19

Case fileTRACE-DETECTIVE-CANONICAL-E001

The deterministic teaching trace begins: natural-log reverse KL in nats; no model is loaded.

Inspect exact event payload

{
  "contractRef": {
    "id": "OPD-CORE-v1",
    "sha256": "9AAD0A2560BFF8696A65B49F5AA72058A75C102AA2C9E17535C40471FFD0E697",
    "toyId": "OPD-TOY-001"
  },
  "epistemicStatus": "invented deterministic teaching fixture; not a trained language model or benchmark",
  "logarithm": "natural",
  "units": "nats"
}

No note flagged

Choose the one note whose reasoning would misclassify another trace.

complete fallback

The full case, with every branch open

Without JavaScript, commitment and replay controls are unavailable. Nothing technical is withheld: all possible diagnostic actions, feedback, trace events, and the worked resolution follow.

DETECTIVE-BRANCH-PROVENANCE · action: provenance

Decisive evidence

Record. behavior student-v0 supplied [0.75, 0.25] at x; u=0.8 selected B; the child prefix is xB.

Ruling. Decisive for the continuation-side state source: B and xB came from the current student behavior policy. The fixed prompt x does not change that completion provenance.

DETECTIVE-BRANCH-OBJECTIVE · action: objective

Insufficient by itself

Record. the declared local objective is reverse KL.

Ruling. Insufficient for provenance. Reverse KL names the objective direction, not the policy that generated the visited prefix.

DETECTIVE-BRANCH-CONFIDENCE · action: teacher-confidence

Insufficient by itself

Record. teacher-v0 supplies [0.9, 0.1] at xB.

Ruling. Insufficient for provenance and correctness. The row says what this teacher predicts at xB; confidence neither identifies the behavior policy nor proves the teacher is right.

Canonical trace

TRACE-OPD-TOY-001 · 19 deterministic events
Seq	Event ID	Type	Evidence summary
1	`TRACE-DETECTIVE-CANONICAL-E001`	`trace.started`	The deterministic teaching trace begins: natural-log reverse KL in nats; no model is loaded.
2	`TRACE-DETECTIVE-CANONICAL-E002`	`rollout.prompt_loaded`	Fixed prompt x is loaded. Prompt provenance and completion provenance are separate fields.
3	`TRACE-DETECTIVE-CANONICAL-E003`	`rollout.prefix_visited`	Position 0 visits x, the fixed starting prefix included in the canonical loss.
4	`TRACE-DETECTIVE-CANONICAL-E004`	`student.behavior_distribution_evaluated`	Behavior policy student-v0 gives [0.75, 0.25] at x for categorical sampling.
5	`TRACE-DETECTIVE-CANONICAL-E005`	`student.token_sampled`	The detached draw u=0.8 falls in B's interval, so the sampled token is B.
6	`TRACE-DETECTIVE-CANONICAL-E006`	`rollout.prefix_visited`	Position 1 visits xB because current behavior policy student-v0 appended B.
7	`TRACE-DETECTIVE-CANONICAL-E007`	`student.loss_distribution_evaluated`	Loss policy student-v0 is evaluated at the same position-0 prefix x.
8	`TRACE-DETECTIVE-CANONICAL-E008`	`teacher.distribution_evaluated`	Teacher-v0 supplies a full-vocabulary distribution [0.5, 0.5] at that same prefix x.
9	`TRACE-DETECTIVE-CANONICAL-E009`	`loss.local_computed`	Reverse KL at x is 0.130812 nats.
10	`TRACE-DETECTIVE-CANONICAL-E010`	`student.loss_distribution_evaluated`	Loss policy student-v0 is evaluated at the student-visited position-1 prefix xB.
11	`TRACE-DETECTIVE-CANONICAL-E011`	`teacher.distribution_evaluated`	Teacher-v0 supplies [0.9, 0.1] at the identical student-visited prefix xB.
12	`TRACE-DETECTIVE-CANONICAL-E012`	`loss.local_computed`	Reverse KL at xB is 0.311239 nats.
13	`TRACE-DETECTIVE-CANONICAL-E013`	`loss.reduced`	The unweighted mean of the two unrounded local losses is 0.221025 nats.
14	`TRACE-DETECTIVE-CANONICAL-E014`	`comparison.fixed_teacher_prefix_loaded`	Comparison branch x -> A -> xA comes from a fixed teacher demonstration and is excluded.
15	`TRACE-DETECTIVE-CANONICAL-E015`	`comparison.student_distribution_evaluated`	For comparison only, student-v0 gives [0.3, 0.7] at xA.
16	`TRACE-DETECTIVE-CANONICAL-E016`	`comparison.teacher_distribution_evaluated`	For comparison only, teacher-v0 gives [0.8, 0.2] at xA.
17	`TRACE-DETECTIVE-CANONICAL-E017`	`comparison.loss_computed`	Reverse KL at xA is 0.582685 nats, but it is not part of the canonical mean.
18	`TRACE-DETECTIVE-CANONICAL-E018`	`parameter_update.boundary`	The reduced loss reaches a separate update boundary; no parameter update is executed or invented.
19	`TRACE-DETECTIVE-CANONICAL-E019`	`trace.completed`	Case resolved: current-student continuation provenance and reverse-KL objective are recorded as separate axes.

The review note to flag

This run is on-policy because it uses reverse KL.

Misleading. The student-generated xB prefix establishes the completion-side provenance; reverse KL is a separate objective choice.

Read the machine record

The teacher sees the path the student actually visited

At xB, both student and teacher distributions are evaluated at that same prefix. The teacher is not asked to pretend the student took A.

Metaphorical recorder with blank plates, one inspection lens, and a separate adjustment tool; the event table supplies the technical meaning. — DETECTIVE-FIG-003Metaphor only: recording, inspection, and adjustment are visually separate. The trace below defines the actual operations.

[sample]

Discrete draw

u=.8 → B. Detached in this GKD-style toy.

[visit]

New state

x + B → xB. Provenance: current student-v0.

[teacher]

Same prefix

Teacher-v0 supplies its row at xB.

[boundary]

No invented update

The reduced loss reaches an update boundary; parameters are not modeled.

Canonical GKD detaches student sampling; that local estimator is not every OPD method. OPD-CL-002 OPD-CL-004 OPD-CL-005 OPD-CL-007

DETECTIVE-CALC-001 · derived

Only now do we work the loss

The trace has already answered whose states these are. The equation answers a different question: how far apart are the two declared distributions at each included prefix?

D_{KL}(p_S\,\|\,p_T;s)=\sum_{v\in\{A,B\}}p_S(v\mid s)\ln\frac{p_S(v\mid s)}{p_T(v\mid s)}

Read it as: at one visited prefix s, weight each natural-log student/teacher ratio by the student's probability. The unit is nats. Teacher and student rows must refer to the same prefix and aligned vocabulary.

Included · prefix x

0.130812 nats

Token	Student	Teacher	Weighted log-ratio
A	0.75	0.5	0.304098831081123
B	0.25	0.5	-0.173286795139986

Included · prefix xB

0.311239 nats

Token	Student	Teacher	Weighted log-ratio
A	0.6	0.9	-0.243279064864899
B	0.4	0.1	0.554517744447956

Excluded comparison · prefix xA

0.582685 nats

This is a valid reverse-KL calculation on a fixed teacher branch. It does not enter the current-student canonical reduction.

\bar{D}=\frac{D(x)+D(xB)}{2}=0.2210253577620973\ldots\ \text{nats}

Display: 0.221025 nats. Average the unrounded locals, then round once. Averaging the displayed values gives 0.2210255 and can produce a different six-decimal result.

Derived from the invented OPD-TOY-001 fixture; not model evidence. DETECTIVE-CL-013 DETECTIVE-CL-014 DETECTIVE-CL-015 DETECTIVE-CL-016

The misleading note

This run is on-policy because it uses reverse KL.

Misleading. The student-generated xB prefix establishes the completion-side provenance; reverse KL is a separate objective choice.

sourced OPD-CL-003

DETECTIVE-FIG-004 · transfer comparison

Four files where the label is not enough

Each ruling asks for one missing implementation contract. These are invented diagnostic cases, not observations from a trainer.

01 · toy

STALE BEHAVIOR

stale relative to update policy

Evidence: behaviorPolicyId=student-v0; updatePolicyId=student-v1.
Ruling: The prefix is student-generated, but it is not strictly current-policy data for student-v1 without a declared correction/freshness rule.
Needed next: importance correction, clipping, or allowed policy-lag rule.

02 · toy

TOPK

full-vocabulary KL not computable

Evidence: teacher returns renormalized top-k entries and does not expose omitted tail mass.
Ruling: Do not equate the record with full-vocabulary KL or a sampled-token log-ratio.
Needed next: k, tail-mass treatment, renormalization, and estimator definition.

03 · toy

TOKENIZER

mapping required

Evidence: studentTokenizerId differs from teacherTokenizerId even though decoded text matches.
Ruling: Do not zip token-probability vectors by index; the next-token event spaces are not aligned by the supplied evidence.
Needed next: explicit vocabulary/tokenizer mapping or another comparable event space.

04 · toy

WRONG BRANCH

local feedback only

Evidence: after an earlier wrong turn, the teacher assigns high probability to a locally plausible next token at the already-wrong prefix.
Ruling: Local agreement at the visited prefix is not causal/process supervision and does not retroactively certify the earlier branch.
Needed next: a separately justified process or causal supervision signal.

The distinctions draw on the cited primary methods; the concrete records are toys. DETECTIVE-CL-008 DETECTIVE-CL-009 DETECTIVE-CL-010 DETECTIVE-CL-011 DETECTIVE-CL-012

Transfer desk · no score, no mastery claim

Change one axis at a time

Answer in two sentences: first name the divergence; then name the state source. Open the worked resolution only after you have committed to both.

OPD-XFER-A

Reverse objective, fixed branch

A learner uses reverse KL on a fixed teacher-generated xA prefix. Is the divergence reverse KL, and is the state sampling on-policy? Explain both axes separately.

Open worked resolution

The divergence is reverse KL. The xA state is fixed and teacher-generated, so its state sampling is not current-student on-policy.

OPD-XFER-B

Forward objective, student branch

A learner samples B from the student, queries the teacher at xB, and computes forward KL. Is this on-policy distillation? Explain why divergence direction does not change the state source.

Open worked resolution

Yes with respect to the completion-side state source: B and xB came from the current student. Forward KL is the independent objective choice.

The compact rule: trace the behavior policy that generated the visited prefix. Then inspect teacher query, feedback granularity, objective, reduction, and update semantics as separate fields.