Skip to main content
Engineering notes 9 min read

When the Evaluation Is the Bug

I shipped a passage splitter behind 9,929 green tests, ran the evaluation with it on and with it off, and got identical numbers both times. Three readings from one harness: a change that never ran, a corpus that counted itself twice, and my fix for that, which collapsed 300 recordings onto 24.

Cited chat answering a question from a local recording library
The retrieval path these eval runs were grading — answers with citations back to the source.

I shipped a change to the way my app cuts transcript passages before it hands them to a language model. The idea was small. Some passages hold more than one speaker, and a question about one person could drag in a neighbour’s line, so I split those passages at speaker boundaries.

It shipped green: 9,929 tests, 100% branch coverage, every guard the repository runs on a commit.

Then I ran the evaluation twice — once with the change switched on, once with it off — and every number came back the same. Not close. The same. Not one test question got a different answer.

My first draft of the write-up said “zero failing checks”. That is true, and it is worth nothing, because identical numbers have three different causes and I could not tell which one I had. The change worked and disturbed nothing. Or the change did nothing. Or the evaluation never asked a question the change could affect.

I hit this building the library chat in Kajo Voice, a local-first audio knowledge base where a retriever answers questions over your own recordings and cites them.

In one day I opened four investigations into that evaluation, and all four found the instrument broken rather than the pipeline. Across the project I have catalogued eleven distinct measurement defects. Below are three readings from that harness — two from that day, one from later. My first guess was wrong all three times, and the third reading is what my fix for the second one did.

What the harness reports

Two paragraphs of setup, because all three readings lean on them.

The harness holds a fixed set of test recordings and a fixed list of questions about them. Each question was written against one particular passage — the passage that answers it. A run asks every question and ends with a report of named checks. A check is a share of questions that behaved: the fraction of person-scoped questions where every citation came from the person the question named, the fraction of deliberately unanswerable questions the model correctly declined. Each one has a threshold, so it passes or fails.

Two runs get compared by putting their reports side by side. That report is the instrument. Everything below is a story about the instrument, not about retrieval being bad.

Reading one: two runs, every number identical

My splitter found speaker boundaries by looking for lines that open with a bracketed speaker id — the anonymous placeholder the transcription stage assigns before anyone has named anybody, [S1]:. Nine unit tests covered it. All nine were green.

Nothing in the summary separates the three worlds at the top of this post: every check reading the same number is consistent with all of them at once. Going finer-grained turned up two more things that resemble evidence and are not. The number of passages retrieved per question matched across the runs; so did the list of citations each answer produced. Both match for innocent reasons in all three worlds.

What settled it was the artifact the change exists to rewrite — the passage text itself, the literal string handed to the model. I dumped it from both runs and compared it byte for byte.

It was the same string. Character for character, on every question, in both runs.

A function whose entire job is to rewrite a string, handed that string, returning it unchanged, has not run.

That has one explanation. An unchanged metric has many, and the asymmetry is the whole technique. Metrics aggregate, so they come out equal for a dozen honest reasons: questions that never exercised the path, effects that cancel, a threshold too coarse to move. The artifact does not aggregate. It is the literal output of the code under test, and a transform that leaves its input untouched did not execute.

The cause took a minute to find once I knew what I was looking for. The last step of retrieval rewrites those placeholders into whatever the user has called that speaker — [S1]: becomes [Mr. Cortez]: — and it runs before the stage my code sat in. Nothing that reached my splitter carried a placeholder. The pattern matched nothing, every passage went through untouched, and the pipeline behaved exactly as it had before I wrote a line.

Which leaves the nine green tests. They were green because I wrote them and fed them the shape I believed the function would be handed: text with bracketed placeholder ids. Given that shape the function is correct. It was never given that shape. A unit test you write for your own change tests your model of the caller, not the caller. The fix is cheap and I should have done it first: capture one real artifact the previous stage produced and assert against that. When I did, the speaker labels in real passages were Practitioner, Client R, Speaker 1, 韩梅, 三宅 郁 — display names, every one, and not one of them matching the pattern.

The same investigation turned up a second reason the comparison was empty: the harness carries a quick question set and a full one, four of the six questions about who said what live only in the full set, and I had run the quick one.

Reading two: the answer said 8, the test data said 6

This one announced itself as two questions failing a check for being right. One check compares a number stated in an answer against the number the test data records as correct. “How many sessions mention the brother” answered 8. The test data said 6.

The reflex is to suspect the model, and the reflex is wrong: pure counting questions never reach the model. They are answered by a database query over the recordings. So, the query — except the query was not wrong either. “Brother” really did appear in 8 entries. Two of those entries were the same recording, loaded twice.

The harness builds its corpus from a list of test recordings, filtered down to whichever folders the run covers. Each loaded entry got an id assembled from two numbers, eval_<folderIndex>_<fileIndex>, and both were positions in that filtered list — which folder among the folders being loaded, which file inside that folder.

Loading a corpus is slow, because every recording has to be transcribed first, so the database gets reused between runs — and the written procedure for running an evaluation encouraged exactly that. Reuse only works if a recording arrives carrying the same id every time. Same id, and the second load overwrites the row already sitting there. Different id, and it is inserted beside it.

Change which folders a run covers, and every position after the change shifts. Nothing about the recording had changed: same file, same name, same transcript, same content. Only its position in a filtered list had moved — and the position was the identity.

entries=62  distinct transcripts=55  duplicated=7

Seven recordings existed in that corpus twice. Every count taken over it was inflated, and the check comparing counts was faithfully reporting that: the answer 8 was correct about the database, and the database was wrong about the world. Two questions failed for telling the truth about a corpus that had quietly doubled part of itself.

An id derived from a position — a row number, an index into a filtered list, the order things happened to arrive in — is an address, not an identity. It is stable exactly as long as nothing above it moves, which is the condition the next experiment exists to break. So I derived the id from the folder and transcript name instead, neither of which moves when the folder selection does. On a corpus loaded once, the counting check went to zero error and the share of deliberately unanswerable questions correctly declined went to 1.0. No pipeline change at all. The duplication had also inflated every absolute number I quoted that day.

That fix shipped, and it did exactly what it was written to do.

Reading three: the answer said 15, the test data said 43

Same check, a later run, a question about how many recordings mention a warehouse. The test data says 43. The pipeline said 15.

Nothing failed — no exception, no empty result, no zero. 15 is a thoroughly plausible number of recordings to mention a warehouse, every stage that produced it was working, and 15 was exactly right about the corpus it actually had.

One folder in that corpus is named archive, and its manifest lists 300 files which are only 24 distinct paths: archive/03.json appears 20 times, verbatim. That is deliberate — it is how the fixture simulates a 300-recording library without shipping 300 transcripts. The positional id, whatever else was wrong with it, gave each of those 300 entries a distinct id, so the database really held 300 rows. Name-derived ids collapsed all 300 onto 24, because they share a name.

Deduplicating a reused corpus is exactly what I had asked for, and it could not tell my accidental duplicates from the fixture’s deliberate ones. “Derive identity from content” is the right rule and I would write it again — it also assumes something I never said out loud: that content is unique. A fixture built to repeat content on purpose breaks that assumption, and because the deduplication and the count were both doing their jobs, it broke as a believable number instead of an error.

The fix for the fix is small: a repeat gets an occurrence suffix, counted within one folder’s own file list — stable when the folder selection changes, which was the original bug, and unique, which was what I had just broken.

What every run prints now

Three readings, and the retrieval pipeline was innocent in all three. What was defective was the thing I was reading it with — a change that never executed, then a corpus that counted part of itself twice, then a corpus that collapsed the fixture’s deliberate repeats down to one row each.

One habit makes all three cheaper, and it is the first thing I would install in a new harness. Every run summary now opens with the commit it ran, a flag for uncommitted changes, and a hash of the working diff. Running with uncommitted changes is normal here, and two runs carrying the same commit and the same diff hash provably ran the same code. Without that, no improvement claim is falsifiable.

None of this makes a retrieval pipeline better. It makes its problems findable, which is the harder half.

Waitlist

Be first to know

Kajo Voice launches on October 10, 2026. Leave your email and we will send one message that day — while Lifetime is $49 for the first two months, before it goes to $99.

A confirmation now, then one email at launch. Privacy policy

Related articles