Skip to main content
Engineering notes 8 min read

My Chunker Counted Characters and Called Them Tokens

The function deciding where every chunk of every transcript ended was a characters-per-token estimate nobody had compared with a tokenizer. It read 0.96 of the truth in Latin script and 0.48 in Devanagari — and the multilingual folder scoring 12 of 12 held ten documents of one chunk each.

Recordings being imported and indexed into a local library
Every imported recording is chunked and embedded on the device, once, at this point.

Somewhere in my pipeline a small function decided where every chunk of every transcript ended — a chunk being the unit a retrieval system stores, searches and eventually hands to a model. The function was this:

ceil(ascii / 4) + ceil(nonAscii / 2)

An estimate of how many tokens a string is worth — four ASCII characters to a token, two of anything else — written as a stand-in for a real tokenizer that was not at hand, and run against a target of 256 tokens per chunk. Nobody had ever compared it with the tokenizer it was standing in for.

Meanwhile three of my working sessions had argued, from first principles and without ever settling it, about whether each stage of the pipeline should have its own tokenizer.

I compared the estimate with the truth instead. 388 real chunk texts, pulled out of a saved evaluation run — the exact strings the product had been indexing, not samples I typed. n is how many chunks of that script the sample held. The ratio is the estimate divided by the true count, so under 1 means the chunks came out too big.

script n estimate ÷ real
Latin 370 0.96
Arabic 7 1.10
CJK Han 8 0.68
Devanagari 3 0.48

nonAscii / 2 assumes two characters per token. For Han it is nearer 1.5, and for Devanagari nearer 1. So “256 tokens” meant 256 tokens in English, about 375 in Chinese, and about 530 in Hindi. Those chunks run 1.5 to 2 times their intended size.

Pulling those strings took minutes. What made them worth anything is that they were real indexed text and not a sample I typed, which would have been in the script I think in.

I hit this building the retrieval index for Kajo Voice, a local-first audio knowledge base that transcribes many languages and indexes all of them on the user’s own machine.

Scoring reweights, truncation discards

An oversized chunk costs nothing, necessarily. It depends entirely on what the next stage does when it gets one, and the stages downstream of my chunker do two different things with length.

The keyword index scores by it. A long document gets a slightly lower score than a short one for the same match, on the reasoning that a word appearing once in a long document tells you less about that document than the same word appearing once in a short one. That adjustment is length normalisation. But every word of the long document is indexed, and every word can still be matched. Hand that stage a chunk at twice its intended size and all you have done is change a weight.

The models truncate by it. The cross-encoder that reranks candidates — bge-reranker-v2-m3, a model that reads the question and one candidate chunk together and scores the pair — takes at most 512 tokens, and whatever falls past that limit does not exist as far as the model is concerned. No error. No warning. No penalty on the score, which would at least be visible. The tail is simply never read.

One of the 388 chunks measured 756 real tokens. Every time that chunk was ranked, everything past its 512th token was cut away and the model scored what was left as though it were the whole thing.

Correction (September 21): the reranker half of this section is wrong. 512 tokens is the length BAAI trained bge-reranker-v2-m3 at, not its window: the model reports 8,192, Kajo opens its ranking context at that size, and an over-long input would throw an error rather than be cut. The 756-token chunk was ranked whole. The undercount still mattered, one stage over: the chat model reads transcripts in windows sized by the same estimate, and a Chinese window budgeted at 4,500 tokens was really about 7,360 against an 8,192-token sequence.

That asymmetry also buys the one assertion worth adding today, whatever you decide about the chunker: at index time, check that no chunk exceeds the limit of the model that will read it. It needs no tokenizer inside the chunker and no change to how boundaries are chosen — it is one check after the fact, and it stops a corpus quietly growing another silent truncation.

The argument was two arguments

Scoring reweights. Truncation discards. That distinction settles what three sessions had been arguing about, once you notice the question is two questions.

Counting — how long is this text, to this model? Every stage needs its own count, and there is no choice about it. The embedding model’s count is the one the chunker must respect. The keyword engine counts for its own scoring. The reranker counts against its own input limit. One tokenizer per job is already true here, whether or not anyone decided it.

Segmenting — where do the units break? There is exactly one set of boundaries, because there is one stored row per chunk. One per job is impossible.

So the single segmentation decision belongs to the stage with a hard truncation limit, because that is the stage that loses content rather than merely reweighting it. My embedder and my reranker both have one. The keyword engine has none — an over-long document is indexed in full and just scored a little lower — so keyword search gets no vote on chunk size.

Which leaves the obvious fix: call the embedding model’s real tokenize() instead of estimating. That makes the chunker asynchronous and model-dependent, which is the genuine cost, and it is worth paying for an exact answer over a constant wrong by 2x on two scripts. Indexing already loads the embedding model in order to embed those chunks; the tokenizer had been sitting in the same process the whole time. Targeting 256 embedder tokens then keeps every chunk far inside every model’s window.

Retuning the constant per script is the alternative, and it is worse than it sounds: Arabic already over-estimates at 1.10, so one blanket change makes one script worse in order to fix two, on samples of 3 to 8 chunks. Re-tuning a wrong heuristic is how you end up with a second wrong heuristic and a table of exceptions.

Per-job text, unlike per-job boundaries, is worth having. A chunk row here carries display text, embedding text, and a separately transformed string for the keyword index. Give keyword search its own text rather than its own boundaries.

The best folder in my corpus proved nothing

Two reasons none of this ever looked wrong. The first is boring: nearly all my content is Latin script, where the estimate reads 0.96 of the truth, so chunk sizes looked right nearly all of the time.

The second is the one worth the post.

My test corpus is a set of recordings with a fixed list of questions about them. Every question was written against one particular passage — the one that answers it — and the headline metric asks whether that passage reached the prompt. The folder of non-English recordings scored 12 of 12. Twelve questions, twelve designated passages retrieved. It was the best folder in the corpus, ahead of every English one, and for a while I read that as evidence there was nothing to fix.

Then I looked at what was actually in the index. Of 3,214 chunks, 3,202 were English. Eleven non-English entries in the entire corpus, and ten of them were a single chunk each.

Take a recording short enough to fit into one chunk and ask what defects it is even capable of showing. It has no chunk boundary, so a boundary in the wrong place cannot happen to it. It has no siblings, so there is no ranking among one document’s chunks to get wrong. It has one length, so length normalisation has almost nothing to do. And it is short, so it is nowhere near the reranker’s limit, so the truncation above can never bite.

Twelve questions over eleven documents, ten of which held one chunk. That is not weak evidence about chunk sizing. It is no evidence about chunk sizing — nor about ranking, tokenizer choice or Unicode normalisation, the four things three sessions had reasoned about at length. The best-scoring folder in my corpus had been retiring hypotheses it was structurally incapable of testing.

A green metric over a corpus that cannot exhibit the defect is not a null result. It is not a measurement.

The fixture that had to replace it

That set the bar for what I built next: hard, not merely long. A long easy recording scores 12 of 12 as well, and teaches exactly as little.

The recipe came from the English long-form folder, the one that does catch things: long sessions, invented distinctive proper nouns, exact figures with a near-miss distractor in the same sentence (“716 sampling points, not 406”), control questions beside the answerable ones (no answer anywhere in the recording, to catch a model inventing one) — and sized to produce 8 to 15 chunks per recording rather than one.

I kept the folder that scored 12 of 12. It is a fine test of transcribing and retrieving non-English audio. It was never a test of chunk sizing, and I had been reading it as one.

Waitlist

Be first to know

Kajo Voice launches on October 10, 2026. Leave your email and we will send one message that day — while Lifetime is $49 for the first two months, before it goes to $99.

A confirmation now, then one email at launch. Privacy policy

Related articles