Receipts, because a naked claim like this deserves them:
What breaks: common tokenizers (BERT/WordPiece-style, some BPE vocabs) silently map out-of-vocab tokens to [UNK] instead of erroring. When an entity is emoji, CJK, a SKU code, or a pharma brand name, it can get reduced to mostly/all [UNK] tokens. Multiple distinct inputs that all reduce to the same [UNK]-heavy token sequence then embed to near-identical vectors — so your retriever can't tell them apart. No exception, no warning, just wrong nearest-neighbors.
The 60-second repro:reproduce.py in the repo is standalone — no GPU, minimal deps. It tokenizes a handful of adversarial strings (emoji clusters, CJK terms, SKU-style codes) and shows the collapsed token/embedding overlap directly.
The killshot:killshot_chromadb.py runs the failure end-to-end on a stock ChromaDB + all-MiniLM-L6-v2 setup (the default most RAG tutorials use) with a tiny emoji reaction-log dataset. Query for a specific reaction and it retrieves the wrong entries 3/3 — full miss, not a ranking nit.
Methodology: this isn't a single anecdote. The paper pre-registers falsification tests t2 through t35 covering different entity classes (emoji, CJK, SKU/alphanumeric codes, pharma names) and different tokenizer/model combos, so the claim is falsifiable rather than vibes — if a test passes, it's reported as a pass, not buried.
Receipts, because a naked claim like this deserves them:
What breaks: common tokenizers (BERT/WordPiece-style, some BPE vocabs) silently map out-of-vocab tokens to
[UNK]instead of erroring. When an entity is emoji, CJK, a SKU code, or a pharma brand name, it can get reduced to mostly/all[UNK]tokens. Multiple distinct inputs that all reduce to the same[UNK]-heavy token sequence then embed to near-identical vectors — so your retriever can't tell them apart. No exception, no warning, just wrong nearest-neighbors.The 60-second repro:
reproduce.pyin the repo is standalone — no GPU, minimal deps. It tokenizes a handful of adversarial strings (emoji clusters, CJK terms, SKU-style codes) and shows the collapsed token/embedding overlap directly.The killshot:
killshot_chromadb.pyruns the failure end-to-end on a stock ChromaDB + all-MiniLM-L6-v2 setup (the default most RAG tutorials use) with a tiny emoji reaction-log dataset. Query for a specific reaction and it retrieves the wrong entries 3/3 — full miss, not a ranking nit.Methodology: this isn't a single anecdote. The paper pre-registers falsification tests t2 through t35 covering different entity classes (emoji, CJK, SKU/alphanumeric codes, pharma names) and different tokenizer/model combos, so the claim is falsifiable rather than vibes — if a test passes, it's reported as a pass, not buried.
Repo: https://github.com/Jaybell31/dreamwalk/tree/master/tokenizer-collapse-paper (reproduce.py, PAPER_DRAFT.md, killshot_chromadb.py)
Happy to answer questions on the tokenizer internals or the eval design.