A vocabulary that lies about itself

Most scanners stop at the tensor table. But a GGUF file also carries the entire tokenizer — tokenizer.ggml.tokens, a plain list of strings where each string's position in the list is its token ID — and downstream pipelines reconstruct the tokenizer straight from these fields to decode whatever the model outputs back into text a human reads.

We tampered with two things in that list. First, the string 'hello' appears at two different positions — two different token IDs for the same surface text, which is not something a correctly produced vocabulary ever does; it breaks the 1:1 string<->ID mapping every detokenizer assumes, and can make one of the two IDs decode to text the model never actually intended. Second, one token string has a Unicode U+202E RIGHT-TO-LEFT OVERRIDE character embedded in it — invisible in the raw bytes, but it silently reorders however the surrounding text renders in a terminal, a chat UI, or a log viewer, without the model itself ever being aware of it.

Neither of these touches a single tensor weight — the model's actual computation is untouched. That's exactly why they're easy to miss: a scanner that only validates tensor structure has nothing to say about them. llmscan parses tokenizer.ggml.tokens directly and flags both classes: duplicate vocabulary entries, and bidi-override / control-character / malformed-UTF-8 bytes hidden inside token strings.