Tokenizer Expansion for Multilingual LLMs

An abstract-grounded analysis of an in-place recipe that expands a pre-trained LLM tokenizer, preserves source tokens, and reduces multilingual token fragmentation.

TL;DR — The abstract presents an in-place tokenizer expansion recipe for pre-trained LLMs: continue the existing BPE merges on a multilingual corpus, preserve most source tokens, initialize new token embeddings from the mean of their source sub-token embeddings, then adapt with embedding-only training followed by full-model continued pre-training. Applied to a continued pre-trained LFM2-8B-A1B checkpoint, the recipe helps produce LFM2.5-8B-A1B with a 128K tokenizer and reports roughly 2.4× fewer Hindi tokens, 2.6× fewer Vietnamese tokens, up to 4.0× fewer Thai tokens, and an estimated 2.2×–3.7× per-character decode speedup across reference devices.

Problem: tokenizer drift after priorities shift

The paper addresses a practical failure mode of fixed LLM tokenizers: they encode the language mix and deployment assumptions present at the start of pre-training. If a model is later used in languages that were not well represented then, common words can be split into many more tokens, making the same visible text more expensive to generate.

The proposed answer is tokenizer expansion rather than tokenizer replacement. The authors describe an in-place recipe that preserves most of the old token interface, adds multilingual tokens, and adapts the model without retraining from scratch.

Full analysis, extracted claims, numerical results, entity graph, FAQ, related work, applications, and BibTeX are available via x402 micropayment.