v0.40.0-rc1: mlx: match publisher tokenizer semantics (#18779)
mlx: match publisher tokenizer semantics Honor pretokenizer stage order, split behavior, Unicode boundaries, added-token normalization, and ranked BPE merges. …
What moved
mlx: match publisher tokenizer semantics Honor pretokenizer stage order, split behavior, Unicode boundaries, added-token normalization, and ranked BPE merges. Handle empty added tokens and empty Metaspace input consistently. …
Why it matters
Filed as a public note from Ollama, 2026-10-04. The desk tags this brief LLM / Hardware.
On the record
- Filed from the Ollama official GitHub releases feed on 2026-10-04.
- Primary source host: github.com.
- mlx: match publisher tokenizer semantics Honor pretokenizer stage order, split behavior, Unicode boundaries, added-token normalization, and ranked BPE merges.
Primary source
github.com
github.com
Desk
Logged as brief 065
Logged as brief 065
Cite this brief
v0.40.0-rc1: mlx: match publisher tokenizer semantics (#18779) (Ollama, 2026-10-04) — https://labledgerdesk.pages.dev/b/2026/10/04/v0-40-0-rc1-mlx-match-publisher-tokenizer-semantics-18779/