LAB LEDGER DESK
00:00:00 UTC2026-11-17
Official sources only·Nothing invented

Ollama · Note ·

v0.40.0-rc1: mlx: match publisher tokenizer semantics (#18779)

mlx: match publisher tokenizer semantics Honor pretokenizer stage order, split behavior, Unicode boundaries, added-token normalization, and ranked BPE merges. …

What moved

mlx: match publisher tokenizer semantics Honor pretokenizer stage order, split behavior, Unicode boundaries, added-token normalization, and ranked BPE merges. Handle empty added tokens and empty Metaspace input consistently. …

Why it matters

Filed as a public note from Ollama, 2026-10-04. The desk tags this brief LLM / Hardware.

On the record

  • Filed from the Ollama official GitHub releases feed on 2026-10-04.
  • Primary source host: github.com.
  • mlx: match publisher tokenizer semantics Honor pretokenizer stage order, split behavior, Unicode boundaries, added-token normalization, and ranked BPE merges.
Primary source
github.com
Desk
Logged as brief 065
Cite this briefv0.40.0-rc1: mlx: match publisher tokenizer semantics (#18779) (Ollama, 2026-10-04) — https://labledgerdesk.pages.dev/b/2026/10/04/v0-40-0-rc1-mlx-match-publisher-tokenizer-semantics-18779/