v0.34.2-rc2: mlxrunner: Release freed KV buffers during speculative decode

What moved

The decode loop releases MLX's pool of freed buffers every 256 generated tokens, which is also how often the KV cache grows and drops its previous, smaller buffers. The check fires only when the token count lands exactly on a multiple of 256. Speculative decoding emits several tokens per round, so most rounds step over the boundary and the pool is never released. Each growth at a long context leaves several GB of buffers that no later allocation can reuse.

Why it matters

The decode loop releases MLX's pool of freed buffers every 256 generated tokens, which is also how often the KV cache grows and drops its previous, smaller buffers. That is a public launch file from Ollama, dated 2026-09-17. Tagged LLM / Hardware.

On the record

• Filed from the Ollama official RSS on 2026-09-17.
• Primary source host: github.com.
• The decode loop releases MLX's pool of freed buffers every 256 generated tokens, which is also how often the KV cache grows and drops its previous, smaller buffers.