Pushing Local Models With Focus And Polish

Armin Ronacher’s diagnosis is that what holds local models back is not the models but the fragmentation and missing polish around them. Running one locally means choosing an inference engine, a model, a quantisation, a template and a context size, then wiring up JSON config, with effort scattered across llama.cpp, Ollama, LM Studio, MLX, Transformers and vLLM. His section heading sums it up: Runnable is not finished.

His test case is coding agents, where the shortfalls are specific rather than vague. Tool parameter streaming, which most local setups do not support, matters as much as token streaming does elsewhere: seeing parameters arrive tells you what edits are happening, so you can interrupt instead of burning tokens on a wrong path. Slow local models also break assumptions elsewhere in the stack, such as dead-connection timeouts that misfire when five minutes can pass between tokens.

The prescription is to stop building generic frameworks and instead take one model-plus-hardware combination end to end until it feels finished. He points to Salvatore Sanfilippo’s ds4.c, a deliberately narrow inference engine for DeepSeek V4 Flash on Macs with 128GB or more of RAM, and has built pi-ds4 to embed it directly into Pi. DeepSeek V4 Flash is the model he likes here, large enough to feel different from smaller dense models but sparse enough that the active parameter count makes it plausible to run.

The idea worth keeping is the standard he sets: Pick a winner hard. If a tool call breaks, that is a product bug and then it’s fixed no matter where in the stack it failed. Treating the whole chain as one product, rather than a pile of components that each work in isolation, is what the local stack is missing.