Local model inference perceived as slow: KV/context aren't the cause; the ≥​64GB default is a 35B MoE + prompt-processing dominates

[Broccolito]

Describe the problem

Local ("Llama Server" / llama.cpp) model inference is reported as very slow. This issue captures an empirical investigation into why, because the obvious suspects (KV cache, context window, "model too big") turn out not to be the main story.

Empirical finding: raw llama.cpp inference is fast; the KV-cache/context-window config is NOT the bottleneck.

Benchmarked on an Apple M4 Max / 128 GiB using the app's own shipped binary (/Applications/BioRouter.app/Contents/Resources/bin/llamacpp/llama-server, build 9611) against the downloaded default-tier model Qwen3.5-4B-Q4_K_M (2.6 GB), a 1990-token prompt, n_predict=200:

Config	model load	prompt processing	generation
A — exact app defaults (--ctx-size 131072 --cache-type-k q8_0 --cache-type-v q8_0 --reasoning off)	1.1 s	1383 tok/s	91.5 tok/s
B — lean (--ctx-size 8192, f16 KV, --flash-attn on)	1.1 s	1396 tok/s	95.7 tok/s
C — app KV, small ctx (--ctx-size 8192 q8_0 K+V)	1.1 s	1382 tok/s	91.5 tok/s
D — app defaults + --flash-attn on	1.1 s	1382 tok/s	91.7 tok/s

Takeaways:

• 91–96 tok/s generation on a 4B Q4 model is healthy/fast for this hardware.
• Context window (128k vs 8k), KV quant (q8_0 vs f16), and flash-attn make a ~4% difference at most — so the user hypotheses "context window" and "KV cache" are ruled out as the cause of large slowness. (--flash-attn already defaults to auto in build 9611, so it's effectively on; the app doesn't need to pass it.) The build args live in crates/biorouter/src/providers/llamacpp_sidecar.rs::build_args (~L732) and the memory-tiered context default in default_context_size (~L620).

So where does perceived slowness actually come from? Two real structural factors:

1. On high-memory machines the default local model is a 35B MoE that must download ~20 GB.
   recommended_model_for_memory_gib (crates/biorouter/src/providers/llamacpp.rs ~L167) returns qwen3.6 for any machine with ≥64 GiB — i.e. unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M, a 35B-total / 3B-active MoE (~20 GB). That is (a) a very long first-run download (and see the companion "download progress" bug), and (b) inherently slower per token and much heavier to load than the small models (gemma4 E4B / the Qwen3.5-4B benchmarked above). A user on a big Mac who accepts the default is running the heaviest catalog model. Worth revisiting whether 35B-MoE is the right default (vs. offering it as an opt-in "large" tier) and surfacing the model's size + expected speed before download.

2. Per-turn latency in the agent is dominated by prompt processing of a large, growing context — not token generation. Biorouter's agent bootstrap (system prompt + all enabled extension tool definitions + skills instructions) can be ~20k tokens before the first user turn, and it grows with each tool result. At the measured ~1383 tok/s prompt-processing rate, a 20k-token context is ~15 seconds before the first generated token, every turn where the prefix isn't cache-hit — which feels like "the model is slow" even though generation is fast. This is amplified by how many extensions/skills are enabled (each adds tool defs + instructions to the prompt); it connects directly to the skills enable/disable request (making it easy to turn off skills you don't need shrinks the prompt). (This point is a hypothesis derived from the prompt-processing throughput above; a follow-up should measure the real agent turn end-to-end.)

Suggested actions

• Don't chase the KV-cache/context-window config — data above shows it isn't the bottleneck.
• Reconsider the ≥64 GiB default model: either keep a smaller/faster default and make qwen3.6 (35B MoE) an explicit "large model" choice, or clearly show download size + expected tokens/sec so users opt in knowingly.
• Investigate first-token latency in the agent path: confirm llama-server prompt caching (cache_prompt) is actually hit across turns, and minimize prompt churn so the ~20k-token prefix is reused rather than reprocessed. Trimming enabled extensions/skills should measurably help.
• Consider a lightweight in-app "why is local inference slow?" diagnostic that reports model size, the live /props context window, prompt vs generation tok/s, and whether the prompt cache is being reused.

Environment

• Apple M4 Max, 128 GiB unified memory, macOS.
• llama-server build 9611 (shipped with the app); models via HF -hf into ~/.local/share/biorouter/llamacpp/models/.
• Companion issue: local model download progress display stops when navigating away (the ~20 GB default download makes it prominent).

[Broccolito]

Adopted the report's recommendations: the ≥64 GiB tier no longer defaults to the 35B MoE — it now recommends the fast dense mid-size Gemma 4 12B (7.6 GB), with the 35B kept in the catalog as an explicit opt-in "large" choice — and every catalog entry now carries structured speed metadata (active_params_b + a human hint like "Fast — dense 12B, quick to load") surfaced next to the download size in onboarding and the settings inventory (OpenAPI + TS client regenerated). Verified live on a 128 GiB machine: onboarding preselects Gemma 4 12B with the size + speed line visible on every option. The KV/context findings were taken as-is (no changes chased there), and the landing-site copy that still claimed the 35B default was synced.

Commits: git log --oneline --grep "(#35)" (merged in d4b6cad).