I’ve spent a lot of time tuning local LLMs over the last few months, but every now and then I run into an optimization that forces me to stop tweaking settings and actually understand how something works.
Recently that optimization was MTP.
I was running Qwen3.6-35B-A3B on an RTX 3090 and started seeing references to Multi-Token Prediction (MTP) and speculative decoding in llama.cpp. The promise was enticing: generate multiple tokens ahead of time and verify them in batches, potentially producing a significant speedup during inference.
Like many people, I assumed the hard part would be finding the right command-line arguments.
I was wrong.
The Setup
My baseline configuration was:
- RTX 3090 (24GB VRAM)
- Intel i7-11700F
- unsloth/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf (MoE, quantized GGUF, non-MTP baseline)
- ~81k context window
- llama.cpp server (recent build)
The model already performed well, but long coding prompts exposed a familiar problem: latency under sustained generation.
Speculative decoding looked like the obvious next step.
The First Mistake: Treating Configuration as Evidence
My first experiments focused on enabling speculative decoding through llama.cpp’s runtime options. I added the appropriate flags, restarted the server, and started watching logs.
At first glance everything looked promising. The server started. Requests completed successfully. I even saw references to speculative decoding in various places throughout the codebase and documentation.
The problem was that I never actually confirmed whether MTP was running.
I was treating the configuration as evidence.
Over the next several hours I went through a familiar optimization cycle: change a setting, run a test, examine logs, repeat. Sometimes performance improved. Sometimes it didn’t. The results were noisy enough that I couldn’t confidently explain what was happening.
The breakthrough came when I stopped looking at throughput numbers and started reading startup logs line by line.
One log entry stood out:
context type MTP requested but model doesn't contain MTP layers
failed to create MTP contextThat single message completely changed my understanding of the problem.
Up until that point I had been assuming that enabling:
--spec-type draft-mtpsomehow activated MTP behavior.
What the logs revealed was much simpler.
The flag does not create MTP capability. It merely tells llama.cpp to use MTP if the model already contains MTP layers.
My model didn’t.
I had been asking llama.cpp to use a feature that wasn’t present in the model itself.
The Fix: Switching Models
Once I understood that distinction, everything else fell into place. I located a version of the model that actually contained MTP layers and switched over to it.
The MTP variant comes from bartowski/Qwen_Qwen3.6-35B-A3B-GGUF — the same repo as the standard quantized models, but with a separate file that includes the MTP layers needed for speculative decoding.
The startup logs immediately looked different:
estimated memory usage of MTP context is 232 MiB
creating MTP draft context
speculative decoding context initializedThese were messages I had never seen before. For the first time, llama.cpp was actually creating an MTP context instead of silently falling back to ordinary decoding.
What I Initially Misunderstood About MTP
During early testing, I conflated several distinct parts of the system. Understanding these distinctions is what makes further tuning predictable rather than experimental.
1. draft-mtp is not a full system switch
I initially treated
--spec-type draft-mtpas if it enabled a complete MTP mode on its own. In reality, it is only a speculative decoding backend selection — a flag that tells llama.cpp to use MTP if the model supports it. The model must already contain MTP layers for anything to happen.
2. The MTP model variant is separate from the algorithm
The model file from bartowski’s repo that includes MTP layers is a build that has been packaged with the additional MTP layers needed for speculative decoding. Using this model enables the algorithm to function, but the two concepts are distinct:
- The algorithm (
draft-mtp) controls how draft tokens are generated and validated during inference - The model provides the MTP layers that the algorithm needs to work
This is easy to confuse because llama.cpp exposes them as separate knobs.
3. KV cache behavior is more nuanced than it appears
I previously interpreted internal log messages suggesting
cache_k=f16
cache_v=f16as meaning the entire system had switched KV cache formats. This was incorrect.
That configuration applies only to the internal speculative draft context, not the main model. In my setup, the main model explicitly uses q4_0 KV cache. The MTP draft context uses f16 for its own internal operations, which is a separate allocation.
What Actually Changed Performance
After enabling draft-mtp and switching to the MTP variant model, the system stabilized and became measurably faster.
Typical results looked like:
- ~67–70% draft token acceptance rate
- ~90–100 tokens/sec generation speed
- consistent performance even at ~30k+ prompt lengths
This confirmed that speculative decoding was actively contributing to throughput, not just enabled in name.
Key Insights from Tuning
Acceptance rate matters most
The most useful signal turned out to be the draft acceptance rate. In my setup, the optimal range was:
~65–70% acceptance
Lower values tend to waste compute on rejected drafts, while higher values often indicate insufficient speculative coverage.
Context size dominates memory behavior
At ~81k tokens, KV cache usage becomes a major factor in performance. This has more impact on stability than most speculative decoding parameters.
MoE models behave differently
Using a mixture-of-experts model changes speculative decoding dynamics:
- token prediction is less uniform than dense models
- acceptance rates are naturally lower
- performance gains are real, but more modest and sensitive to tuning
GPU offloading needs flexibility
An early configuration forcing full GPU offload caused memory pressure:
--n-gpu-layers 999Switching to:
--n-gpu-layers -1allowed llama.cpp to automatically balance layer placement, improving stability without reducing performance.
Final Configuration
--spec-type draft-mtp- MTP variant from bartowski/Qwen_Qwen3.6-35B-A3B-GGUF
- Baseline: unsloth/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
--n-gpu-layers -1- 81k context window
- KV cache in Q4_0
- flash attention enabled
No external draft model was used.
Conclusion
The most important lesson from this process wasn’t about squeezing out a few extra tokens per second — it was about understanding the structure of the system itself.
Modern llama.cpp speculative decoding is no longer a single, monolithic feature. It operates across multiple layers:
- a decoding algorithm (
draft-mtp) - a required MTP model variant
- and runtime optimizations that adapt dynamically based on workload
Once those layers are separated conceptually, the tuning process becomes much clearer — and far less confusing than the flags initially suggest.
What struck me most about this experience was how easy it is to confuse configuration with functionality. Modern inference servers expose hundreds of knobs and switches, and it’s tempting to assume that enabling a feature means that feature is active.
In reality, the logs were telling the truth the entire time. I just wasn’t asking the right question.
Instead of asking, “Did I set the flag correctly?” I should have been asking, “Can I prove the feature is actually running?”
Once I started looking for proof instead of configuration, the answer became obvious.
Leave a Reply