Category: OpenClaw

I’ve spent a lot of time tuning local LLMs over the last few months, but every now and then I run into an optimization that forces me to stop tweaking settings and actually understand how something works.

Recently that optimization was MTP.

I was running Qwen3.6-35B-A3B on an RTX 3090 and started seeing references to Multi-Token Prediction (MTP) and speculative decoding in llama.cpp. The promise was enticing: generate multiple tokens ahead of time and verify them in batches, potentially producing a significant speedup during inference.

Like many people, I assumed the hard part would be finding the right command-line arguments.

I was wrong.

The Setup

My baseline configuration was:

• RTX 3090 (24GB VRAM)
• Intel i7-11700F
• unsloth/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf (MoE, quantized GGUF, non-MTP baseline)
• ~81k context window
• llama.cpp server (recent build)

The model already performed well, but long coding prompts exposed a familiar problem: latency under sustained generation.

Speculative decoding looked like the obvious next step.

The First Mistake: Treating Configuration as Evidence

My first experiments focused on enabling speculative decoding through llama.cpp’s runtime options. I added the appropriate flags, restarted the server, and started watching logs.

At first glance everything looked promising. The server started. Requests completed successfully. I even saw references to speculative decoding in various places throughout the codebase and documentation.

The problem was that I never actually confirmed whether MTP was running.

I was treating the configuration as evidence.

Over the next several hours I went through a familiar optimization cycle: change a setting, run a test, examine logs, repeat. Sometimes performance improved. Sometimes it didn’t. The results were noisy enough that I couldn’t confidently explain what was happening.

The breakthrough came when I stopped looking at throughput numbers and started reading startup logs line by line.

One log entry stood out:

context type MTP requested but model doesn't contain MTP layers
failed to create MTP context

That single message completely changed my understanding of the problem.

Up until that point I had been assuming that enabling:

--spec-type draft-mtp

somehow activated MTP behavior.

What the logs revealed was much simpler.

The flag does not create MTP capability. It merely tells llama.cpp to use MTP if the model already contains MTP layers.

My model didn’t.

I had been asking llama.cpp to use a feature that wasn’t present in the model itself.

The Fix: Switching Models

Once I understood that distinction, everything else fell into place. I located a version of the model that actually contained MTP layers and switched over to it.

The MTP variant comes from bartowski/Qwen_Qwen3.6-35B-A3B-GGUF — the same repo as the standard quantized models, but with a separate file that includes the MTP layers needed for speculative decoding.

The startup logs immediately looked different:

estimated memory usage of MTP context is 232 MiB
creating MTP draft context
speculative decoding context initialized

These were messages I had never seen before. For the first time, llama.cpp was actually creating an MTP context instead of silently falling back to ordinary decoding.

What I Initially Misunderstood About MTP

During early testing, I conflated several distinct parts of the system. Understanding these distinctions is what makes further tuning predictable rather than experimental.

1. draft-mtp is not a full system switch

I initially treated

--spec-type draft-mtp

as if it enabled a complete MTP mode on its own. In reality, it is only a speculative decoding backend selection — a flag that tells llama.cpp to use MTP if the model supports it. The model must already contain MTP layers for anything to happen.

2. The MTP model variant is separate from the algorithm

The model file from bartowski’s repo that includes MTP layers is a build that has been packaged with the additional MTP layers needed for speculative decoding. Using this model enables the algorithm to function, but the two concepts are distinct:

• The algorithm (draft-mtp) controls how draft tokens are generated and validated during inference
• The model provides the MTP layers that the algorithm needs to work

This is easy to confuse because llama.cpp exposes them as separate knobs.

3. KV cache behavior is more nuanced than it appears

I previously interpreted internal log messages suggesting

cache_k=f16
cache_v=f16

as meaning the entire system had switched KV cache formats. This was incorrect.

That configuration applies only to the internal speculative draft context, not the main model. In my setup, the main model explicitly uses q4_0 KV cache. The MTP draft context uses f16 for its own internal operations, which is a separate allocation.

What Actually Changed Performance

After enabling draft-mtp and switching to the MTP variant model, the system stabilized and became measurably faster.

Typical results looked like:

• ~67–70% draft token acceptance rate
• ~90–100 tokens/sec generation speed
• consistent performance even at ~30k+ prompt lengths

This confirmed that speculative decoding was actively contributing to throughput, not just enabled in name.

Key Insights from Tuning

Acceptance rate matters most

The most useful signal turned out to be the draft acceptance rate. In my setup, the optimal range was:

~65–70% acceptance

Lower values tend to waste compute on rejected drafts, while higher values often indicate insufficient speculative coverage.

Context size dominates memory behavior

At ~81k tokens, KV cache usage becomes a major factor in performance. This has more impact on stability than most speculative decoding parameters.

MoE models behave differently

Using a mixture-of-experts model changes speculative decoding dynamics:

• token prediction is less uniform than dense models
• acceptance rates are naturally lower
• performance gains are real, but more modest and sensitive to tuning

GPU offloading needs flexibility

An early configuration forcing full GPU offload caused memory pressure:

--n-gpu-layers 999

Switching to:

--n-gpu-layers -1

allowed llama.cpp to automatically balance layer placement, improving stability without reducing performance.

Final Configuration

• --spec-type draft-mtp
• MTP variant from bartowski/Qwen_Qwen3.6-35B-A3B-GGUF
• Baseline: unsloth/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
• --n-gpu-layers -1
• 81k context window
• KV cache in Q4_0
• flash attention enabled

No external draft model was used.

Conclusion

The most important lesson from this process wasn’t about squeezing out a few extra tokens per second — it was about understanding the structure of the system itself.

Modern llama.cpp speculative decoding is no longer a single, monolithic feature. It operates across multiple layers:

• a decoding algorithm (draft-mtp)
• a required MTP model variant
• and runtime optimizations that adapt dynamically based on workload

Once those layers are separated conceptually, the tuning process becomes much clearer — and far less confusing than the flags initially suggest.

What struck me most about this experience was how easy it is to confuse configuration with functionality. Modern inference servers expose hundreds of knobs and switches, and it’s tempting to assume that enabling a feature means that feature is active.

In reality, the logs were telling the truth the entire time. I just wasn’t asking the right question.

Instead of asking, “Did I set the flag correctly?” I should have been asking, “Can I prove the feature is actually running?”

Once I started looking for proof instead of configuration, the answer became obvious.

When I first became interested in running AI locally, I had a very simple goal: I wanted an assistant. Not a research project, not a homelab experiment, not a collection of interconnected services spread across multiple containers. I just wanted a capable AI system that ran on my own hardware, respected my privacy, and was available whenever I needed it. At the time, that seemed like a fairly modest ambition.

The basic recipe appeared straightforward enough. Download a model, install some software, connect a user interface, and start chatting. Every tutorial made the process look manageable. Every YouTube video seemed to end with someone happily talking to a model running on their desktop. I assumed I would eventually arrive at the same destination. In a sense, I did. I just took a much stranger route than I expected.

The first few days were exactly what I had imagined. I experimented with different models, compared responses, and learned the basics of running inference locally. There was something satisfying about watching my own hardware generate answers without relying on external services. Every improvement felt tangible—a better model produced better responses, a larger context window remembered more information. Progress was easy to recognize.

What I didn’t notice at the time was that every improvement was quietly increasing the complexity of the system underneath me. The first sign of this came when I started looking at memory. Like many people using AI regularly, I quickly discovered that forgetting is one of the biggest limitations of conversational systems. Having to repeat information in every session gets old surprisingly fast. Naturally, I started investigating ways to give the assistant long-term memory.

That seemed like a feature. What I didn’t realize was that it was actually an entirely new subsystem. Suddenly I was reading about embeddings, retrieval pipelines, indexing strategies, and vector databases. The assistant still looked the same from the outside, but behind the scenes a whole new collection of moving parts had appeared. Information now had to be processed, stored, searched, and maintained. The system wasn’t simply generating answers anymore—it was managing information. At some point I stopped being an AI user and became the person responsible for keeping the memory system alive. The transition was subtle enough that I barely noticed it happening.

The same thing occurred with model serving. Early on, I treated model backends as implementation details. Ollama, llama.cpp, and similar tools all appeared to solve the same basic problem: they loaded models and generated responses. From a user’s perspective, they were interchangeable. From an operator’s perspective, they were not.

The deeper I went, the more I found myself caring about things that had never crossed my mind before. Resource allocation. GPU utilization. Context management. Startup parameters. Container behavior. Suddenly I wasn’t evaluating software based solely on what it could do. I was evaluating it based on how predictable it was, how easy it was to troubleshoot, and how well it behaved under real-world conditions. Without realizing it, I had started thinking like an administrator.

One of the strangest moments came when I realized how much time I was spending reading logs. When I began this project, I viewed logs as something you consulted after a failure. If a service crashed, you checked the logs to determine why. That seemed reasonable enough. Months later, I found myself reading logs even when nothing was obviously wrong—looking at startup messages to understand context allocation, examining memory usage, investigating warnings that might or might not matter. I was trying to understand why a system behaved a certain way rather than simply whether it was functioning. That isn’t user behavior. That’s administrator behavior.

The same pattern appeared elsewhere. I started caring about backup strategies for configuration files. I became interested in container restart behavior. I found myself thinking about maintenance windows and scheduled jobs. When a service update became available, my first thought wasn’t whether it introduced new features. My first thought was whether it might break something that currently worked.

Nobody gets excited about AI because they want to spend their evenings thinking about maintenance windows. Yet there I was, doing exactly that.

Part of the reason this transformation happens, I think, is that local AI systems are deceptively simple from the outside. The interface presents a single assistant. You type a question. You receive an answer. The experience feels unified. Behind the scenes, however, there is often an entire ecosystem making that interaction possible—inference servers loading models into memory, databases storing information, embedding systems generating vectors, containers, APIs, schedulers, storage systems, and configuration layers all working together to produce what appears to be a single conversation.

As the system grows, someone has to understand how those pieces interact. If you’re running everything yourself, that someone is you.

What surprised me most wasn’t the complexity itself—technology is often complicated. What surprised me was how quickly my priorities changed. Early on, I was obsessed with capability. I wanted better models, larger contexts, improved reasoning, and more advanced features. Over time, I became increasingly interested in stability. A feature that works ninety-nine percent of the time sounds impressive until you’re the person responsible for the remaining one percent. Suddenly reliability starts looking a lot more attractive than cutting-edge functionality. Predictable behavior becomes more valuable than clever behavior. You begin appreciating software that quietly does its job and stays out of your way. Those are the kinds of opinions that systems administrators tend to develop. I never expected local AI to teach me that lesson.

Looking back, I think the most interesting part of this journey is that my original goal never actually changed. I still wanted an assistant. I still wanted a useful AI system running on my own hardware. The destination remained remarkably consistent. What changed was my understanding of what it takes to get there.

The assistant was never just a model. It was a model, a serving layer, a memory system, a collection of databases, a retrieval pipeline, a storage layer, a scheduling mechanism, and a growing assortment of supporting infrastructure. The more capable I wanted the assistant to become, the more infrastructure appeared behind it. Every new capability required someone to keep that infrastructure running. Gradually, almost accidentally, that became my job.

The funny thing is that I don’t regret it. The project has occasionally been frustrating—there have certainly been moments where I questioned why a seemingly simple goal required so much troubleshooting. Yet the process has also taught me far more about how modern software systems actually work than I expected when I started. I originally thought I was learning about AI. In reality, I was learning about systems. The AI just happened to be the reason I started paying attention.