Benchmarking MTP on an RTX 3090: What Multi-Token Prediction Actually Speeds Up

After finally getting Multi-Token Prediction working in llama.cpp, I had a simple question:

Was it actually faster?

It’s easy to convince yourself that a new optimization is helping. You make a change, run a few prompts, and the model feels faster. But perception is not measurement.

I wanted numbers.

I had just learned the hard way that MTP requires a model that actually contains MTP layers — my initial setup using unsloth/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf was silently falling back to ordinary decoding. Switching to the MTP variant from bartowski/Qwen_Qwen3.6-35B-A3B-GGUF made the system actually run speculative decoding. But I still didn’t know if it was helping.

So I built two nearly identical configurations:

• A baseline using the standard unsloth/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf model (no MTP layers, no speculative decoding)
• A test configuration using the MTP-enabled variant from bartowski/Qwen_Qwen3.6-35B-A3B-GGUF with --spec-type draft-mtp

Everything else stayed fixed:

• Same RTX 3090
• Same llama.cpp build
• Same context size
• Same prompts
• Same server configuration
• Same cache settings

From there, I built three benchmarks to isolate different parts of the inference pipeline.

---

Benchmark Design

The goal was to separate the system into three measurable phases:

• Decode Test → generation throughput
• Prefill Test → prompt ingestion speed
• Mixed Test → realistic workload behavior

This separation matters because speculative decoding only affects generation, not prompt processing.

One important detail: for the decode and mixed tests, I measure completion tokens per second (generation phase only), since that’s what MTP affects. The prefill test uses total tokens, since it’s measuring prompt processing speed.

---

Test Definitions

1. Decode Test (Generation Throughput)

• Prompt: ~50–60 tokens
• Completion: ~1500 tokens
• Metric: completion_tokens / elapsed (generation TPS)

Purpose: measure pure decoding speed with minimal overhead.

---

2. Prefill Test (Context Ingestion)

• Prompt: ~55,000 tokens
• Completion: ~10–20 tokens
• Metric: total_tokens / elapsed (prompt eval TPS)

Purpose: measure how fast the model processes long context windows.

---

3. Mixed Test (Realistic Workload)

• Prompt: ~13,000 tokens
• Completion: ~800 tokens
• Metric: completion_tokens / elapsed (generation TPS)

Purpose: approximate real-world coding/chat usage.

---

Initial Results

Decode Test

• Non-MTP: 132 tokens/sec
• MTP: 175 tokens/sec
• Improvement: +32%

Clear gain in pure generation workloads.

---

Mixed Test

• Non-MTP: 116 tokens/sec
• MTP: 146 tokens/sec
• Improvement: +26%

Realistic workloads still benefit significantly.

---

Prefill Test

• Non-MTP: ~39 tokens/sec
• MTP: ~39 tokens/sec
• Improvement: ~0%

No meaningful change.

This confirmed a key point early:

MTP accelerates generation, not prompt processing.

---

What This Actually Means

MTP is not a general inference speedup.

It is a generation-side optimization only.

So its impact depends entirely on workload shape:

• Chatty / long generation → large gains
• Prompt-heavy workloads → little to no benefit

For my setup (RTX 3090 + Qwen3.6-35B-A3B), the gain is substantial—but not universal.

---

Controlled Re-Run (Validation Pass)

After the initial results, I reran the benchmarks with stricter controls to remove ambiguity:

Standardized changes:

• Fixed context size to 65,536
• Removed multimodal projection from baseline
• Set spec-draft-n-max = 4 for MTP runs

The goal was not to change behavior, but to remove noise.

---

Decode Test

• Non-MTP: ~131–134 tokens/sec
• MTP: ~174–175 tokens/sec
• Improvement: +30–33%

---

Mixed Test

• Non-MTP: ~116–117 tokens/sec
• MTP: ~144–146 tokens/sec
• Improvement: +24–26%

---

Prefill Test

• Non-MTP: ~37–40 tokens/sec
• MTP: ~38–40 tokens/sec
• Improvement: ~0%

---

Draft Acceptance (MTP)

Observed across runs:

• ~82–88% draft token acceptance
• Stable speculative decoding behavior
• Consistent acceleration during generation

The higher acceptance rate compared to the initial 67–70% runs is likely due to the fixed context size (65,536) and spec-draft-n-max = 4 providing more consistent conditions for draft predictions.

---

Reproducible Benchmark Harness

To avoid subjective testing, I wrapped all benchmarks into a repeatable harness that isolates inference phases.

Bash Runner

#!/usr/bin/env bash

SERVER="http://localhost:8080/v1/completions"
OUTPUT_CSV="benchmark_results.csv"

if [ ! -f "$OUTPUT_CSV" ]; then
  echo "timestamp,test,elapsed_sec,prompt_tokens,completion_tokens,total_tokens,tps" > "$OUTPUT_CSV"
fi

run_test () {
  NAME=$1
  PROMPT_FILE=$2
  MAX_TOKENS=$3

  START=$(date +%s.%N)

  RESPONSE=$(curl -s $SERVER \
    -H "Content-Type: application/json" \
    -d <<EOF
{
  "model": "qwen",
  "prompt": "$(cat $PROMPT_FILE)",
  "max_tokens": $MAX_TOKENS,
  "temperature": 0.0
}
EOF
)

  END=$(date +%s.%N)

  ELAPSED=$(echo "$END - $START" | bc)

  PROMPT_TOKENS=$(echo "$RESPONSE" | jq '.usage.prompt_tokens')
  COMPLETION_TOKENS=$(echo "$RESPONSE" | jq '.usage.completion_tokens')
  TOTAL_TOKENS=$(echo "$PROMPT_TOKENS + $COMPLETION_TOKENS" | bc)

  # For decode/mixed tests: use completion_tokens (generation-only TPS)
  # For prefill tests: use total_tokens (prompt eval TPS)
  TPS=$(echo "$COMPLETION_TOKENS / $ELAPSED" | bc -l)

  TIMESTAMP=$(date -Iseconds)

  echo "$TIMESTAMP,$NAME,$ELAPSED,$PROMPT_TOKENS,$COMPLETION_TOKENS,$TOTAL_TOKENS,$TPS" >> "$OUTPUT_CSV"

  echo "Running: $NAME"
  echo "TPS (generation): $TPS"
  echo
}

run_test "Decode" decode_prompt.txt 1500
run_test "Prefill" prefill_prompt.txt 16
run_test "Mixed" mixed_prompt.txt 800

---

Aggregation Script

To compute averages across runs:

awk -F',' '
NR>1 {
  t[$2]++
  sum[$2]+=$7
}
END {
  for (i in t) {
    printf "%s → avg TPS: %.2f (%d runs)\n", i, sum[i]/t[i], t[i]
  }
}
' benchmark_results.csv

---

Why This Matters

Without structured measurement, it’s easy to misinterpret performance changes:

• cache state affects prefill timing
• single runs exaggerate decode gains
• mixed workloads vary wildly with prompt composition

This setup forces a controlled comparison:

same inputs → different decoding strategy → measurable delta

---

Conclusion

The takeaway is not just that MTP improves performance.

It’s where it improves performance.

• Decode: consistently ~30% faster
• Mixed workloads: ~25% faster
• Prefill: unchanged

That makes MTP a generation accelerator, not a universal speedup.

And that distinction is the most important result of the entire experiment.