RCG Solutions

Author: Ron Glover

  • Building a Vue 3 Chat UI with Streaming LLM Responses (OpenClaw + Node.js + MariaDB)

    The Chat UI Project Timeline (May–July 2026)

    What started as a quick experiment to talk to a local LLM through OpenClaw slowly turned into something much larger than expected — a full chat application with streaming responses, authentication, persistent sessions, and eventually a modular Vue 3 frontend backed by Node.js and MariaDB.

    None of this was planned in advance. It grew the way most working systems do: one useful feature at a time, until the structure had to catch up.

    When it was just a single file (May 17–30)

    The first version was almost embarrassingly simple.

    A single HTML file. No framework. No build system. Just direct calls to OpenClaw’s /v1/chat/completions endpoint and a bit of JavaScript to make it feel interactive.

    It had just enough to feel real:

    • dark/light mode toggle using localStorage
    • a basic settings panel (API URL, model name, gateway config)
    • markdown rendering for responses
    • syntax highlighting for code blocks
    • streaming output that felt surprisingly responsive

    It ran locally through:

    python -m http.server

    And for a moment, that was enough.

    The one obvious limitation was also the one you could ignore at first: nothing persisted. Refresh the page, everything was gone. But at that stage, persistence wasn’t the point — proving the streaming pipeline was.

    The moment it started growing out of control (June 1)

    The problem wasn’t any single feature — it was that everything kept feeling “just one more step away” from being useful.

    So features accumulated:

    • a “thinking…” indicator while waiting for first token
    • streaming cursor animation
    • copy buttons (both plain and formatted text)
    • export options (HTML / JSON / text)
    • inline chat renaming
    • light mode fixes caused by CSS variable timing issues
    • session persistence bugs from object rehydration mismatches

    At the same time, the backend quietly stopped being a toy.

    The original Python proxy started showing instability, so it was replaced with a Node.js server. That shift ended up being bigger than expected:

    • stable API routing
    • proper REST endpoints
    • static file serving
    • rate limiting for login attempts
    • MariaDB-backed session storage
    • bcrypt password hashing

    At that point, the project stopped feeling like a frontend experiment and started feeling like a real system.

    But the frontend was clearly reaching its limit. Everything lived together in the same place — streaming logic, UI state, DOM manipulation, and configuration handling all overlapping in ways that made every change slightly risky.

    Vue 3 changes the way everything feels (Mid-June)

    Moving to Vue wasn’t just a refactor — it changed the mental model of the entire app.

    Instead of one growing script, there were now components:

    • Login handled authentication cleanly
    • Chat became an orchestrator instead of a dumping ground
    • routing replaced hash-based navigation
    • reactive state replaced manual DOM updates

    Streaming responses were the first thing that immediately improved. Instead of manually pushing chunks into the DOM, Vue reacted to state changes naturally.

    It wasn’t perfect — there were still rough edges:

    • thinking animation behavior needed adjustment
    • partial stream chunks occasionally rendered awkwardly
    • old imperative patterns kept bleeding into components

    But the structure was finally strong enough that problems became fixable instead of structural.

    When it stopped being single-user (June 12–17)

    Authentication changed the shape of the project.

    What started as “just let me log in” turned into a full user system:

    • session cookies (HTTP-only)
    • bcrypt password hashing
    • rate limiting on login attempts
    • MariaDB-backed auth sessions
    • per-user configuration

    And with that, the UI started to shift subtly too.

    “Model selection” stopped feeling like the right term. It became “Agent selection” — a small naming change, but it made the interface feel more aligned with how it was actually being used.

    The UI also started consolidating. Export and print features that were previously scattered across the interface were pulled into a single clean dropdown in the header. Small change, but it reduced a lot of visual noise.

    The point where refactoring became unavoidable (July 6)

    By early July, Chat.vue had grown past 1600 lines.

    At that point it wasn’t a question of whether to refactor — it was just a question of how painful it would be.

    Everything lived in it:

    • message rendering
    • streaming logic
    • session management
    • input handling
    • header + sidebar coordination
    • UI state transitions

    So the breakdown started slowly.

    HeaderBar was extracted first — taking title editing, theme toggles, settings, logout, export, and user display with it. Then the input area. Then thinking indicator. Then sidebar components were split into smaller responsibilities.

    Once that happened, something interesting became obvious: the complexity didn’t disappear — it just stopped colliding with itself.

    Even utility logic followed the same pattern:

    • theme handling moved into theme.js
    • settings and persistence moved into settings.js

    Nothing revolutionary — just finally separating things that had no business living together.

    Small UX fixes that mattered more than expected (July 6 — current)

    Once the structure stabilized, the changes became smaller but more noticeable.

    Copy buttons inside messages were redesigned into subtle hover-only controls:

    • 📋 copy full message
    • 📝 copy formatted text
    • ✓ temporary confirmation feedback

    Instead of taking up space, they now only appear when needed. The conversation stays visually clean, and the actions stay available without being intrusive.

    The sidebar bug that revealed the real issue

    One bug stood out because it felt like it shouldn’t have been a bug.

    Renaming a chat from the header didn’t immediately update the sidebar.

    Nothing was wrong with the API. The backend was updating correctly. The issue was simpler: the sidebar and header were not actually looking at the same reactive source.

    Once the session object inside the shared array was mutated directly, Vue handled the rest automatically. No extra events, no forced refresh — just proper state flow.

    It was one of those fixes that feels almost obvious in hindsight.

    Where it ended up

    As of July 6, 2026, the system has become a full-stack chat application:

    Frontend (Vue 3)

    • Login system
    • Header controls
    • Chat orchestration layer
    • Markdown + streaming message renderer
    • Input system
    • Sidebar session manager

    Backend

    • Node.js proxy server (REST + static hosting)
    • MariaDB session storage (JSON chat history)
    • bcrypt authentication system
    • OpenClaw gateway integration
    • llama.cpp backend (Qwen3.6-35B-A3B model)

    Everything runs containerized with multiple services coordinated together.

    Closing reflection

    The project didn’t follow a planned architecture. It didn’t start with a design document or a long-term structure in mind.

    It evolved through pressure.

    Features got added because they were useful. Structure got introduced when the lack of it started to hurt. Refactoring happened when change became too expensive without it.

    And the most consistent lesson throughout wasn’t about Vue, or Node, or even system design.

    It was this:

    You don’t really design systems like this in advance.
    You discover their shape by pushing them until they break — and then reshaping them just enough so they stop breaking in the same way.

  • Benchmarking MTP on an RTX 3090: What Multi-Token Prediction Actually Speeds Up

    After finally getting Multi-Token Prediction working in llama.cpp, I had a simple question:

    Was it actually faster?

    It’s easy to convince yourself that a new optimization is helping. You make a change, run a few prompts, and the model feels faster. But perception is not measurement.

    I wanted numbers.

    I had just learned the hard way that MTP requires a model that actually contains MTP layers — my initial setup using unsloth/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf was silently falling back to ordinary decoding. Switching to the MTP variant from bartowski/Qwen_Qwen3.6-35B-A3B-GGUF made the system actually run speculative decoding. But I still didn’t know if it was helping.

    So I built two nearly identical configurations:

    Everything else stayed fixed:

    • Same RTX 3090
    • Same llama.cpp build
    • Same context size
    • Same prompts
    • Same server configuration
    • Same cache settings

    From there, I built three benchmarks to isolate different parts of the inference pipeline.

    Benchmark Design

    The goal was to separate the system into three measurable phases:

    • Decode Test → generation throughput
    • Prefill Test → prompt ingestion speed
    • Mixed Test → realistic workload behavior

    This separation matters because speculative decoding only affects generation, not prompt processing.

    One important detail: for the decode and mixed tests, I measure completion tokens per second (generation phase only), since that’s what MTP affects. The prefill test uses total tokens, since it’s measuring prompt processing speed.

    Test Definitions

    1. Decode Test (Generation Throughput)

    • Prompt: ~50–60 tokens
    • Completion: ~1500 tokens
    • Metric: completion_tokens / elapsed (generation TPS)

    Purpose: measure pure decoding speed with minimal overhead.

    2. Prefill Test (Context Ingestion)

    • Prompt: ~55,000 tokens
    • Completion: ~10–20 tokens
    • Metric: total_tokens / elapsed (prompt eval TPS)

    Purpose: measure how fast the model processes long context windows.

    3. Mixed Test (Realistic Workload)

    • Prompt: ~13,000 tokens
    • Completion: ~800 tokens
    • Metric: completion_tokens / elapsed (generation TPS)

    Purpose: approximate real-world coding/chat usage.

    Initial Results

    Decode Test

    • Non-MTP: 132 tokens/sec
    • MTP: 175 tokens/sec
    • Improvement: +32%

    Clear gain in pure generation workloads.

    Mixed Test

    • Non-MTP: 116 tokens/sec
    • MTP: 146 tokens/sec
    • Improvement: +26%

    Realistic workloads still benefit significantly.

    Prefill Test

    • Non-MTP: ~39 tokens/sec
    • MTP: ~39 tokens/sec
    • Improvement: ~0%

    No meaningful change.

    This confirmed a key point early:

    MTP accelerates generation, not prompt processing.

    What This Actually Means

    MTP is not a general inference speedup.

    It is a generation-side optimization only.

    So its impact depends entirely on workload shape:

    • Chatty / long generation → large gains
    • Prompt-heavy workloads → little to no benefit

    For my setup (RTX 3090 + Qwen3.6-35B-A3B), the gain is substantial—but not universal.

    Controlled Re-Run (Validation Pass)

    After the initial results, I reran the benchmarks with stricter controls to remove ambiguity:

    Standardized changes:

    • Fixed context size to 65,536
    • Removed multimodal projection from baseline
    • Set spec-draft-n-max = 4 for MTP runs

    The goal was not to change behavior, but to remove noise.

    Decode Test

    • Non-MTP: ~131–134 tokens/sec
    • MTP: ~174–175 tokens/sec
    • Improvement: +30–33%

    Mixed Test

    • Non-MTP: ~116–117 tokens/sec
    • MTP: ~144–146 tokens/sec
    • Improvement: +24–26%

    Prefill Test

    • Non-MTP: ~37–40 tokens/sec
    • MTP: ~38–40 tokens/sec
    • Improvement: ~0%

    Draft Acceptance (MTP)

    Observed across runs:

    • ~82–88% draft token acceptance
    • Stable speculative decoding behavior
    • Consistent acceleration during generation

    The higher acceptance rate compared to the initial 67–70% runs is likely due to the fixed context size (65,536) and spec-draft-n-max = 4 providing more consistent conditions for draft predictions.

    Reproducible Benchmark Harness

    To avoid subjective testing, I wrapped all benchmarks into a repeatable harness that isolates inference phases.

    Bash Runner

    #!/usr/bin/env bash

SERVER="http://localhost:8080/v1/completions"
OUTPUT_CSV="benchmark_results.csv"

if [ ! -f "$OUTPUT_CSV" ]; then
  echo "timestamp,test,elapsed_sec,prompt_tokens,completion_tokens,total_tokens,tps" > "$OUTPUT_CSV"
fi

run_test () {
  NAME=$1
  PROMPT_FILE=$2
  MAX_TOKENS=$3

  START=$(date +%s.%N)

  RESPONSE=$(curl -s $SERVER \
    -H "Content-Type: application/json" \
    -d <<EOF
{
  "model": "qwen",
  "prompt": "$(cat $PROMPT_FILE)",
  "max_tokens": $MAX_TOKENS,
  "temperature": 0.0
}
EOF
)

  END=$(date +%s.%N)

  ELAPSED=$(echo "$END - $START" | bc)

  PROMPT_TOKENS=$(echo "$RESPONSE" | jq '.usage.prompt_tokens')
  COMPLETION_TOKENS=$(echo "$RESPONSE" | jq '.usage.completion_tokens')
  TOTAL_TOKENS=$(echo "$PROMPT_TOKENS + $COMPLETION_TOKENS" | bc)

  # For decode/mixed tests: use completion_tokens (generation-only TPS)
  # For prefill tests: use total_tokens (prompt eval TPS)
  TPS=$(echo "$COMPLETION_TOKENS / $ELAPSED" | bc -l)

  TIMESTAMP=$(date -Iseconds)

  echo "$TIMESTAMP,$NAME,$ELAPSED,$PROMPT_TOKENS,$COMPLETION_TOKENS,$TOTAL_TOKENS,$TPS" >> "$OUTPUT_CSV"

  echo "Running: $NAME"
  echo "TPS (generation): $TPS"
  echo
}

run_test "Decode" decode_prompt.txt 1500
run_test "Prefill" prefill_prompt.txt 16
run_test "Mixed" mixed_prompt.txt 800

    Aggregation Script

    To compute averages across runs:

    awk -F',' '
NR>1 {
  t[$2]++
  sum[$2]+=$7
}
END {
  for (i in t) {
    printf "%s → avg TPS: %.2f (%d runs)\n", i, sum[i]/t[i], t[i]
  }
}
' benchmark_results.csv

    Why This Matters

    Without structured measurement, it’s easy to misinterpret performance changes:

    • cache state affects prefill timing
    • single runs exaggerate decode gains
    • mixed workloads vary wildly with prompt composition

    This setup forces a controlled comparison:

    same inputs → different decoding strategy → measurable delta

    Conclusion

    The takeaway is not just that MTP improves performance.

    It’s where it improves performance.

    • Decode: consistently ~30% faster
    • Mixed workloads: ~25% faster
    • Prefill: unchanged

    That makes MTP a generation accelerator, not a universal speedup.

    And that distinction is the most important result of the entire experiment.

  • Discovering MTP: How I Confused Configuration With Functionality

    I’ve spent a lot of time tuning local LLMs over the last few months, but every now and then I run into an optimization that forces me to stop tweaking settings and actually understand how something works.

    Recently that optimization was MTP.

    I was running Qwen3.6-35B-A3B on an RTX 3090 and started seeing references to Multi-Token Prediction (MTP) and speculative decoding in llama.cpp. The promise was enticing: generate multiple tokens ahead of time and verify them in batches, potentially producing a significant speedup during inference.

    Like many people, I assumed the hard part would be finding the right command-line arguments.

    I was wrong.

    The Setup

    My baseline configuration was:

    The model already performed well, but long coding prompts exposed a familiar problem: latency under sustained generation.

    Speculative decoding looked like the obvious next step.

    The First Mistake: Treating Configuration as Evidence

    My first experiments focused on enabling speculative decoding through llama.cpp’s runtime options. I added the appropriate flags, restarted the server, and started watching logs.

    At first glance everything looked promising. The server started. Requests completed successfully. I even saw references to speculative decoding in various places throughout the codebase and documentation.

    The problem was that I never actually confirmed whether MTP was running.

    I was treating the configuration as evidence.

    Over the next several hours I went through a familiar optimization cycle: change a setting, run a test, examine logs, repeat. Sometimes performance improved. Sometimes it didn’t. The results were noisy enough that I couldn’t confidently explain what was happening.

    The breakthrough came when I stopped looking at throughput numbers and started reading startup logs line by line.

    One log entry stood out:

    context type MTP requested but model doesn't contain MTP layers
failed to create MTP context

    That single message completely changed my understanding of the problem.

    Up until that point I had been assuming that enabling:

    --spec-type draft-mtp

    somehow activated MTP behavior.

    What the logs revealed was much simpler.

    The flag does not create MTP capability. It merely tells llama.cpp to use MTP if the model already contains MTP layers.

    My model didn’t.

    I had been asking llama.cpp to use a feature that wasn’t present in the model itself.

    The Fix: Switching Models

    Once I understood that distinction, everything else fell into place. I located a version of the model that actually contained MTP layers and switched over to it.

    The MTP variant comes from bartowski/Qwen_Qwen3.6-35B-A3B-GGUF — the same repo as the standard quantized models, but with a separate file that includes the MTP layers needed for speculative decoding.

    The startup logs immediately looked different:

    estimated memory usage of MTP context is 232 MiB
creating MTP draft context
speculative decoding context initialized

    These were messages I had never seen before. For the first time, llama.cpp was actually creating an MTP context instead of silently falling back to ordinary decoding.

    What I Initially Misunderstood About MTP

    During early testing, I conflated several distinct parts of the system. Understanding these distinctions is what makes further tuning predictable rather than experimental.

    1. draft-mtp is not a full system switch

    I initially treated

    --spec-type draft-mtp

    as if it enabled a complete MTP mode on its own. In reality, it is only a speculative decoding backend selection — a flag that tells llama.cpp to use MTP if the model supports it. The model must already contain MTP layers for anything to happen.

    2. The MTP model variant is separate from the algorithm

    The model file from bartowski’s repo that includes MTP layers is a build that has been packaged with the additional MTP layers needed for speculative decoding. Using this model enables the algorithm to function, but the two concepts are distinct:

    • The algorithm (draft-mtp) controls how draft tokens are generated and validated during inference
    • The model provides the MTP layers that the algorithm needs to work

    This is easy to confuse because llama.cpp exposes them as separate knobs.

    3. KV cache behavior is more nuanced than it appears

    I previously interpreted internal log messages suggesting

    cache_k=f16
cache_v=f16

    as meaning the entire system had switched KV cache formats. This was incorrect.

    That configuration applies only to the internal speculative draft context, not the main model. In my setup, the main model explicitly uses q4_0 KV cache. The MTP draft context uses f16 for its own internal operations, which is a separate allocation.

    What Actually Changed Performance

    After enabling draft-mtp and switching to the MTP variant model, the system stabilized and became measurably faster.

    Typical results looked like:

    • ~67–70% draft token acceptance rate
    • ~90–100 tokens/sec generation speed
    • consistent performance even at ~30k+ prompt lengths

    This confirmed that speculative decoding was actively contributing to throughput, not just enabled in name.

    Key Insights from Tuning

    Acceptance rate matters most

    The most useful signal turned out to be the draft acceptance rate. In my setup, the optimal range was:

    ~65–70% acceptance

    Lower values tend to waste compute on rejected drafts, while higher values often indicate insufficient speculative coverage.

    Context size dominates memory behavior

    At ~81k tokens, KV cache usage becomes a major factor in performance. This has more impact on stability than most speculative decoding parameters.

    MoE models behave differently

    Using a mixture-of-experts model changes speculative decoding dynamics:

    • token prediction is less uniform than dense models
    • acceptance rates are naturally lower
    • performance gains are real, but more modest and sensitive to tuning

    GPU offloading needs flexibility

    An early configuration forcing full GPU offload caused memory pressure:

    --n-gpu-layers 999

    Switching to:

    --n-gpu-layers -1

    allowed llama.cpp to automatically balance layer placement, improving stability without reducing performance.

    Final Configuration

    No external draft model was used.

    Conclusion

    The most important lesson from this process wasn’t about squeezing out a few extra tokens per second — it was about understanding the structure of the system itself.

    Modern llama.cpp speculative decoding is no longer a single, monolithic feature. It operates across multiple layers:

    • a decoding algorithm (draft-mtp)
    • a required MTP model variant
    • and runtime optimizations that adapt dynamically based on workload

    Once those layers are separated conceptually, the tuning process becomes much clearer — and far less confusing than the flags initially suggest.

    What struck me most about this experience was how easy it is to confuse configuration with functionality. Modern inference servers expose hundreds of knobs and switches, and it’s tempting to assume that enabling a feature means that feature is active.

    In reality, the logs were telling the truth the entire time. I just wasn’t asking the right question.

    Instead of asking, “Did I set the flag correctly?” I should have been asking, “Can I prove the feature is actually running?”

    Once I started looking for proof instead of configuration, the answer became obvious.

  • My Unexpected Journey From AI User to Systems Administrator

    When I first became interested in running AI locally, I had a very simple goal: I wanted an assistant. Not a research project, not a homelab experiment, not a collection of interconnected services spread across multiple containers. I just wanted a capable AI system that ran on my own hardware, respected my privacy, and was available whenever I needed it. At the time, that seemed like a fairly modest ambition.

    The basic recipe appeared straightforward enough. Download a model, install some software, connect a user interface, and start chatting. Every tutorial made the process look manageable. Every YouTube video seemed to end with someone happily talking to a model running on their desktop. I assumed I would eventually arrive at the same destination. In a sense, I did. I just took a much stranger route than I expected.

    The first few days were exactly what I had imagined. I experimented with different models, compared responses, and learned the basics of running inference locally. There was something satisfying about watching my own hardware generate answers without relying on external services. Every improvement felt tangible—a better model produced better responses, a larger context window remembered more information. Progress was easy to recognize.

    What I didn’t notice at the time was that every improvement was quietly increasing the complexity of the system underneath me. The first sign of this came when I started looking at memory. Like many people using AI regularly, I quickly discovered that forgetting is one of the biggest limitations of conversational systems. Having to repeat information in every session gets old surprisingly fast. Naturally, I started investigating ways to give the assistant long-term memory.

    That seemed like a feature. What I didn’t realize was that it was actually an entirely new subsystem. Suddenly I was reading about embeddings, retrieval pipelines, indexing strategies, and vector databases. The assistant still looked the same from the outside, but behind the scenes a whole new collection of moving parts had appeared. Information now had to be processed, stored, searched, and maintained. The system wasn’t simply generating answers anymore—it was managing information. At some point I stopped being an AI user and became the person responsible for keeping the memory system alive. The transition was subtle enough that I barely noticed it happening.

    The same thing occurred with model serving. Early on, I treated model backends as implementation details. Ollama, llama.cpp, and similar tools all appeared to solve the same basic problem: they loaded models and generated responses. From a user’s perspective, they were interchangeable. From an operator’s perspective, they were not.

    The deeper I went, the more I found myself caring about things that had never crossed my mind before. Resource allocation. GPU utilization. Context management. Startup parameters. Container behavior. Suddenly I wasn’t evaluating software based solely on what it could do. I was evaluating it based on how predictable it was, how easy it was to troubleshoot, and how well it behaved under real-world conditions. Without realizing it, I had started thinking like an administrator.

    One of the strangest moments came when I realized how much time I was spending reading logs. When I began this project, I viewed logs as something you consulted after a failure. If a service crashed, you checked the logs to determine why. That seemed reasonable enough. Months later, I found myself reading logs even when nothing was obviously wrong—looking at startup messages to understand context allocation, examining memory usage, investigating warnings that might or might not matter. I was trying to understand why a system behaved a certain way rather than simply whether it was functioning. That isn’t user behavior. That’s administrator behavior.

    The same pattern appeared elsewhere. I started caring about backup strategies for configuration files. I became interested in container restart behavior. I found myself thinking about maintenance windows and scheduled jobs. When a service update became available, my first thought wasn’t whether it introduced new features. My first thought was whether it might break something that currently worked.

    Nobody gets excited about AI because they want to spend their evenings thinking about maintenance windows. Yet there I was, doing exactly that.

    Part of the reason this transformation happens, I think, is that local AI systems are deceptively simple from the outside. The interface presents a single assistant. You type a question. You receive an answer. The experience feels unified. Behind the scenes, however, there is often an entire ecosystem making that interaction possible—inference servers loading models into memory, databases storing information, embedding systems generating vectors, containers, APIs, schedulers, storage systems, and configuration layers all working together to produce what appears to be a single conversation.

    As the system grows, someone has to understand how those pieces interact. If you’re running everything yourself, that someone is you.

    What surprised me most wasn’t the complexity itself—technology is often complicated. What surprised me was how quickly my priorities changed. Early on, I was obsessed with capability. I wanted better models, larger contexts, improved reasoning, and more advanced features. Over time, I became increasingly interested in stability. A feature that works ninety-nine percent of the time sounds impressive until you’re the person responsible for the remaining one percent. Suddenly reliability starts looking a lot more attractive than cutting-edge functionality. Predictable behavior becomes more valuable than clever behavior. You begin appreciating software that quietly does its job and stays out of your way. Those are the kinds of opinions that systems administrators tend to develop. I never expected local AI to teach me that lesson.

    Looking back, I think the most interesting part of this journey is that my original goal never actually changed. I still wanted an assistant. I still wanted a useful AI system running on my own hardware. The destination remained remarkably consistent. What changed was my understanding of what it takes to get there.

    The assistant was never just a model. It was a model, a serving layer, a memory system, a collection of databases, a retrieval pipeline, a storage layer, a scheduling mechanism, and a growing assortment of supporting infrastructure. The more capable I wanted the assistant to become, the more infrastructure appeared behind it. Every new capability required someone to keep that infrastructure running. Gradually, almost accidentally, that became my job.

    The funny thing is that I don’t regret it. The project has occasionally been frustrating—there have certainly been moments where I questioned why a seemingly simple goal required so much troubleshooting. Yet the process has also taught me far more about how modern software systems actually work than I expected when I started. I originally thought I was learning about AI. In reality, I was learning about systems. The AI just happened to be the reason I started paying attention.

  • The Infrastructure Tax of Local AI

    One of the things I find amusing about local AI discussions is how often they focus on the fun part. People love talking about models. We compare benchmark scores. We debate reasoning ability. We argue about context windows, quantization formats, and whether the latest release is finally the model that changes everything. If you’ve spent any time in AI communities, you’ve probably seen the pattern: a new model appears, everyone rushes to test it, screenshots start circulating, and for a few days it feels like the entire field has been reinvented. I enjoy those conversations too, but the problem is that models are only part of the story.

    What I’ve slowly learned while building a local AI stack is that every capability comes with a cost. Not necessarily a financial cost, although there can be some of that as well. The cost I’m talking about is operational. Every feature you add increases the amount of infrastructure you have to maintain. Every improvement creates new dependencies. Every dependency introduces new opportunities for things to behave in ways you weren’t expecting. I think of this as the infrastructure tax of local AI. Like most taxes, you don’t notice it immediately.

    When I started, the system was simple. There was a model and an interface, essentially. I could load a model, ask a question, receive an answer, and feel reasonably satisfied that I had accomplished what I set out to do. Then I started wanting more. That sentence is responsible for an alarming percentage of my technical projects. The first request is always innocent: maybe I wanted a larger context window, maybe I wanted better memory, maybe I wanted the system to remember information across conversations. None of these sounded unreasonable. In fact, they sounded like features that any modern AI assistant should probably have. The problem is that features rarely arrive alone.

    Take memory as an example. Memory sounds straightforward when people describe it in articles or YouTube videos—store information, retrieve information, problem solved. Except now you need embeddings. Embeddings need models. Those models need compute resources. The stored data needs a database. The database needs maintenance. The index occasionally needs rebuilding. Documents need processing. Retrieval needs tuning. Suddenly what looked like a feature has become an entire subsystem.

    The same thing happened with context windows. At first, larger context seemed like a simple upgrade. More context means more information available to the model, and that’s generally true. What isn’t immediately obvious is that larger contexts consume resources. They affect memory usage. They influence performance. They change the kinds of models you can comfortably run. A feature that looked like a single configuration setting turned out to have consequences throughout the rest of the stack.

    This pattern repeated itself so many times that I eventually stopped being surprised. What made it particularly interesting was that the bills weren’t always due immediately. Sometimes a new feature would work perfectly for weeks before revealing its true cost. A model would load successfully and I’d assume the problem was solved, only to discover later that there wasn’t enough remaining GPU memory for another workload I wanted to run. A memory system would function beautifully until maintenance tasks entered the picture. A configuration would appear stable until an edge case exposed assumptions I didn’t realize I had made. The infrastructure tax has a habit of arriving after you’ve already committed to the purchase.

    One of the clearest examples from my own experience involved embeddings and memory maintenance. On paper, this was exactly the sort of capability I wanted. A local AI assistant that could retain information over time feels substantially more useful than one that forgets everything between sessions. What I hadn’t fully considered was that memory systems aren’t passive. They need work—information has to be processed, embeddings have to be generated, databases have to remain healthy. Maintenance jobs eventually need to run, and those jobs compete for resources with the very inference server that makes the assistant possible in the first place. At that point, the question stops being whether memory is useful and starts becoming whether the surrounding infrastructure can support it.

    That’s a different conversation, and a much less glamorous one. Nobody gets excited about maintenance windows. Nobody rushes to social media to celebrate a successful database optimization. Nobody posts benchmark charts comparing index rebuild strategies. Yet these are often the things that determine whether a system remains usable over the long term.

    As my setup became more sophisticated, I noticed another change in the way I evaluated technology. Early on, I cared mostly about capability. I wanted the smartest model, the largest context window, the best memory system, and the most advanced features I could reasonably run. Over time, I became increasingly interested in operational cost—not dollars, but complexity. If a new feature required additional services, how difficult would those services be to maintain? If a new model consumed more resources, what would it prevent me from doing elsewhere? If a new capability introduced another database, another container, or another background process, what would happen six months later when I was trying to remember how all the pieces fit together? These questions sound less exciting than benchmark comparisons, but they’re the ones that determine whether a project remains enjoyable.

    I think this is something many local AI enthusiasts eventually discover. At some point, the challenge stops being about making the model smarter and starts being about keeping the ecosystem healthy. The model is only one participant in a much larger system. There are inference servers, databases, memory pipelines, schedulers, APIs, containers, monitoring tools, and storage systems all interacting with one another. Every one of them has requirements. Every one of them occasionally needs attention. And every one of them contributes to the infrastructure tax.

    None of this is meant as a criticism of local AI. If anything, it’s one of the reasons I find the space so fascinating. Building these systems has taught me far more about infrastructure, resource management, and software architecture than I expected when I started. The process has been frustrating at times, but it has also been remarkably educational.

    Still, if I could go back and give myself one piece of advice at the beginning, it would probably be this: whenever you add a feature, don’t just ask what it gives you. Ask what it costs. Not the purchase price, and not the hardware requirements listed in a README. Ask what new systems it introduces. Ask what new dependencies it creates. Ask what maintenance it will require six months from now when the excitement has worn off and you’re simply trying to keep everything running. Because in local AI, every feature comes with a receipt. The clever part is remembering that you’ll eventually have to pay it.

  • The Day I Learned That “Running” and “Working” Are Different Things

    Pretty early in the process of building my local AI stack, I learned one of the most important lessons (ironically it was also one of the simplest): running and working are not the same thing.

    That probably sounds obvious. Every experienced computer user has encountered software that technically runs while simultaneously failing to accomplish anything useful. Yet I still fell into the trap — more than once.

    When I first started experimenting with OpenClaw, I measured progress in a straightforward way. If a service started successfully, that was a win. If a model loaded, that was a win. If I could send a prompt and receive a response, that was definitely a win. And to be fair, those things are wins — there’s no point worrying about advanced functionality if the basic system won’t even start. The problem is that getting something to run is often only the first ten percent of the journey. The remaining ninety percent is discovering whether it actually behaves the way you need it to.

    I think the first time this really became obvious was when I started experimenting with context windows. Like many people getting into local AI, I initially viewed context as a fairly straightforward resource. Bigger seemed better — if a model could remember more of the conversation, the overall experience should improve. So I started increasing context sizes. The settings looked correct. The logs looked correct. The model started successfully. Everything appeared to be working. Except I wasn’t entirely convinced that it was.

    There’s a peculiar frustration that comes from a system that appears healthy while producing results that don’t quite match your expectations. Nothing is obviously broken. There are no error messages, no crashes. There is simply a growing suspicion that reality and configuration are no longer aligned. I would eventually encounter that feeling many times throughout this project — sometimes it involved context windows, sometimes memory systems, sometimes OpenClaw itself. Occasionally it involved features that appeared to be functioning correctly until I spent enough time using them to discover subtle limitations that weren’t immediately obvious. The pattern was always the same: everything looked fine until I looked closer.

    One example that stands out involved memory. At a high level, memory systems sound wonderfully simple. Store information, generate embeddings, retrieve relevant information later. When people describe the process in articles or videos, it’s often presented as a natural extension of a language model — just add memory and the assistant becomes dramatically more useful. Technically, that’s true. Operationally, it’s a little more complicated.

    I eventually found myself spending time on things I never expected to care about: embedding generation, index maintenance, memory databases, background processing. Suddenly I wasn’t evaluating whether memory existed. I was evaluating whether memory remained useful over time, whether retrieval quality was good enough, and whether maintaining the system introduced new problems elsewhere. The memory system was running. That didn’t automatically mean it was working. The distinction mattered.

    The same thing happened when I started comparing inference backends. At various points I experimented with both Ollama and llama.cpp. From a distance, they seemed to solve the same problem. Both could load models. Both could generate responses. Both integrated with the rest of the stack. In a purely technical sense, both were working. Yet the deeper I went, the more differences I discovered. Resource management behaved differently. Configuration behaved differently. Visibility into system behavior differed. Features appeared at different times. What initially looked like two interchangeable solutions gradually revealed itself to be a collection of tradeoffs. The software was running. The question became whether it was working for my specific goals. That turned out to be a much harder question to answer.

    One of the unexpected consequences of this project was how much time I spent reading logs. When I started, I thought logs existed primarily to explain failures. If something crashed, the logs would tell me why. That’s still true. What I didn’t anticipate was how often I would use logs to understand systems that weren’t failing. A service would start successfully, but I’d still find myself reading startup messages to understand context allocation, or GPU utilization, or memory management, or tensor placement. Not because anything was broken, but because I was trying to verify that the system was behaving the way I thought it was.

    That became a recurring theme: the absence of errors is not proof that everything is functioning as intended. It’s merely proof that nothing has failed loudly enough to complain. Some of the most difficult problems I encountered never produced a clear error message. Instead, they showed up as inconsistencies — a context window that didn’t seem to behave quite right, a memory system that felt less effective than expected, a configuration that appeared correct while producing surprising results. Those situations were far more challenging than outright failures. At least when something crashes, you know where to start looking. A system that operates successfully while doing the wrong thing can keep you occupied for days.

    Perhaps the clearest example involved stability itself. There were times when individual components appeared healthy — OpenClaw was running, the inference server was running, the memory systems were running, containers were healthy, status indicators were green. Yet the overall experience still wasn’t where I wanted it to be. The issue wasn’t any single component. The issue was how those components interacted.

    That’s when I started realizing that building a useful AI system is fundamentally different from building a functional one. A functional system can answer questions. A useful system can do so reliably, predictably, and consistently enough that you stop thinking about the machinery behind it. That distinction took me much longer to appreciate than I would like to admit.

    Looking back, I think I spent the first phase of this project chasing green status indicators. I wanted containers running, services connected, models loaded, and dashboards reporting success. Those things matter. But eventually I discovered that a green status indicator is often the beginning of the investigation rather than the end. Once everything is running, you finally get to discover whether it’s actually working. And strangely enough, that’s where the interesting problems begin.

    Today, when I bring a new service online, I still enjoy seeing it start successfully. I still appreciate a clean startup log and a healthy status dashboard. I’ve just learned not to confuse those things with success. Running is easy to measure. Working takes longer to prove.

  • Building a Custom Chat UI for OpenClaw

    When I first started experimenting with OpenClaw, I wanted a simple, fast way to talk to my assistant — no mobile app, no web framework, no dependencies. Just a chat interface.

    The OpenClaw gateway exposes a REST API on 127.0.0.1:18789 with streaming responses. That’s enough to build a full chat interface with. So that’s what I did.

    The whole thing is four files: 68 lines of HTML, 651 lines of CSS, 824 lines of JavaScript, and a 173-line Node.js proxy server. 1,736 lines total. No build step. No npm install. No frameworks. Just static files served by a simple Node.js proxy on port 8888.

    The proxy does one thing: it forwards API requests to the gateway, handles authentication transparently, and serves static files from disk so edits take effect instantly without any restart. That last part turned out to be the most underrated feature — I could tweak CSS, fix a JavaScript bug, and see the result in the browser immediately. No reload. No hot-reload config. Just edit and refresh.

    The chat UI itself uses localStorage to persist sessions. A sidebar shows your chat history, a main area renders messages as Markdown with syntax highlighting (via highlight.js), and the input bar handles streaming responses with a thinking animation. Sessions are saved to localStorage as JSON, keyed by session ID.

    One of the more interesting challenges was the model selection. OpenClaw has multiple models available, and you can set a model per session. The UI includes a dropdown to pick a model, with a lock icon to indicate when a session’s model has been overridden from the default. That required fetching the model list from the gateway on load, storing the current model per session, and syncing with the gateway API.

    The styling was another interesting problem. I wanted a clean, minimal design that worked well in both light and dark modes. The CSS uses custom properties for theming (--bg-*, --text-*, --border-*), and data-theme is set on the <html> element. The sidebar collapses on mobile, messages use a chat-bubble layout, and code blocks get syntax highlighting with copy buttons.

    The proxy server itself is simple: it serves static files directly from disk (so edits are live), proxies API requests to the gateway with proper headers, and handles CORS. No session management, no authentication UI — just forwarding requests with the gateway token from the configuration.

    One thing I found interesting about the architecture is the separation of concerns. The HTML is structure, the CSS is presentation, the JS is behavior, and the proxy is infrastructure. Each layer is thin and does one thing. That makes it easy to debug and modify.

    The biggest bug I ran into was with the session list. When a chat was deleted, the UI removed it from localStorage but the sidebar still showed the old list because the in-memory cache wasn’t updated. The sidebar rendered from the stale cache instead of localStorage. The fix was straightforward — update the cache after filtering. But it highlighted a common pattern in small apps: state lives in multiple places, and keeping them in sync is the actual work.

    The print functionality turned out to be one of the more surprising features. The original “Save as HTML” was replaced with a print button that opens a print-friendly window with all CSS inlined, auto-triggers the print dialog, and handles page breaks for code blocks while allowing chat bubbles to flow across pages. It was built with a single window.open() call and document.write() of an inlined document.

    The project started on June 1 and is now on commit d3966e4 with 24 commits. It’s a small project, but it’s exactly what I needed — a fast, local chat interface with no dependencies, no cloud services, and zero setup beyond pointing a browser at localhost:8888.

    I keep thinking about the separation between what people think building software involves and what it actually involves. On the surface, it’s just writing HTML and CSS. But the real work is the state management, the edge cases, the bugs that reveal themselves only after you’ve been using it for a while. The chat UI looks simple. It took more iterations than I expected to get it right.

    The most valuable thing about it isn’t the features — it’s the feedback loop. Edit, refresh, done. That kind of speed changes how you build things. You stop being afraid to make changes. You start iterating instead of planning. You end up with something that works better than the plan would have suggested, simply because you had room to discover what worked as you went.

  • The Biggest Problem With My Local AI Stack Wasn’t the Model. It Was VRAM.

    When I started building a local AI stack, I thought I was signing up for a model-selection problem. Like most people getting into local AI, I spent a ridiculous amount of time comparing benchmarks, reading Reddit threads, watching YouTube reviews, and trying to decipher whether one quantization format was secretly better than another. Every few days a new model would appear that was supposedly faster, smarter, or somehow both. The assumption underneath all of that research was simple: if I could just find the right model, the rest of the system would mostly take care of itself.

    That assumption survived for about a week.

    The first version of the system was actually pretty straightforward. I got OpenClaw running, connected it to a model server, and started experimenting. The responses looked good. The hardware seemed capable. I was running everything on a 3090, which felt like an absurd amount of GPU memory compared to what I had been working with before. Twenty-four gigabytes sounded enormous. More importantly, it felt sufficient. I wasn’t trying to train a foundation model. I just wanted a capable local assistant with memory and some room to grow.

    The phrase “room to grow” ended up doing a lot of damage.

    One of the things that happens when you start building local AI systems is that every successful step creates three new ideas. Once the model is working, you start thinking about memory. Once memory is working, you start thinking about larger context windows. Then you start reading about agents, retrieval systems, vector databases, and document ingestion pipelines. Before long, you’re no longer running a model. You’re assembling an ecosystem, and every new component arrives with its own expectations about memory, storage, compute resources, and maintenance.

    I don’t think I appreciated that at first because the model was always the center of the conversation. Every discussion online seemed to revolve around model quality. People compared reasoning ability, coding performance, benchmark scores, context lengths, and tokens per second. Those things matter, of course, but what I slowly discovered is that they matter a lot less once you start combining multiple AI workloads on the same machine. At that point the interesting problems stop being about intelligence and start being about resource management.

    The most dramatic example of this was VRAM contention. My inference server already occupied most of the GPU — that was expected. Modern models are expensive. What I hadn’t fully appreciated was what would happen when embedding maintenance wanted access to the same hardware.

    Initially I expected slower performance. What I got was something more serious. Embedding operations would run into problems. The system could become unstable. In some cases the machine itself appeared to hang. Those moments changed the way I thought about the entire stack.

    Suddenly benchmark scores didn’t seem nearly as important. It turns out that when you’re staring at a frozen server, you stop caring whether one model is three percent faster than another. The deeper I investigated, the more I realized that resource contention is one of the least discussed aspects of local AI. Most model reviews evaluate a single workload running under ideal conditions. Real systems rarely operate that way.

    Real systems have memory databases. Real systems have maintenance jobs. Real systems have background tasks. Real systems have users who inevitably decide to run something new at exactly the wrong moment. The challenge isn’t simply getting individual components to work. The challenge is making them coexist.

    That realization led me down a number of interesting rabbit holes. I investigated memory mapping. I looked at tensor placement. I examined offloading behavior. I explored idle unloading options in llama.cpp. Some of those investigations were productive. Others turned out to be distractions. One thing I learned is that warnings are often easier to find than actual problems — a startup log might contain a message that appears critically important while the real issue is occurring somewhere else entirely. It’s tempting to chase every warning because warnings feel actionable, but the difficult part is determining which ones actually matter.

    Eventually I found myself focusing less on optimization and more on reliability. This represented a significant change in mindset. Early in the project, I wanted elegant solutions — automatic unloading, dynamic resource sharing, sophisticated scheduling, ideally everything would manage itself. Over time I became more pragmatic. If stopping a service, running a maintenance job, and restarting the service afterward produces predictable results, that’s a perfectly respectable solution. It may not be elegant, but elegance is overrated when compared to stability.

    The more I worked with local AI systems, the more I came to appreciate that they are really collections of interacting services rather than single applications. The model is only one component. There are databases, embedding systems, schedulers, containers, APIs, and supporting infrastructure all competing for finite resources. VRAM just happens to be the place where those competing demands become visible first.

    In hindsight, the biggest lesson wasn’t about GPU memory at all. It was about systems thinking. The question I ask myself now when evaluating a model is no longer simply “How good is it?” Instead, I find myself asking a different set of questions: how much VRAM does it require? What does it do to the rest of the stack? What tradeoffs does it introduce? What happens when maintenance jobs need to run? How much complexity does it add?

    Those questions are less exciting than benchmark charts. They’re also much better predictors of how the system will behave in the real world. The model may be the star of the show, but the resources determine whether the show can go on.

  • I Thought I Was Installing an AI Assistant. I Ended Up Running an AI Platform.

    When I first started experimenting with OpenClaw, I thought I knew what the project would involve. I would pick a model, connect it to OpenClaw, give it some memory, and end up with a useful local AI assistant running on my own hardware. There would probably be a few configuration headaches along the way, but nothing particularly unusual.

    That seemed like a reasonable assumption at the time.

    The funny thing is that none of those individual tasks turned out to be especially difficult. I got models running. I connected OpenClaw. I experimented with different backends. Responses came back. Everything worked, at least in the narrow sense that software was running and producing output. What I didn’t understand yet was the difference between getting something running and getting something usable. That’s a lesson that kept repeating itself throughout the project.

    Early on I found myself spending a lot of time comparing models. That’s where most people start. Every few days a new release would appear that promised better reasoning, better coding ability, larger context windows, or some new capability that was supposed to change everything. I tried Ollama. I experimented with llama.cpp. I compared quantizations. I watched benchmark videos. I read forum discussions from people who seemed absolutely certain they had discovered the best possible model.

    The more I experimented, the more I realized that benchmarks were only telling a small part of the story. A model doesn’t exist in isolation. It lives inside a larger system. Context size matters. Memory usage matters. Tool support matters. Reliability matters. The smartest model in the world isn’t very useful if the rest of the stack struggles to support it. That realization arrived slowly rather than all at once.

    At first I was focused on inference. Then I became interested in context windows. Then I started looking at memory systems because the idea of an assistant that could actually remember things over time was hard to ignore. Once I started exploring memory, I discovered embeddings. Embeddings led to vector databases. Vector databases led to maintenance jobs, indexing, retrieval pipelines, and a growing collection of services that all needed to cooperate. The project was expanding in ways I hadn’t anticipated, and every successful step seemed to reveal another layer underneath it.

    One week I was investigating OpenClaw configuration options. The next week I was trying to understand why a context window wasn’t behaving the way I expected. Then I was troubleshooting memory flush behavior. Then I was comparing Ollama and llama.cpp to understand which one better fit my goals. The more I learned, the more I realized how much was happening beneath the surface. One thing that surprised me was how often I found myself troubleshooting interactions between systems rather than individual systems themselves. OpenClaw might be working correctly. The inference server might be working correctly. The memory backend might be working correctly. Yet the overall experience could still be problematic because of the way those pieces interacted.

    Those were often the hardest problems to diagnose. There is a certain comfort in a bug that has a clear cause — a configuration error can be fixed, a missing dependency can be installed, a broken setting can be corrected. It’s much more difficult when everything appears to be functioning independently but the combination behaves unpredictably. I spent a surprising amount of time reading logs, and more than once I found myself staring at startup messages trying to determine whether a warning actually mattered. Sometimes it did. Sometimes it didn’t. One evening I might be investigating memory mapping behavior. Another evening I might be trying to understand why a context window had changed. Yet another might involve reasoning budgets, offloading settings, or the subtleties of MoE models.

    Some of those investigations solved real problems. Others simply taught me how the system worked. In retrospect, both were valuable.

    As the project grew, I eventually moved everything onto TrueNAS SCALE. That solved some problems and introduced others. Containers made deployment easier, but they also added another layer of abstraction. I found myself learning about application management, container behavior, startup sequences, and resource allocation. At some point I realized I was spending almost as much time learning infrastructure as I was learning AI. That wasn’t something I expected when I started.

    Perhaps the biggest shift in perspective came when I stopped evaluating the project based on what it could do and started evaluating it based on how reliably it could do it. Early on, I cared about capability above all else. I wanted larger context windows, better reasoning, improved memory, and more advanced features. Over time, reliability became more important. A slightly less capable system that works every day is often more useful than a highly capable system that occasionally surprises you. That lesson sounds obvious now, but it wasn’t obvious when I started.

    Today, the system is far more capable than the one I originally envisioned. OpenClaw is running. The memory infrastructure is improving. The models are better. The overall experience feels far more mature than it did during those first experiments. What changed most, however, was my understanding of the problem. I thought I was installing an AI assistant. What I was actually building was a small AI platform consisting of inference servers, memory systems, embedding pipelines, databases, containers, and scheduling mechanisms that all needed to work together. The assistant was simply the visible part. Everything else was the machinery required to make that assistant useful.

    Ironically, I think that’s why local AI has remained so interesting to me. Every layer you uncover reveals another layer beneath it. Every answer leads to another question. Every problem solved teaches you something about the system you didn’t know before. It’s occasionally frustrating. It’s frequently educational. And somehow it’s still fun.

  • Getting OpenClaw and Jarvis Running Locally

    Introduction

    OpenClaw is a self-hosted AI assistant framework that runs on your own hardware. It gives you an AI agent that can manage your files, execute commands, browse the web, and integrate with your tools — all without relying on a cloud service. The appeal of running it locally is obvious: privacy, control, no subscription fees. But it’s also a pretty ambitious project, which is why I’m documenting the setup here. It might save you some time.

    The Setup

    I run OpenClaw inside a Docker container on a TrueNAS server. The container includes:

    • A Node.js-based gateway (running on port 18789)
    • A WebSocket proxy for the chat UI (port 8888)
    • Access to local tools: filesystem, cron scheduling, browser automation, messaging

    The Hardware

    The server has an NVIDIA GeForce RTX 3090 with 24GB of VRAM, which I use for local LLM inference. The main assistant model runs through llama.cpp, and I use Ollama for embeddings and reranking. The 3090 is old enough that it’s affordable, but powerful enough that it handles most tasks well. It’s a good starting point for local AI.

    The Challenge

    The biggest issue I ran into early on was VRAM contention between the persistent llama-server and QMD embeddings. They can’t both run on GPU at the same time, and when they collide the server crashes. The workaround is straightforward: set QMD_FORCE_CPU=1 to run embeddings on CPU, or stop the server temporarily while embeddings need to run. It’s not elegant, but it works.

    What Comes Next

    This blog will document the ongoing setup and troubleshooting. Expect posts about WordPress integration, QMD search indexing, Home Assistant integration, and whatever technical rabbit holes I fall down along the way. If you’re running something similar, feel free to follow along or reach out with your own experience.