When I started building a local AI stack, I thought I was signing up for a model-selection problem. Like most people getting into local AI, I spent a ridiculous amount of time comparing benchmarks, reading Reddit threads, watching YouTube reviews, and trying to decipher whether one quantization format was secretly better than another. Every few days a new model would appear that was supposedly faster, smarter, or somehow both. The assumption underneath all of that research was simple: if I could just find the right model, the rest of the system would mostly take care of itself.
That assumption survived for about a week.
The first version of the system was actually pretty straightforward. I got OpenClaw running, connected it to a model server, and started experimenting. The responses looked good. The hardware seemed capable. I was running everything on a 3090, which felt like an absurd amount of GPU memory compared to what I had been working with before. Twenty-four gigabytes sounded enormous. More importantly, it felt sufficient. I wasn’t trying to train a foundation model. I just wanted a capable local assistant with memory and some room to grow.
The phrase “room to grow” ended up doing a lot of damage.
One of the things that happens when you start building local AI systems is that every successful step creates three new ideas. Once the model is working, you start thinking about memory. Once memory is working, you start thinking about larger context windows. Then you start reading about agents, retrieval systems, vector databases, and document ingestion pipelines. Before long, you’re no longer running a model. You’re assembling an ecosystem, and every new component arrives with its own expectations about memory, storage, compute resources, and maintenance.
I don’t think I appreciated that at first because the model was always the center of the conversation. Every discussion online seemed to revolve around model quality. People compared reasoning ability, coding performance, benchmark scores, context lengths, and tokens per second. Those things matter, of course, but what I slowly discovered is that they matter a lot less once you start combining multiple AI workloads on the same machine. At that point the interesting problems stop being about intelligence and start being about resource management.
The most dramatic example of this was VRAM contention. My inference server already occupied most of the GPU — that was expected. Modern models are expensive. What I hadn’t fully appreciated was what would happen when embedding maintenance wanted access to the same hardware.
Initially I expected slower performance. What I got was something more serious. Embedding operations would run into problems. The system could become unstable. In some cases the machine itself appeared to hang. Those moments changed the way I thought about the entire stack.
Suddenly benchmark scores didn’t seem nearly as important. It turns out that when you’re staring at a frozen server, you stop caring whether one model is three percent faster than another. The deeper I investigated, the more I realized that resource contention is one of the least discussed aspects of local AI. Most model reviews evaluate a single workload running under ideal conditions. Real systems rarely operate that way.
Real systems have memory databases. Real systems have maintenance jobs. Real systems have background tasks. Real systems have users who inevitably decide to run something new at exactly the wrong moment. The challenge isn’t simply getting individual components to work. The challenge is making them coexist.
That realization led me down a number of interesting rabbit holes. I investigated memory mapping. I looked at tensor placement. I examined offloading behavior. I explored idle unloading options in llama.cpp. Some of those investigations were productive. Others turned out to be distractions. One thing I learned is that warnings are often easier to find than actual problems — a startup log might contain a message that appears critically important while the real issue is occurring somewhere else entirely. It’s tempting to chase every warning because warnings feel actionable, but the difficult part is determining which ones actually matter.
Eventually I found myself focusing less on optimization and more on reliability. This represented a significant change in mindset. Early in the project, I wanted elegant solutions — automatic unloading, dynamic resource sharing, sophisticated scheduling, ideally everything would manage itself. Over time I became more pragmatic. If stopping a service, running a maintenance job, and restarting the service afterward produces predictable results, that’s a perfectly respectable solution. It may not be elegant, but elegance is overrated when compared to stability.
The more I worked with local AI systems, the more I came to appreciate that they are really collections of interacting services rather than single applications. The model is only one component. There are databases, embedding systems, schedulers, containers, APIs, and supporting infrastructure all competing for finite resources. VRAM just happens to be the place where those competing demands become visible first.
In hindsight, the biggest lesson wasn’t about GPU memory at all. It was about systems thinking. The question I ask myself now when evaluating a model is no longer simply “How good is it?” Instead, I find myself asking a different set of questions: how much VRAM does it require? What does it do to the rest of the stack? What tradeoffs does it introduce? What happens when maintenance jobs need to run? How much complexity does it add?
Those questions are less exciting than benchmark charts. They’re also much better predictors of how the system will behave in the real world. The model may be the star of the show, but the resources determine whether the show can go on.
Leave a Reply