A local language-model box, measured rather than guessed

Why run a model locally at all

Two reasons, and neither of them is cost. The first is that a good deal of what I would like to hand a language model is data I am not free to paste into someone else's API - unpublished genotypes, draft manuscripts under review, collaborators' phenotype files. The second is that I wanted to understand the failure modes of these systems the way I understand the failure modes of a variant caller, and you cannot learn where something breaks if you only ever see it through a hosted endpoint that hides the knobs.

So the goal was not to run the largest possible model. It was to have one machine where every parameter is visible and every claim about performance has a number behind it.

The shape of the thing

The stack is a web panel plus a small router. Tools point once at a single OpenAI-compatible URL, and the panel decides what is behind it - a local engine (llama.cpp, vLLM or Ollama) or a cloud provider. Switching backends does not mean reconfiguring every tool that talks to it, which was the actual daily annoyance I was trying to remove.

The router is deliberately a much smaller thing than LiteLLM: no spend tracking, no load balancing, no dependencies. It forwards requests and applies generation defaults. If the needs grow past that, the right move is to swap in LiteLLM rather than to keep growing this one.

What actually fits on 16 GB

The model I settled on is Qwen3.8-27B in Unsloth's dynamic IQ4_XS quantisation. The weights alone are 14.25 GB. The card holds 15.92 GB. Every decision after that is an argument about the remaining 1.67 GB, which has to cover the KV cache, the vision projector, and any speculative-decoding head.

I wanted four roles out of this one quantisation: highest quality for text, fastest interactive response, working vision, and a long-context mode. All four load at 15.4 GB.

The interesting part is the last column. The panel ships a VRAM estimator - a linear fit over measured runs - and for two of these presets it predicted a number past the end of the card: 16.5 GB for the speculative-decoding preset, 16.1 GB for the 107k one. On that basis both should fail to load. Both run, at 15.4 GB, and the 107k preset is serving requests as I write this.

I should be blunt about how I found that out, because it is the same mistake the piece is about. I had the fitted coefficients, I extrapolated them to 107k, I got 16.7 GB, and I wrote that long context does not fit on this card. It fits. A linear fit is only evidence inside the range it was fitted on, and I used one as if it were a measurement. The machine was running the answer the whole time.

So the arithmetic is not the constraint. The real one is subtler, and it is the vision preset. That preset loads cleanly, reports itself ready, and sits at the same 15.4 GB as the others - and then an image request runs out of memory. The number that would have predicted that failure is not VRAM at load. It is peak VRAM during a real request, which nothing in the panel reports and no load-time figure can stand in for.

That is the useful generalisation. Three of these four presets are fine. The one that is broken is broken in a way that every load-time measurement in the system says is fine, which is exactly why it cost me time. A preset that loads is not a preset that works.

The one number worth reporting

Speculative decoding through a multi-token-prediction head is the largest single performance lever on this box, and it is also the one that standard tooling cannot see. Measured on the smaller GSQ-RCO IQ2_S build at a 113k context with q8_0 KV, generating 256 tokens, averaged over two runs:

That is +32% on decode for an 11% drop in prefill - a trade worth taking for interactive work, where you wait on generation and not on prompt ingestion. It would be the wrong trade for a batch job that feeds in long documents and asks for short answers.

Note which build that was measured on. It is the smaller IQ2_S quantisation, not the IQ4_XS one described above, so the throughput figures do not transfer - the IQ4_XS speed preset carries a draft head at the same 15.4 GB, but its tokens per second have not been measured. The shape of the trade should hold. The magnitude is an open question, and I would rather leave it open than quote a number from the wrong build.

Two traps that cost me a day each

1. The benchmark binary and the serving binary do not load models the same way. llama-server defaults to fitting a model into available VRAM; llama-bench does not. A model larger than VRAM therefore fails outright under the benchmark unless you pass a fit target and leave layer offload on automatic. Pinning -ngl 99 on an 82 GB model simply errors. The deeper lesson is the one that generalises: a configuration that benchmarks well is not proven until the binary that actually serves requests has run it. I ended up building a mode that starts the real server with the benchmarked settings, precisely because the two disagreed often enough to matter.

2. Flag lists silently multiply. Most llama-bench flags take a list and run the cross product. Setting the KV cache type to q4_0,q8_0 sets both the key and the value cache to that list - which is four combinations, not two. I read the first results table as having duplicate rows. It did not; I had run twice the jobs I thought I had. The fix was to show key and value types as separate columns so the rows cannot be misread.

Neither of these is subtle once you know it. Both cost real time because the tool reported success while measuring something other than what I believed it was measuring - which is the same failure mode that makes a genomics pipeline produce a clean-looking VCF of the wrong thing.

What this is not

It is not a contribution to modelling, and it is not a general-purpose deployment story. Sixteen gigabytes is a hard ceiling, and a good deal of the configuration in the repo exists only to work around that ceiling - quantised KV caches, expert offload to CPU, aggressive context management. On a larger card most of it would be unnecessary.

Nor is the record complete. The speed preset still shows its estimated 16.5 GB in the panel rather than the 15.4 GB it actually uses, and it has no throughput measurement at all. Neither does the long-context preset. The standard I am aiming at is that every number says which kind it is - and the estimates that are still standing in for measurements are the next thing to fix, not something to describe as finished.

What the project is, I think, is a worked example of holding a fast-moving tool to the same standard as a scientific method: write down the machine, publish the configuration that produced each number, and say which of the numbers you actually measured. That is a low bar. It is also not where most of this tooling currently sits.