What a language model does, and what it costs to run

A language model takes a sequence of tokens and predicts the next one, repeatedly. Everything a deployment does with it — answering from documents, extracting fields, drafting a reply — is that operation with a carefully built input. The useful mental model is not intelligence, it is a fixed-size budget you are filling: the context window is the number of tokens the model can attend to at once, and everything you want it to consider must fit inside it alongside the answer.

This is why retrieval exists. You cannot put 100,000 documents in the window, so you put the six passages most likely to contain the answer, and the quality of the deployment becomes the quality of that selection. It is also why long prompts get slower and more expensive in a predictable way rather than a mysterious one.

Parameter count is not quality; it is capacity, and capacity costs memory. Quantisation reduces the bytes used per parameter — four-bit weights are a quarter of the sixteen-bit ones — which is what lets a fourteen-billion-parameter model run on a single workstation card at all. The trade is quality, and how much you lose depends on the task, which means you measure it rather than read it.

The rule worth carrying: choose the smallest model that passes your evaluation for your task, then spend the memory you saved on context and concurrency.

Memory for serving is three terms: the weights, the key-value cache that grows with context length and with the number of requests in flight, and an overhead allowance. This site computes it and prints every input, because a hardware recommendation with no arithmetic behind it is a sales figure.

It reports two totals, and the difference matters. The typical figure assumes a fraction of your users are generating at any moment; the peak assumes all of them are. Quote the peak alone and you specify a machine nobody needs. Quote the typical alone and you have hidden the nine-o’clock problem, when everyone opens the tool at once.