Serve a model on hardware you chose

A model fits when its weights and cache are inside the card’s memory. It serves when it answers your users at an acceptable latency while they are all using it. Those are different tests, and the second one fails on machines that pass the first.

Unified-memory workstations are the clearest example: they hold large models comfortably because they have a lot of memory, and they generate slowly because memory bandwidth rather than capacity is the limit. This site keeps one in the alternatives for that reason and says so, rather than recommending it first.

Measure three things and record the conditions: tokens per second for a single request, time to first token, and what both become at your expected concurrency. A server that produces sixty tokens per second alone and eight per user with ten in flight has told you your capacity, and that is the number that belongs in a design document.

When more requests arrive than can be held in flight they queue, and queueing shows up to users as latency rather than as an error. Know where that threshold is on your hardware before a customer discovers it at nine in the morning.

There are two families. One optimises for throughput under concurrency and is what you want when a team depends on the deployment. The other optimises for getting a model running in one command and is right for a pilot, a single user, or a machine that is not a server. The recipes here carry both and say which is which.

Pick the simple one for the first slice if it meets your measured target. Complexity you did not need is complexity the customer will have to operate after you leave.