GPU sizing for on-premise LLMs: a worked calculation for 50, 200 and 1,000 concurrent users

Sizing GPUs for on-premise inference is arithmetic, not judgement, right up until the last step. Four numbers determine almost everything: the bytes your weights occupy, the bytes one token of KV cache occupies, how many requests are genuinely in flight at once, and how many bytes per second your cards can move. This article does that arithmetic in full for three deployment sizes, states every assumption in a form you can replace with your own measurements, and shows the result that surprises most teams — that beyond a few dozen concurrent long-context requests you are not serving a model, you are streaming a KV cache.

All memory figures below use GB as 10⁹ bytes, matching how GPU vendors label cards.

Start from the model config, not from the parameter count

The two memory numbers that matter both come out of the model’s config.json, and only one of them tracks parameter count. Weight memory is parameters times bytes per parameter. KV cache memory is set by layers, key/value heads and head dimension — and is almost independent of how large the model is.

For Llama 3.1 70B Instruct, from its published config.json: 80 layers, hidden size 8192, 64 attention heads, 8 key/value heads, head dimension 128, vocabulary 128,256.

The 4-bit row is the one people get wrong. A group-128 scheme stores an FP16 scale per group, which adds roughly 0.02 bytes per parameter, and most published checkpoints leave the embedding and output projection at higher precision — on Llama 3.1 70B that is 2 × 128,256 × 8192 parameters, about 3 GB of extra weight you did not budget for. Read the file sizes on disk rather than multiplying.

KV cache is a single formula, and it does not care about parameter count:

bytes_per_token = 2 · num_hidden_layers · num_key_value_heads · head_dim · dtype_bytes

The leading 2 is keys and values stored separately. For Llama 3.1 70B at FP16:

2 · 80 · 8 · 128 · 2  =  327,680 bytes  =  0.328 MB per token

Which means an 8,000-token working context costs 2.62 GB per concurrent request. Compare that with Llama 3.1 8B — 32 layers, 8 KV heads, head dimension 128 — at 131,072 bytes per token, or 1.05 GB for the same 8,000 tokens. The 70B model has nine times the parameters and only 2.5 times the KV cost, because KV scales with layers, not width.

Compute it from the config file rather than from memory:

# kvcalc.py — python 3.11, no dependencies. Point it at a downloaded config.json.
import json, sys

cfg = json.load(open(sys.argv[1]))
layers   = cfg["num_hidden_layers"]
kv_heads = cfg.get("num_key_value_heads", cfg["num_attention_heads"])
# Qwen3 and several newer families set head_dim independently of
# hidden_size / num_attention_heads. Do not assume the division holds.
head_dim = cfg.get("head_dim", cfg["hidden_size"] // cfg["num_attention_heads"])

for name, nbytes in (("fp16", 2), ("fp8", 1)):
    per_tok = 2 * layers * kv_heads * head_dim * nbytes
    print(f"{name}: {per_tok/1e6:8.3f} MB/token"
          f"   {per_tok*8000/1e9:7.3f} GB per 8k-token request")

Concurrent users are not concurrent requests, and the gap is the whole calculation

“1,000 concurrent users” is not a hardware requirement. The hardware requirement is the number of requests in flight at the same instant, and for interactive analytical work that is between a fifth and a quarter of the user count. Getting this conversion wrong by 4x is the most expensive mistake in on-premise sizing, in both directions.

Little’s law gives the mean: in-flight = arrival rate × service time.

State the assumptions explicitly, because they are the parameters you should replace with your own logs before signing a purchase order:

Mean in-flight requests for C concurrent users is C × 21/90 = 0.233 C. But you do not size for the mean, because arrivals cluster. Treating arrivals as Poisson, the 99th percentile is approximately mean + 2.33 · √mean:

Note what the square root does: the design slot count grows sub-linearly with users. Going from 50 to 1,000 users — a factor of 20 — needs 13.5 times the slots, not 20. That is the only economy of scale in this whole calculation, and it is worth knowing before you plan a phased rollout.

The memory-bandwidth roofline tells you when a configuration is impossible

Token generation is memory-bandwidth bound, not compute bound. Every decode step re-reads the weights the GPU is responsible for, plus the KV cache of every active sequence. That gives a hard ceiling that no kernel optimisation can beat:

step_time  ≥  (weight_bytes_per_card + kv_bytes_of_active_sequences_per_card) / HBM_bandwidth
tokens_per_second_per_stream  =  1 / step_time

Published bandwidths: H100 SXM 80 GB at 3.35 TB/s, A100 80 GB at about 2.0 TB/s, L40S 48 GB at 864 GB/s. Real systems land somewhere between 40% and 60% of this roofline once scheduling, kernel launch overhead and prefill interleaving are accounted for. Use the roofline to rule configurations out; use a load test to rule one in.

Prefill obeys different physics. It is compute bound, at roughly 2 · N_params · prompt_tokens FLOPs — for a 6,000-token prompt against 70.6B parameters, about 847 TFLOPs, which is well under a second across four modern cards. If your prompts share a long prefix, which they do in any system that puts schema documentation in front of every question, vLLM’s automatic prefix caching reuses those KV blocks and removes most of that cost. Turn it on before you buy more cards.

Worked example: 50 concurrent users

24 design slots, Llama 3.1 70B at 4-bit, 8,000-token working context. Answer: two H100 80 GB cards, or four A100 80 GB cards.

Memory:

weights                  40 GB
KV  24 slots × 2.62 GB   63 GB
framework overhead     ~10 GB   (CUDA context, graphs, NCCL buffers, activations)
                       -------
total                   113 GB

With vLLM’s default gpu_memory_utilization of 0.9, an 80 GB card offers about 72 GB. Two cards at tensor parallel 2 give 144 GB of usable space against a 113 GB requirement — and TP=2 splits the 8 KV heads cleanly, 4 per rank. Per card: 20 GB weights, 31.5 GB KV, roughly 58 GB used of 72 available.

Now check the roofline. Each card reads 20 GB of weights plus 31.5 GB of KV per decode step: 51.5 GB ÷ 3.35 TB/s = 15.4 ms, a ceiling of 65 tokens/s per stream. At a realistic 50% of roofline that is 32 tokens/s — just above the 30 target. It works, without much margin.

The same configuration on two A100s does not. At 2.0 TB/s the step floor is 25.8 ms, a ceiling of 39 tokens/s, which at 50% efficiency lands at 19 tokens/s per stream — visibly slow. Splitting across four A100s at TP=4 halves the per-card bytes to 25.8 GB, giving a 12.9 ms floor and roughly 39 tokens/s achieved. Older cards do not need more memory here; they need more of them, for bandwidth.

Worked example: 200 concurrent users

76 design slots, same model and context. Answer: four H100 80 GB cards with an FP8 KV cache, or eight with FP16 KV.

weights                  40 GB
KV  76 slots × 2.62 GB  199 GB     <-- five times the weights
framework overhead      ~15 GB
                        -------
total                   254 GB

This is the point the calculation is really about. At 24 slots the weights were 35% of the memory bill. At 76 slots they are 16%, and the KV cache is the system. Four cards at 72 GB usable give 288 GB, which fits — but per card that is 10 GB of weights and 49.8 GB of KV, 65 GB of a 72 GB budget, with no room for a context spike.

Switching the KV cache to FP8 (--kv-cache-dtype fp8, available on Hopper and Ada) halves it to 99.5 GB, taking per-card usage to about 40 GB and restoring headroom. It also halves the bytes read per decode step: 34.9 GB per card, a 10.4 ms floor, roughly 48 tokens/s per stream at 50% of roofline. Four H100s with FP8 KV is a comfortable configuration; four H100s with FP16 KV is one bad day away from preemption, where vLLM evicts and later recomputes requests to free blocks.

If FP8 KV is unacceptable for quality reasons — and it is worth evaluating rather than assuming — the alternative is eight cards at FP16, not four.

Worked example: 1,000 concurrent users

323 design slots. Answer: sixteen H100 80 GB cards as two independent replicas of eight — and first, a serious argument for a smaller model.

At 323 slots with FP8 KV, the cache alone is 423 GB. Eight cards give 576 GB usable, so a single TP=8 replica fits on paper: 5 GB of weights and 52.9 GB of KV per card, about 62 GB of 72. The roofline says 57.9 GB per step at 3.35 TB/s, a 17.3 ms floor, roughly 29 tokens/s per stream at 50% efficiency. That is exactly the target, which means eight cards is the floor of feasibility, not a design.

Two replicas of eight fixes both problems at once. Each replica carries about 162 slots, 26.5 GB of KV per card, a 9.4 ms step floor and roughly 53 tokens/s per stream — and losing a replica degrades the service instead of ending it. At this size, single-replica deployments are a decision to have no maintenance window.

The better question at 1,000 users is whether every request needs the 70B model. A 32B-class model at 4-bit is roughly 18 GB of weights, and — because Qwen3-32B has 64 layers against Llama 3.1 70B’s 80, with the same 8 KV heads and head dimension 128 — about 80% of the KV cost per token. Eight cards serve it comfortably. Routing the routine 80% of traffic to the smaller model and reserving the large one for the queries that need it is worth more than any quantisation decision on this list, and it is the reason DataCopilot holds the active model in a settings table rather than in configuration: model choice has to be changeable without a deployment.

The overheads people forget

Six things consume GPU memory that never appear in a sizing spreadsheet:

• The framework’s own reservation. vLLM’s gpu_memory_utilization defaults to 0.9; you never get 80 GB from an 80 GB card.
• CUDA graphs, captured per batch-size bucket, typically 1–3 GB.
• Activation memory for chunked prefill, governed by max_num_batched_tokens — this scales with your longest prompt, not your average.
• NCCL communication buffers, which grow with tensor-parallel degree.
• Fragmentation, which is why paged attention exists and why the reserve is not zero.
• Co-resident services. A self-hosted BGE-M3 embedding service — 1024-dimension vectors, XLM-RoBERTa based — is about 1.2 GB in FP16. Small, but it needs a card, and putting it on an inference card costs you KV blocks.

Where this calculation stops applying

This arithmetic assumes dense transformer decoders with grouped-query attention, served by a paged-attention runtime, with roughly uniform request shapes. Four situations break it.

Mixture-of-experts models decouple parameter count from bytes read per token. Weight memory follows total parameters; the roofline follows active parameters — but only if expert routing is stable enough to keep the working set resident. Compute both bounds and expect the gap between them to be wide.

Reasoning models with long generated outputs invalidate the service-time assumption completely. If mean output goes from 600 to 6,000 tokens, service time goes from 21 s to about 3 minutes, and in-flight requests rise by roughly 9x for the same user count. Re-run Little’s law; do not scale the GPU count you already computed.

Very long contexts break the linear KV model in practice rather than in theory. The formula stays exact, but at 128,000 tokens a single Llama 3.1 70B request holds 42 GB of KV cache, and concurrency collapses to single digits per card. Above roughly 32,000 tokens, per-request context becomes a capacity policy, not a user preference.

Bursty batch workloads — a nightly report run, a bulk classification job — are not modelled by Little’s law at all, because they have no think time. Size them as throughput problems with a deadline and schedule them outside interactive hours; otherwise a single job will consume every KV block the interactive users need.

Finally, treat every number in this article as a starting point that you then measure. The arithmetic tells you which configurations are impossible, which is genuinely useful and saves a purchase order. It does not tell you which one will feel fast to the person waiting for an answer.