LLM inference infrastructure, explained: GPUs, the KV cache, batching, the network and load balancing

This post goes with a deep-dive video on the LevelUP I.T. YouTube channel. It builds the infrastructure behind a language model's answer from the bottom up: one model file, one GPU, one request, then batching, multiple GPUs, the network that joins them, and the load balancer in front. Every number is either measured on screen or taken from a published specification, and each one says which.

One request to a 7-billion-parameter model (Qwen2.5-7B, FP16): a 2,150-token prompt, then 256 generated tokens. The server read the prompt at 789 tokens per second and wrote the answer at 29.6. Same weights, same chip, same request: reading was 27 times faster than writing. The rest of this post explains why, and what that one fact does to every layer of AI infrastructure.

The whole stack in one picture

This is the map for everything below. Requests arrive over the front-end network at a gateway; an endpoint picker chooses a replica from the model servers' own signals (queue length, KV cache use, cached prefix); GPU servers run the model in a prefill pool (reading prompts, compute-bound) and a decode pool (writing tokens, memory-bound), each with eight GPUs joined by NVLink and HBM holding the weights plus the KV cache; and a back-end RDMA fabric moves the KV cache between them. The two lines underneath are the rules that size it. Each section of this post takes one part of the picture, starting from the GPU.

What you'll find here

• What a model physically is, and the arithmetic that sizes it.
• Prefill and decode, and the two latencies users feel.
• The three numbers of a GPU, the roofline, and why decode is memory-bound.
• The decode ceiling, predicted and then measured on three models.
• The KV cache: exact sizing, grouped-query attention, prefix caching.
• Batching, continuous batching, PagedAttention, and the real batch-size limit.
• Splitting models across GPUs, and the traffic each split puts on the network.
• The back-end fabric: RDMA, RoCEv2, rails, lossless and loss-tolerant designs.
• Why round robin and least connections fail for inference, and what replaces them.
• Disaggregated prefill and decode, autoscaling, and goodput.