LLM inference infrastructure, explained: GPUs, the KV cache, batching, the network and load balancing
The whole stack behind a language model's answer, from one model file to a fleet: why decode is memory-bound, how the KV cache sets capacity, what batching really buys, how models are split across GPUs, the RDMA fabric behind them, and why round-robin load balancing fails for inference.
