Inference-Aware Load Balancing for LLMs (part 1): from connections to tokens, what changes when the backend is an LLM

Two requests reach the same load balancer, for the same model, on the same server. Each is one TCP connection and one HTTP POST. The first asks a language model to say OK and is finished in 0.21 seconds. The second asks for a detailed paragraph about BGP and streams back for 15.6 seconds. Nothing in either request told the load balancer which one it would be. This is part 1 of a new series, Inference-Aware Load Balancing for LLMs, which builds, on a real data-centre fabric with an F5 BIG-IP at the edge, the layer that can see tokens, models and caches. This post goes with a video on the LevelUP I.T. YouTube channel.

Part 1 is the foundation: what a load balancer knows about a request, what a token is, how a chat completion streams back, the two phases of every request (prefill and decode), the three numbers that describe it (TTFT, TPOT and end-to-end), what batching does to them, and Little's law. Every number in this post was measured on the lab, against one real model, LFM2.5-24B-A2B, running on vLLM on a CPU node.

What you'll see

• What a classic load balancer knows about a request, and why connection counting worked for web traffic.
• Tokens, measured on the model's own tokenizer, and how a model writes one token at a time.
• The OpenAI chat completions API: what crosses the wire, and what it never says.
• The lab for this series, and one real 24-billion-parameter model deployed on a CPU node, configuration first.
• A streamed answer event by event, a 48-second first request after start, and a reasoning model thinking before it answers.
• Prefill and decode, TTFT, TPOT and end-to-end time, with a long and a short prompt measured.
• One, four and eight streams at once, and Little's law checking the client.
• What a load balancer can and cannot know, and what a BIG-IP in front of model servers can already do today.

What a load balancer knows about a request

A BIG-IP sees the client's and the server's addresses and ports, and the state of every TCP connection; it counts the open connections on each pool member. With an HTTP profile it reads the method, path, host and headers; a health monitor tells it whether a member is up; a cookie or a source address keeps a client on one member. The load-balancing methods measured in part 5 of the F5 series use exactly that: round robin sends each new connection to the next member, least connections to the member with the fewest open connections, and the ratio, fastest, observed and predictive methods weight members by ratios, connection counts or response times averaged over the last few seconds.

All of them rest on one assumption: that a connection is roughly one unit of work, and that it ends soon. For web traffic it holds. A page, an image or an API call is small, the server has most of the answer when it starts sending, and requests differ over a narrow range. So the number of open connections tracks the work in progress on each server, and a recent average response time is a fair forecast of the next one.

Now the two requests from the opening, measured on this lab against the same model, one at a time:

A 74-times difference, one connection each, and nothing in the headers to tell them apart. The second answer streams for the whole 15.6 seconds, so its connection stays open for the whole time, and while it is being written it shares the server's compute with every other request on that server.

Tokens

A language model does not read letters or words. A tokenizer splits the text into pieces from a fixed vocabulary (128,000 entries for this model), and the model works on the numbers (ids) of those pieces. When it writes, it produces one new token id at a time. Every price, limit and speed in LLM serving is therefore counted in tokens.

The series' client (llm.py, one Python file, standard library only, in the lab repository) shows the split on the model itself: it calls vLLM's /tokenize endpoint for the ids, then /detokenize on each id to show its piece of text.

llm tokens http://lfm-24b-a2b:8000 lfm2.5-24b-a2b 'The BIG-IP load-balances connections, not tokens.'
6 words -> 14 tokens
[The]  [ B]  [IG]  [-]  [IP]  [ load]  [-b]  [al]  [ances]  [ connections]  [,]  [ not]  [ tokens]  [.]
ids: [597, 378, 5320, 21, 5563, 7524, 1484, 285, 2452, 13212, 20, 513, 40043, 22]

Plain English is a little over one token per word; addresses, names, numbers and configuration are much denser. The length of a request in bytes or words is only a rough guide to its length in tokens: a gateway that wants the real number must run the model's tokenizer.

How a model writes one token

Each token so far becomes a vector (2,048 numbers for this model). The vectors pass through the model's 40 layers; in the attention layers each position looks back at every earlier position. At the end, the last position's vector becomes a score for each of the 128,000 vocabulary entries, one entry is chosen, and that is the next token. It is appended, and the whole step runs again. This is autoregressive generation: every token depends on all the tokens before it, so an answer is written one step after another, never in parallel, and takes time in proportion to its length.

Choosing the entry is sampling. The scores become probabilities; with a temperature above zero the pick is random, weighted by those probabilities, and a higher temperature gives more varied text. With temperature 0 the most likely token is always chosen, so the same prompt gives the same answer. Every measurement in this series uses temperature 0, so a change in speed is the system, not the text. The client's --fixed option also sets min_tokens equal to max_tokens and ignore_eos, so an answer is exactly N tokens long.

The chat completions API

Almost every model server speaks the API OpenAI defined: POST /v1/chat/completions with a JSON body.

{
  "model": "lfm2.5-24b-a2b",
  "messages": [{"role": "user", "content": "In two sentences, what does a load balancer do?"}],
  "max_tokens": 220,
  "temperature": 0,
  "stream": true,
  "stream_options": {"include_usage": true}
}

A gateway could read the model (so the pool that can answer: part 6 routes on it), the messages (so the prompt's length in tokens, and the prefill cost), max_tokens (only a ceiling), and whether the answer streams. What the request never says is how many tokens the answer will have. The model decides that while it writes: it stops at its end-of-sequence token or at max_tokens.