On this page
This is a standalone episode of the F5 BIG-IP series on the LevelUP I.T. YouTube channel. It takes one design problem, rate limiting a large language model API fairly, and builds the answer on an inference platform: an F5 BIG-IP 21.1 at the edge, a token-aware limiter running in Kubernetes, NGINX Gateway Fabric as the inference gateway, the llm-d endpoint picker choosing a replica for every request, and two vLLM replicas serving a Liquid AI model at its full 128,000-token context. Three synthetic developers share the API, and each of them wants something very different from it.
Here is the problem. A public LLM inference API is used by thousands of developers. Their requests run from a few tokens to the longest context the model accepts, so one request can cost orders of magnitude more than another. How do you rate limit that API so it is fair?
Most of the words in that sentence carry weight. "Public" means the clients are not under our control: some are careless, some are hostile, and all of them retry. "Thousands of developers" means every limit is per tenant and has to hold when many tenants are busy at once. "From a few tokens to the longest context" means a request is not a unit of cost, so anything that counts requests counts the wrong thing. And "fair" has at least three different meanings, each of which needs its own mechanism. The answer therefore starts with the cost of a request, then defines fairness, then builds the mechanisms one at a time and shows what each one fixes and what it leaves broken. Every step below shows its configuration and its code first, then the result table, then what the result means.

What you'll learn
- Why a request to a language model has no single price, and which parts of its cost are known when it arrives.
- What vLLM's scheduler actually does with a 120,000-token prompt: chunked prefill, first-come-first-served admission, whole-prompt KV reservation, preemption when the cache runs out, and aborts when a client leaves.
- Three definitions of fairness (rate, capacity under overload, isolation) and the mechanism each one needs.
- Why a requests-per-minute limit punishes the wrong developer, and why a token bucket on prompt tokens leaves the output side unbilled.
- Reserve, reconcile, refund: billing the worst case at the door and settling the real usage at the end, including streams and clients that disconnect, counted by the model's own tokenizer.
- Per-tenant caps on in-flight tokens and concurrent requests, and why they, not the quota, protect another tenant's time to first token.
- Weighted fair queuing by tokens under overload, how it differs from first come first served, and how it compares with llm-d's flow control, which takes turns by request.
- 429 with
Retry-Afterand remaining-budget headers, client backoff with full jitter, and what a retry storm costs the server. - Tiers by priority without killing running requests.
- Running the limiter as several instances with its state in Valkey, changed only by atomic Lua scripts, or with the BIG-IP pinning each API key to one instance.
- What the BIG-IP does at the edge: TLS, a sliding-window request rate per API key in an iRule over the session table, a body cap, a connection limit, OneConnect, and the built-in per-key 429 of APM API Protection.
- Where NGINX Gateway Fabric and the llm-d endpoint picker stop, and why the token-aware layer sits in front of the gateway.
A request has no single price
Start with what a request to a large language model consumes. Inference happens in two phases. In prefill, the model reads the whole prompt and builds, for every prompt token and every attention layer, the keys and values that later tokens will look back at. Prefill compute grows with the prompt's length, and faster than linearly at long contexts, because attention compares each token with every token before it. In decode, the model writes the answer one token at a time; each new token is one more forward pass over the model, so decode time grows with the output's length. And for the whole life of the request, from the first prefill step to the last output token, those keys and values stay in memory: the KV cache. The memory a request holds grows with prompt plus output, and it is held for the whole duration.
On the platform used here, the numbers are concrete. The model is LFM2.5-1.2B-Instruct, run by vLLM 0.32 on CPU with 16 threads per replica. Prefill costs 0.40 ms per token at 1,000 tokens of context, 0.60 ms at 32,000, 0.87 ms at 65,000, 1.15 ms at 100,000 and 1.43 ms at 127,000: the per-token cost more than triples across the context range, which is attention growing with the context. A chat prompt of about 75 tokens (the user's message plus the chat template) gets its first token in 0.07 seconds through the whole path. A 120,000-token prompt, alone on a replica, gets its first token after 167.8 seconds. Decode runs at about 10 ms per output token for a single sequence. The KV cache of this model is about 12 KB per token, and each replica has 174,023 tokens of cache.
So the chat request and the bulk request differ by about 1,600 times in prompt tokens, by about 2,400 times in prefill time, and by about 440 times in the KV cache they reserve (274 tokens against about 121,000, once max_tokens is added). One bulk request holds about 70 percent of a replica's cache by itself, and two cannot fit on one replica at the same time.
Three resources, then: compute for the prompt, time for the output, memory for both. Two facts make rate limiting hard. The first is that spread. The second is the unknown: when the request arrives, the prompt can be counted exactly, with the model's own tokenizer, but the length of the answer cannot. The only thing known about the output is its upper bound, max_tokens. A request that will produce one word and a request that will produce four thousand look the same until the model stops. Any limit that counts requests is counting the wrong thing; any limit that counts only the prompt misses the output; and any limit that only acts on rates misses what a request holds in flight.
