On this page
A small model for cheap requests, a big one for hard ones, and an adapter for one task, behind one address. This is part 6 of Inference-Aware Load Balancing for LLMs: two models and a LoRA adapter behind one Gateway, a pool and an endpoint picker per model, a header that chooses the pool and what happens when it disagrees with the request body, a real LoRA adapter with its cost measured, and an 80/20 split between two models for a rollout. Configured on camera, counted. This post goes with a video on the LevelUP I.T. YouTube channel.

Why many models
Size (the 1.2B model answers simple requests at about 66 tokens/s on this lab, the 24B reasoning model at about 35), cost (a few CPUs against a whole node), tasks (a fine-tuned adapter for one domain), and versions (a new model taking a growing share). The client talks to one address; the gateway must route on what the request asks for.
A pool per model
An InferencePool selects the pods of one model, because its picker scores replicas that can all serve the request. A second model is a second pool with its own picker, from the same chart (generated lab/k8s/inference/epp-24b-values.yaml: the same values with modelServers.matchLabels.app: lfm-24b-a2b):
helm install lfm24pool /stage/charts/llm-d-router-gateway-v1.0.0.tgz -n llm -f /k8s/inference/epp-24b-values.yaml --wait
NAME SELECTOR EPP
lfm12pool lfm-1-2b lfm12pool-epp
lfm24pool lfm-24b-a2b lfm24pool-eppWhat an InferencePool holds
kubectl -n llm get inferencepool lfm12pool -o yaml
spec:
appProtocol: http
endpointPickerRef: {failureMode: FailOpen, group: "", kind: Service, name: lfm12pool-epp, port: {number: 9002}}
selector: {matchLabels: {app: lfm-1-2b}}
targetPorts: [{number: 8000}]The selector names the model servers, the target port is where vLLM's OpenAI API listens, and the endpoint picker reference is the Service the gateway calls over gRPC on 9002 for every request. FailOpen means the gateway picks a pod by itself if the picker cannot answer; part 7 measures what that does under overload. The route never names a pod or a Service: it names the pool, and the pool names its picker.
Choosing the pool
NGINX Gateway Fabric 2.8.1 matches paths, headers and query parameters, not JSON bodies, and the llm-d router v1.0.0 has no body-based router, so a header chooses (generated models-routes.yaml):
kind: HTTPRoute
metadata: {name: llm, namespace: llm}
spec:
parentRefs: [{name: llm-gw}]
hostnames: ["llm.f5lab.test"]
rules:
- matches: [{headers: [{name: x-llm-model, value: big}]}]
backendRefs: [{group: inference.networking.k8s.io, kind: InferencePool, name: lfm24pool}]
- backendRefs: [{group: inference.networking.k8s.io, kind: InferencePool, name: lfm12pool}]
---
kind: HTTPRoute
metadata: {name: rollout, namespace: llm}
spec:
hostnames: ["rollout.f5lab.test"]
rules:
- backendRefs:
- {group: inference.networking.k8s.io, kind: InferencePool, name: lfm12pool, weight: 80}
- {group: inference.networking.k8s.io, kind: InferencePool, name: lfm24pool, weight: 20}When several rules match, Gateway API takes the most specific: exact path, then the longest path prefix, then a method match, then the most header matches, then the most query-parameter matches. The default rule has no match, which means path prefix / with no headers, so the header rule wins whatever order the rules are written in. A header value that matches no rule (x-llm-model: huge) falls to the default pool, not to an error. The two routes never compete: their hostnames differ.