Inference-Aware Load Balancing for LLMs (part 6): many models behind one gateway, model routing, LoRA and a rollout split

Two models and a LoRA adapter behind one Gateway: a pool and picker per model, a header that picks the pool and the 404 when it disagrees with the body, a real LoRA adapter and its measured cost, and an 80/20 rollout split.
Hand-drawn: LoRA slows the base model 66.6 to 49.7 tokens/s, and an 80/20 split measured at 166/34

On this page

A small model for cheap requests, a big one for hard ones, and an adapter for one task, behind one address. This is part 6 of Inference-Aware Load Balancing for LLMs: two models and a LoRA adapter behind one Gateway, a pool and an endpoint picker per model, a header that chooses the pool and what happens when it disagrees with the request body, a real LoRA adapter with its cost measured, and an 80/20 split between two models for a rollout. Configured on camera, counted. This post goes with a video on the LevelUP I.T. YouTube channel.

Two pools behind one Gateway: a header picks the pool, a second host splits 80/20
Two pools behind one Gateway: a header picks the pool, a second host splits 80/20

Why many models

Size (the 1.2B model answers simple requests at about 66 tokens/s on this lab, the 24B reasoning model at about 35), cost (a few CPUs against a whole node), tasks (a fine-tuned adapter for one domain), and versions (a new model taking a growing share). The client talks to one address; the gateway must route on what the request asks for.

A pool per model

An InferencePool selects the pods of one model, because its picker scores replicas that can all serve the request. A second model is a second pool with its own picker, from the same chart (generated lab/k8s/inference/epp-24b-values.yaml: the same values with modelServers.matchLabels.app: lfm-24b-a2b):

helm install lfm24pool /stage/charts/llm-d-router-gateway-v1.0.0.tgz -n llm -f /k8s/inference/epp-24b-values.yaml --wait
NAME        SELECTOR      EPP
lfm12pool   lfm-1-2b      lfm12pool-epp
lfm24pool   lfm-24b-a2b   lfm24pool-epp

What an InferencePool holds

kubectl -n llm get inferencepool lfm12pool -o yaml
spec:
  appProtocol: http
  endpointPickerRef: {failureMode: FailOpen, group: "", kind: Service, name: lfm12pool-epp, port: {number: 9002}}
  selector: {matchLabels: {app: lfm-1-2b}}
  targetPorts: [{number: 8000}]

The selector names the model servers, the target port is where vLLM's OpenAI API listens, and the endpoint picker reference is the Service the gateway calls over gRPC on 9002 for every request. FailOpen means the gateway picks a pod by itself if the picker cannot answer; part 7 measures what that does under overload. The route never names a pod or a Service: it names the pool, and the pool names its picker.

Choosing the pool

NGINX Gateway Fabric 2.8.1 matches paths, headers and query parameters, not JSON bodies, and the llm-d router v1.0.0 has no body-based router, so a header chooses (generated models-routes.yaml):

kind: HTTPRoute
metadata: {name: llm, namespace: llm}
spec:
  parentRefs: [{name: llm-gw}]
  hostnames: ["llm.f5lab.test"]
  rules:
  - matches: [{headers: [{name: x-llm-model, value: big}]}]
    backendRefs: [{group: inference.networking.k8s.io, kind: InferencePool, name: lfm24pool}]
  - backendRefs: [{group: inference.networking.k8s.io, kind: InferencePool, name: lfm12pool}]
---
kind: HTTPRoute
metadata: {name: rollout, namespace: llm}
spec:
  hostnames: ["rollout.f5lab.test"]
  rules:
  - backendRefs:
    - {group: inference.networking.k8s.io, kind: InferencePool, name: lfm12pool, weight: 80}
    - {group: inference.networking.k8s.io, kind: InferencePool, name: lfm24pool, weight: 20}

When several rules match, Gateway API takes the most specific: exact path, then the longest path prefix, then a method match, then the most header matches, then the most query-parameter matches. The default rule has no match, which means path prefix / with no headers, so the header rule wins whatever order the rules are written in. A header value that matches no rule (x-llm-model: huge) falls to the default pool, not to an error. The two routes never compete: their hostnames differ.

This post is for paying subscribers only

Subscribe to LevelUp I.T. newsletter and stay updated.

Don't miss anything. Get all the latest posts delivered straight to your inbox. It's free!
Great! Check your inbox and click the link to confirm your subscription.
Error! Please enter a valid email address!