Inference-Aware Load Balancing for LLMs (part 3): Gateway API meets inference, NGINX Gateway Fabric and the InferencePool

The Gateway API Inference Extension built on a lab, configuration first: the CRDs, NGINX Gateway Fabric as the inference gateway, the InferencePool and the llm-d endpoint picker, the route and its stream timeouts, and the first requests routed by the picker.
Hand-drawn: NGINX Gateway Fabric and the llm-d router choosing a model replica per request

On this page

A Kubernetes Service sends each connection to a pod without knowing anything about it: not its queue, not its KV cache, not even the model it serves. The Gateway API Inference Extension changes that with one resource, the InferencePool, and one component, the endpoint picker, which chooses a replica for every request from the model servers' own metrics. This is part 3 of Inference-Aware Load Balancing for LLMs: the whole stack built on the lab, every manifest shown before it is applied, with NGINX Gateway Fabric as the inference gateway and the llm-d router as the picker. This post goes with a video on the LevelUP I.T. YouTube channel.

The protocol: the gateway asks the picker, the picker names a pod, the gateway proxies there
The protocol: the gateway asks the picker, the picker names a pod, the gateway proxies there

What you'll see

  • The pool's members: two real LFM2.5-1.2B replicas, spread over two nodes.
  • What a Service knows, and what the Inference Extension adds, compared with F5 CIS.
  • The protocol between gateway and picker, and what the picker can and cannot see.
  • The Gateway API v1.6.3 and Inference Extension v1.6.2 CRDs, server-side, and the safe-upgrades policy.
  • NGINX Gateway Fabric's values, the InferencePool field by field, the Gateway, the route and the stream timeouts.
  • Requests through the gateway: the picker's choice in NGINX's log, the spread over two replicas, and the cost of the extra hop.

The pool's members

A router needs copies of one model to choose between; the big node holds one 24B copy, so the pool here is the 1.2B model, two real replicas (decision 12 of the series plan). The generated Deployment's relevant fields:

replicas: 2
labels: {app: lfm-1-2b, model: lfm2.5-1.2b}
nodeSelector: {f5lab.test/llm-tier: small}
topologySpreadConstraints: [{maxSkew: 1, topologyKey: kubernetes.io/hostname, whenUnsatisfiable: DoNotSchedule, ...}]
args: ["LiquidAI/LFM2.5-1.2B-Instruct", "--revision", "b9d816ab...", "--served-model-name", "lfm2.5-1.2b", ...]
- {name: VLLM_CPU_KVCACHE_SPACE, value: "4"}
- {name: VLLM_CPU_OMP_THREADS_BIND, value: "0-9"}
resources: {requests: {cpu: "10", memory: 12Gi}, limits: {cpu: "10", memory: 12Gi}}

Scaled to two, one replica landed on k8s-inf1 and one on k8s-inf2. A warm-up request to each, straight to its pod address, took 6.5 s to the first token: the cold start of a freshly started server.

What a Service knows, and what the extension adds

A Service sends each new connection to a ready endpoint without looking at it: Cilium's datapath and kube-proxy's iptables mode choose at random, IPVS mode round robin by default. For a model server that means no queue depth, no cache usage, no prefix cache and no model name, and with HTTP keep-alive a client can pin many requests to one pod. F5 CIS (parts 22-23 of the F5 series) solves a different problem: it writes pod addresses into a BIG-IP pool, and the BIG-IP then balances per connection with its own methods. The Inference Extension moves the choice itself, per request, to a component that reads the model servers. In this series the BIG-IP keeps the edge and the gateway and picker choose the replica; part 10 joins them.

The protocol

NGINX receives the request and, instead of choosing a pod, hands it to the endpoint picker using Envoy's external processing protocol (ext_proc) over gRPC. The picker answers with one header, x-gateway-destination-endpoint, naming a pod address and port, and the gateway proxies there. In the background the picker scrapes every member's /metrics (every 50 ms in this version) and keeps its own index of the prompt blocks each pod has served. NGINX Gateway Fabric 2.8.1 implements ext_proc with a small JavaScript handler inside NGINX and a shim container in the same pod: NGINX sends the request's headers and body to the shim on loopback, and the shim speaks gRPC to the picker. The picker therefore sees the whole request body (model, prompt, max_tokens) but never the response, so it cannot count a request as finished when it ends; part 7 shows what that changes.

What the gateway sends the picker, and what it does not
What the gateway sends the picker, and what it does not

This post is for paying subscribers only

Subscribe to LevelUp I.T. newsletter and stay updated.

Don't miss anything. Get all the latest posts delivered straight to your inbox. It's free!
Great! Check your inbox and click the link to confirm your subscription.
Error! Please enter a valid email address!