Inference-Aware Load Balancing for LLMs (part 3): Gateway API meets inference, NGINX Gateway Fabric and the InferencePool
The Gateway API Inference Extension built on a lab, configuration first: the CRDs, NGINX Gateway Fabric as the inference gateway, the InferencePool and the llm-d endpoint picker, the route and its stream timeouts, and the first requests routed by the picker.