On this page
The bare-metal LoadBalancer problem
Create a Service of type LoadBalancer on a cloud cluster and magic happens: an IP appears, traffic arrives. Do it on bare metal and the external IP sits at <pending> forever, because there's no cloud to conjure one. The homelab workarounds are all unsatisfying: NodePort makes users remember :32342, and L2-mode load balancers gArp their way to a single-node bottleneck.
There's a grown-up answer, and it's the same one real datacenters use: make the cluster a BGP speaker and let it tell the network where service IPs live. My k8s cluster sits on a VXLAN/EVPN leaf-spine fabric of NX-OS switches, Cilium is the CNI, and every app URL in my lab resolves to an IP that exists only as a BGP advertisement. This post walks the whole path, from IP pool to leaf routing table.
The Kubernetes side: three CRDs and a Gateway
Cilium ships both halves of the puzzle: LB IPAM (who assigns the IPs) and a BGP control plane (who tells the network). The setup is declarative and small:
The consumer here is Gateway API rather than legacy Ingress: one Gateway (ai-gw) terminates TLS for *.ocp.lab with a wildcard cert, and each app is an HTTPRoute hanging off it. When the Gateway was created, Cilium's IPAM allocated 192.168.200.0 from the pool to its LoadBalancer service. At this point the IP exists in etcd and nowhere else; no switch on the fabric knows it.
Which services get announced (and which never will)
An underrated question in BGP-to-the-fabric designs: does every LoadBalancer service automatically become routable from your entire network? Here the answer is deliberately no. The advertisement CRD carries a label selector, and only services wearing the label get spoken for:
Follow that chain bottom-up and the design clicks. Gateway API's infrastructure.labels stamps everything Cilium generates for the Gateway, including its LoadBalancer service; the CiliumBGPAdvertisement selects on bgp-announce existing; and anything unlabeled stays cluster-internal no matter what type it claims to be. The day a stray LoadBalancer service appears in a random namespace, it gets an IP from the pool but the fabric never hears about it. Access to the routed network is an explicit, greppable decision.
Telling the fabric: eBGP from every node
The BGP control plane runs in the Cilium agent on each node. My nodes are AS 65100 and peer eBGP with their leaf switches' anycast gateway addresses, masters and workers landing on different leaves:
The three CiliumBGPClusterConfigs are worth a look: each selects its nodes by a fabric-pair label and defines the eBGP session for that group, masters toward the cluster leaves (AS 65201/65202) and workers toward the storage leaves (AS 65203), each peering with its leaf pair's anycast gateway address. That structure is not cosmetic. Peer ASN, peer address, and timers differ per leaf pair, and keeping one small config per node group means adding a fabric pair later is a new 15-line CRD rather than surgery on a shared one.
That last detail is the design. Each node claims "192.168.200.0/32 lives at me," which is true, because Cilium can accept traffic for the VIP on any node and forward it to a backend. The receiving network is free to believe all of them at once:
The switch side: ECMP for free
On the NX-OS leaves this arrives as ordinary eBGP in the tenant VRF, and where multiple nodes on the same leaf advertise the VIP, the RIB does exactly what BGP RIBs do with equal-cost external paths:
What happens after the leaf picks a node is Cilium's half of the load balancer. This cluster runs kube-proxy-replacement: true, so the VIP is serviced entirely in eBPF: the receiving node's XDP/tc programs match the destination, pick a healthy Gateway backend, and forward. With routing-mode: tunnel, a request landing on a node that doesn't host the chosen backend takes one extra encapsulated hop inside the cluster. That's the classic externalTrafficPolicy trade: this setup keeps the default (any node accepts, possible second hop, source IP SNATed) in exchange for every advertised path being equally valid, which keeps the BGP story simple and the failure domain per-node. If you need client source IPs preserved, you flip the service to Local and accept that only backend-hosting nodes should win the ECMP hash.
Failure handling falls out of the protocol. A node dies, its BGP session drops, the leaf withdraws that path within the hold time, and traffic re-hashes to the survivors. Nothing to orchestrate: convergence is the feature you inherit by speaking routing instead of gratuitous ARP.
Two design notes that usually surface as production surprises. First, convergence speed is your BGP hold time, and on this fabric the established node sessions negotiate Cilium's default 90 seconds, keepalive 30, unless you tune it; my Palo Alto failover work on the same lab settled on keepalive 1 / hold 3 for aggressive detection where it mattered, and BFD, the usual right answer, is not usable on the N9Ks here. Know your worst-case blackhole window instead of discovering it. Second, the /32: advertising host routes rather than the whole pool means each service IP is individually claimable and individually withdrawable, and the leaves' RIBs stay honest about exactly which VIPs exist. The cost is one route per service, which at homelab or even enterprise scale is noise.
Crossing the edge: the border leaves
Everything so far lives inside the fabric. The border leaves are where the VIP starts its journey toward users, and the route changes shape on the way. The compute and storage leaves aggregate the pool and export it into the EVPN control plane, so the borders learn the VIP not as six node routes but as one type-5 prefix reachable through the VXLAN overlay:
Two details worth noticing. The aggregation boundary is deliberate: inside the fabric, per-node /32s give surgical failover; toward the edge, one summary keeps the firewall RIB small and stable, so a node dying causes churn inside the overlay but the edge never sees it. And the six-way ECMP means the borders spray inbound flows across every VTEP that can reach the cluster, before Cilium even gets involved.
The firewalls: two pinholes and a deny
North of the borders sits a Palo Alto Active/Active pair (route-based, its own story), eBGP AS 65300 against the borders. It learns the fabric's reality, thirteen prefixes including the VIP summary, and advertises two back. Then it does what firewalls are for:
External access is exactly two doors. gw-dnat publishes an office-LAN address, 10.80.5.140, and translates it to the Cilium VIP; the matching security rule allows internet zone to fabric zone for that pre-NAT destination only (PAN evaluates policy on the pre-NAT address with the post-NAT zone, a detail that bites everyone once). dns-dnat publishes 10.80.5.115 and forwards it to port 5353 on the lab's service node, which is the second half of the trick below. Egress in the other direction uses per-device source NAT pools, one bound to each Active/Active member, so return traffic always lands on the firewall that owns the session.
Split-view DNS: same names, two answers
One puzzle remains: internal DNS resolves every app to 192.168.200.0, which the office LAN can't route. The answer is a second dnsmasq instance on the service node, listening on port 5353 behind that DNS pinhole, serving an external view: every ocp.lab name resolves to the firewall's published address instead. Same hostnames everywhere, different truth depending on where you're standing. Tested live from a machine on the office LAN:
End to end
DNS is the last mile: the lab's dnsmasq resolves every app hostname to the VIP, and the Gateway sorts requests by SNI and Host header behind it:
What I'd tell past me
Three notes for anyone building this. First, per-group CiliumBGPClusterConfigs beat one giant config: masters and workers peer with different leaves and different peer ASNs, and separating them keeps each config trivially readable. Second, remember your leaves' BGP defaults: eBGP multipath had to be allowed on the NX-OS side before the second path was installed (maximum-paths under the VRF's address family). Third, advertise specific pools, not everything: the advertisement CRD selects which service IPs get announced, and the day someone creates a stray LoadBalancer service you'll be glad it isn't automatically routable from the entire fabric.
And the obvious comparison: MetalLB in L2 mode gets you the same pending-to-IP magic with less setup, but one node answers ARP for each VIP, making it the throughput ceiling and the failover delay. MetalLB's own BGP mode closes that gap; if you're already running Cilium though, the BGP control plane is built in, the LB IPAM is built in, and one less DaemonSet is one less thing.
The takeaway: BGP isn't just how the internet routes; it's the cleanest API a bare-metal cluster has for talking to real network hardware. Cilium speaks it natively, your switches have spoken it for decades, and the meeting point is a LoadBalancer that behaves like the cloud's, except you can read every line of its state on both sides.
