Build a Cloud-Style Fabric Lab, Part 8: Cilium

One helm install, three jobs: the pod network, kube-proxy's whole duty in eBPF, and BGP from every node to its leaf pair's anycast gateway. Plus the consent chain that keeps VIP announcements deliberate, and the anycast peering quirk worth knowing.
Fabric lab series, part 8: cilium

On this page

Part 7 ended on a cluster that was deliberately broken: NotReady nodes, a Pending CoreDNS, no CNI, and no kube-proxy. This part installs Cilium and all of it resolves at once, because Cilium is taking three jobs here, not one: the pod network, kube-proxy's entire duty in eBPF, and a BGP speaker on every node that teaches the fabric where Kubernetes services live. The third job is what makes this cluster belong on the fabric we built, and it connects directly to the Cilium BGP deep dive already on this blog; this post is the build-out, that one is the theory.

Cilium's three jobs: pod network, kube-proxy replacement in eBPF, and BGP from every node, plus the consent chain for announcements and the switch side of the handshake
Three jobs, one helm install, and a consent chain that keeps announcements deliberate.

One helm install, three jobs

The entire user-supplied configuration is fourteen lines, and each one earns its place:

Cilium helm values: cluster-pool IPAM over 10.244.0.0/16, k8sServiceHost pointing at haproxy, kubeProxyReplacement true, tunnel routing with VXLAN
Fourteen lines. The two subtle ones: k8sServiceHost, and VXLAN riding VXLAN.

Two lines deserve a pause. k8sServiceHost: 10.130.0.11 points the agents at haproxy directly, and it must, because of a chicken-and-egg that kube-proxy clusters never see: normally pods reach the API through the kubernetes ClusterIP, which kube-proxy translates; here Cilium is the thing that will translate ClusterIPs, so it cannot depend on one to reach the API before it is running. Part 7's endpoint design pays for itself again.

And routingMode: tunnel with VXLAN means pod traffic tunnels between nodes, which then rides the fabric's own VXLAN between leaves. Yes, VXLAN inside VXLAN. It nests without drama, costs another 50 bytes of headers, and Part 3's 9216-byte underlay swallows it whole. The native-routing alternative, where pod CIDRs are advertised into the fabric per node, is the sophisticated choice; tunnel mode is the boring one that keeps the fabric ignorant of pod internals, and boring is what you want the first time the two systems meet.

This post is for subscribers only

Subscribe to LevelUp I.T. newsletter and stay updated.

Don't miss anything. Get all the latest posts delivered straight to your inbox. It's free!
Great! Check your inbox and click the link to confirm your subscription.
Error! Please enter a valid email address!