On this page
This is part 1 of a three-part series on a two-site edge for an AI inference API: part 1, a second pod network with Multus on the EVPN VXLAN fabric; part 2, the F5 BIG-IP edge: BGP route health, Advanced WAF and CIS; part 3, failover between the two sites with BIG-IP DNS.
An inference pod usually needs two networks. The cluster network carries the API calls, the readiness probes and the Kubernetes Service that a load balancer points at. A second network carries bulk traffic, such as model weights from storage and cache transfers between replicas, at jumbo MTU straight onto the fabric. Multus adds that second interface in a few lines of YAML. Services, NetworkPolicy and probes keep working only on the first one.
This part measures what changes when a pod has net1 next to eth0 on an EVPN VXLAN fabric. The outputs come from one recorded run plus two short addenda against the same pods. Two results were less obvious: where an asymmetric reply gets dropped, and how two nodes end up with the same address.
The lab
The fabric is Enterprise SONiC 4.6: a spine and two leafs, an eBGP underlay, an EVPN overlay and symmetric IRB with anycast gateways on both leafs. Every Kubernetes node is multihomed to both leafs with EVPN multihoming. Each node has one LACP bond, each bond is an Ethernet segment, and leaf1 is the designated forwarder on every segment by preference (200 against 100). Kubernetes 1.35 is built with kubeadm and uses Cilium as the cluster network. Multus 4.3, the thick plugin, adds net1: a macvlan in bridge mode on the node's VLAN 20 sub-interface at MTU 9000, with addresses from Whereabouts. VLAN 10 carries the nodes in one VRF. VLANs 20 and 30 share a second VRF, and a test host on VLAN 30 acts as a client on another subnet.
What the run shows
- Services, EndpointSlices and cluster DNS return only the
eth0address. Thenet1address appears only in the pod's network-status annotation. - A deny-all NetworkPolicy blocks
eth0. Requests tonet1still get an answer. - The leafs learn the pod's
net1MAC and advertise it in EVPN like any host. - 9000-byte packets cross the fabric routed, inside VXLAN, without fragmenting.
- A request from another subnet that arrives on
net1is answered out ofeth0, and Cilium drops the reply on the node. Routes in the attachment fix it; the sbr plugin fixes it too. - For
rp_filter, the kernel uses the higher of the global and per-interface values, and the drop counters match that in all five combinations tested. - host-local, given the same range on every node, hands two pods the same address.
- With
net1cut off, an HTTP readiness probe stays green. A probe that pings throughnet1fails.
What the Kubernetes API sees
The pod below has both attachments. eth0 holds the Cilium address and the default route. net1 is a macvlan child of bond0.20 at MTU 9000, with an address from the data subnet and one extra route, to the VLAN 30 subnet.
The attachment is a NetworkAttachmentDefinition. The first one below uses Whereabouts for addresses and a routes list for subnets that should be answered through net1. The second chains the CNI sbr plugin instead. The asymmetric-reply section compares the two.
From the cluster's side, a headless Service returns the eth0 address and cluster DNS resolves to it. The EndpointSlice for the model servers lists only their eth0 addresses. A client that needs the data network has to learn the address some other way, from the k8s.v1.cni.cncf.io/network-status annotation or from its own discovery.
NetworkPolicy
A namespace-wide deny-all ingress policy is applied, then a neighbouring pod calls both addresses of the same pod. The request to eth0 times out. The request to net1 returns 200.
This is expected. The cluster network's plugin enforces NetworkPolicy, and net1 traffic never passes through it. Filter a secondary network somewhere else: in the fabric, with an ACL on the VLAN or a separate VRF as in this lab; on the host; or with a plugin built for multi-network policy.
The fabric learns pods
macvlan gives each pod its own MAC on the wire, so the leafs learn pods the same way they learn servers. The pod's net1 MAC is in both leafs' MAC tables on the node's PortChannel. EVPN advertises it as a type-2 route twice, once with only the MAC and once with the MAC and its IP. The Ethernet segment table shows the multihoming underneath: each node's bond is local and remote on leaf1, with leaf2 as the peer.
Every pod on the data network adds a MAC entry and a host route to the fabric, so size its tables for pods as well as servers.
Jumbo frames
The data VLAN runs at 9000 on the pods and 9216 on every fabric interface, which leaves room for the VXLAN header. A don't-fragment ping with an 8972-byte payload (9000 bytes of IP) reaches the host on the other subnet with no loss. One byte more fails on the pod itself, before anything reaches the wire. The spine capture shows the packet routed into the L3VNI, VNI 10900, with 8980 bytes of ICMP inside the VXLAN frame.
Asymmetric replies
The test host on VLAN 30 pings the net1 address of pod pc, whose attachment has no route back to VLAN 30. The request arrives on net1 and the reply follows the default route out of eth0. Nothing comes back to the client, and a capture on the node's uplink sees nothing leave.
Cilium's drop monitor gives the reason: Invalid source ip. The reply carries the net1 address as its source and tries to leave through eth0. Cilium drops anything leaving a pod's interface with a source address that doesn't belong to that endpoint. The leaf's node VRF, which holds only the node subnet, has no route to VLAN 30 either.
Both fixes worked in the run. The first adds the remote subnet to the attachment's IPAM routes, so pod pa answers those requests through net1 and the data gateway. The second chains the sbr plugin. Pod pe gets a rule that sends anything sourced from its net1 address to a separate table whose default route is the data gateway.
Use explicit routes when you know which subnets use the data network, which in an inference fabric is usually the case. Use sbr when clients on net1 can come from anywhere and listing them isn't practical.
rp_filter
Reverse-path filtering decides whether the kernel accepts that asymmetric request at all. It uses the higher number of conf.all.rp_filter and the interface's own value, where 0 is off, 1 is strict and 2 is loose. Because loose is the higher number, it wins over strict. Each row below sets both values, sends three requests from the VLAN 30 host to pod pc, and reads the pod's counters before and after.
With strict as the higher value, either globally (1 and 0) or on net1 (0 and 1), all three requests were dropped by the reverse-path filter. With loose on either level, they were accepted, including the row with strict on the other level. A global strict value overrides a per-interface 0, so set net1 to 2 explicitly if the design depends on loose mode. If net1 drops a remote client, check both values and the reverse-path counter first.
host-local IPAM
host-local keeps its allocation state on each node's disk, so two nodes given the same range allocate from it independently. Two pods, one per node, both got 10.1.20.203. An ARP for that address got replies from two different MACs. Leaf1's EVPN table held a MAC/IP route for 10.1.20.203 with only one of them, the MAC of the pod on the second node. Both nodes' host-local stores show the same reservation.
Whereabouts keeps one pool for the whole cluster. Every pod that used it in the run, across both nodes and both namespaces, appears in that one pool with its own allocation. On a subnet shared by several nodes, use Whereabouts, static addresses, or a different host-local range per node, never the same range everywhere.
Readiness probes
The kubelet sends readiness probes to the pod's cluster address, which is on eth0. VLAN 20 was removed from one node's Ethernet segment on both leafs, which cuts off net1 on that node, and pings to it failed. Pod pa, with an HTTP readiness probe, stayed Ready and stayed in its Service. Pod pf, whose readiness probe pings the data gateway through net1, was NotReady at the next check and marked not ready in the EndpointSlice. With VLAN 20 back, both were Ready again.
If a replica is useless without its data network, make its readiness probe use that network. An exec probe that reaches something on net1, such as the gateway, a storage target or a peer, is enough. The probe then depends on the fabric, so keep it cheap and give it a failure threshold that tolerates a dropped packet.
The next part puts an F5 BIG-IP in front of the inference API on this fabric.
The eBGP underlay and EVPN VXLAN fabric this post runs on are built step by step in my book, BGP for AI Infrastructure, including where Multus and the backend rails meet the pod.
