On this page
I ran two elephant flows across my leaf-spine fabric six times, with different ports every time, and watched them land on the same uplink six times out of six, while the identical link next to it carried zero bytes. Half the bandwidth I built, invisibly idle, with every dashboard green. This post is that experiment, the strange place the investigation ended up, and the math that says production fabrics live with a statistical version of the same disease.
Post 1 established the two facts this one stands on: AI traffic is a handful of giant synchronized flows, and a training step runs at the speed of its slowest one. Today's question: given several elephants and several equal paths, who decides which elephant takes which path? The answer is a hash function, and the hash function was never told about elephants.
What ECMP promises
Equal-cost multipath is the quiet workhorse of every modern fabric, including mine (the leaves each hold two equal BGP paths to every destination, one per spine, exactly as designed in the Fabric Lab underlay). For each packet, the switch hashes header fields, classically source and destination IP and ports, and the hash picks the uplink. The same flow always hashes the same way, so packets never arrive out of order; different flows scatter across paths, so the load spreads.
Notice the assumption doing all the work: different flows, many of them. With thousands of mice, the law of large numbers makes the spread beautiful. Nobody promised anything about four flows. Hash functions do not balance load; they randomize it, and randomizing four elephants is a coin toss you can lose.
What my fabric did
The harness: two concurrent iperf3 elephants from a master node to a worker, crossing the fabric between VTEP pairs, twenty seconds per trial, different client ports every trial so every flow is a fresh hash input. I watched the transmit counters on all four possible exits, both cluster leaves times both spine uplinks, because on this fabric a flow makes two hashing decisions: the server's bond picks a leaf, the leaf picks a spine.
Total polarization, every single time. The bond stage behaved like a hash should, sometimes leaf 1, sometimes leaf 2. The spine stage did not: for this source and destination pair, every flow chose spine-2, six trials out of six, regardless of ports. Two elephants shared one wire and halved each other, exactly the arithmetic from post 1, while the other wire idled.
The chase, because six out of six is not luck
Two flows colliding by chance is a 50 percent event. Twelve flows all picking the same uplink is one in four thousand, and that is not a hash being unlucky, that is a hash not being consulted. So I chased it, and I am showing the chase because the disagreements are the most instructive thing I found all week:
The routing table holds both paths. The configured hash mode includes layer 4 ports. A packet capture on the hypervisor (every link in this lab is a Linux bridge, a fact Part 2 of the Fabric Lab explains and this post cashes in) shows the VXLAN outer source ports varying per flow, so entropy genuinely reaches the wire. And the forwarding table prints exactly one next-hop, via spine-1, for traffic that measurably left via spine-2. Four witnesses, four stories.
Then the counterexample that settled it. Same harness, different destination pair, a single TCP flow this time:
A single flow split exactly 50/50 across both spines. Real per-flow hashing never does that, on purpose: a flow stays on one path so its packets arrive in order. So here is the honest verdict, stated plainly: my virtual switch's data plane is broken in an educational way. Some destination pairs pin all traffic to one uplink, others spray per packet, and neither is what physical Nexus silicon does, which is hash the five-tuple correctly. I am not showing you a vendor bug in a shipping switch; I am showing you what elephant traffic looks like when path selection fails, on a platform where it happens to fail deterministically. Simulators earn their keep by being cheap, and they charge for it in moments like this.
The production version is statistical, and worse than you think
On real hardware the hash works, so replace "always" with "eventually," and eventually is closer than intuition suggests. This is the birthday problem wearing a network engineer's badge. With a working, perfectly uniform hash and eight equal uplinks:
Two elephants dodge each other 87.5 percent of the time. Four elephants: the odds of a clean spread are down to 41 percent, worse than a coin flip. Six: under 8 percent. Eight elephants on eight links, the exact shape of a GPU server pushing gradients through a fabric, avoid each other two times in a thousand. Collision is not the edge case; a clean spread is. And when mice collide nobody notices, buffers absorb microseconds of overlap. When elephants collide, both run at half speed for their entire lifetime, post 1's barrier makes the whole job wait for them, and nothing anywhere logs an error, because nothing is wrong. Every link is up. The hash did its job. The job is just the wrong job for this traffic.
What the industry actually does about it
Everything real fabrics deploy against this is a variation on one idea: stop letting a static per-flow hash make a decision this expensive. Flowlet switching splits an elephant at its natural pauses and re-hashes each burst, safe because a gap longer than the path delay difference cannot reorder packets. Adaptive routing lets the switch watch queue depths and steer around the collision the hash created. Packet spraying goes furthest, spreading every packet across all paths and accepting reordering as the price, which only works when something downstream, a DPU, a reordering-tolerant transport, puts the puzzle back together, and my accidental 50/50 split is a live demonstration of exactly why that half of the system is mandatory. And rail-optimized topologies sidestep the lottery entirely by giving each GPU's NIC its own dedicated plane: no shared choice, no collision to lose. Those last two get their own posts, rails at post 5 and the spray-and-reorder future at post 8.
Until then, the operational takeaway fits in one line: monitor the maximum member link, never the aggregate. My six trials averaged a healthy-looking 50 percent utilization across the uplink pair while one wire ran hot and the other ran empty. The aggregate lies. The max tells the truth.
Runs: iperf3 3.16 between Kubernetes nodes over the VXLAN/EVPN fabric from the Fabric Lab series, virtual Nexus 9300v leaves and spines, counters read before and after each 20 second run, six trials with distinct client ports, plus the single-flow counterexample and the full diagnostic chain shown above. The polarization you saw is real measured behavior of this lab and exaggerated relative to physical switches, for the data plane reasons I documented rather than hid. The birthday table is arithmetic, not measurement, and you can check every entry with a pocket calculator. Next post, the fabric stops forgiving: lossless Ethernet, where I configure PFC and ECN on real ConnectX-6 silicon and show you the counters that make RoCE possible, and the pause storms that make it dangerous.
