On this page
This is the first post in a new series about the networks underneath AI, written from the network engineer's side of the room. There is a lot of good content explaining GPUs, NVLink, and training clusters to software people. This series goes the other way: what does AI actually do to a network, and what does a network person need to unlearn, relearn, and design for. One rule throughout: wherever a claim can be measured, I measure it on my own lab and show the output. Where it cannot (I do not own a rack of GPUs, and I will say so every time it matters), I label it analysis.
Post 1 is the foundation everything else stands on: AI traffic is not a bigger version of the traffic you know. It is a different species, and two of its defining behaviors fit in a small lab.
Two kinds of traffic
Here is a Kubernetes cluster and an RDMA link, both mine, photographed in the same hour:
The Kubernetes cluster is carrying 91,685 connection-tracking entries right now, and a sample of them by port tells you what they are: application requests, DNS lookups, Cilium health probes, API server calls. Thousands of flows, each tiny and short lived. This is the traffic profile every enterprise network you have ever touched was shaped by, and everything in the classical toolbox, ECMP hashing, buffers, oversubscription ratios, connection-oriented load balancing, quietly assumes it. Networking people call these flows mice.
The second capture is one ib_write_bw flow between two nodes: 7.94 Gb/s, held flat for sixty seconds, roughly 59.5 GB through a single flow. No ramp-up sawtooth, no idle gaps, just a constant firehose until the transfer is done; I re-ran it with five-second interval reporting and every window came back an identical 7.94. That is an elephant, and it is what a GPU pushing gradients looks like to your network. (My link runs at SDR because of a cable with opinions, a story already told on this blog. The shape is what matters here, and the shape is the same at 400 Gb/s, just with more zeros.)
An AI training cluster inverts your traffic distribution. Instead of millions of mice with the occasional elephant, the workload IS the elephants: every GPU pair exchanging gradient shards, every step, for weeks. The mice are still there, control plane, storage metadata, but they are rounding error. Designs tuned for the first world behave badly in the second, and the rest of this series is about the specific ways.
The barrier: why averages lie to you
The second behavior is stranger and matters more. Distributed training is built on collective operations: after each compute step, every worker must exchange its piece of the result with the others (the canonical one is called allreduce) before anyone can start the next step. If you want the two-sentence version of how those bytes move without the kernel touching them, that is RDMA, and this blog has a whole post on the verbs machinery. What matters tonight is not how the bytes move but the completion rule: a collective is a barrier. Nobody proceeds until everybody finishes.
That one rule demolishes the most comfortable metric in networking: the average. I emulated one collective step on the lab: three workers, an identical 9.83 GB to deliver each, launched simultaneously. One worker's path is throttled to 2 Gb/s, my stand-in for a degraded link, a congested queue, a flapping optic, any of the ordinary ways one path in a fabric goes bad while the rest stay healthy:
The two healthy workers finished their share in 27.6 seconds and then sat idle for 13.1 seconds, 32 percent of the step, waiting for the throttled one to crawl home at 40.75. The step took 40.7 seconds. Not the average of the three flows, not the aggregate bandwidth, the maximum of the completion times, full stop. In a real cluster those idle workers are the most expensive hardware in the building doing nothing, and the step repeats thousands of times, so the 32 percent is not an anecdote, it is a tax rate.
This is the single most important mental shift for a network engineer entering this world: you are no longer designing for throughput, you are designing for the worst path. A fabric that delivers 99 percent of flows beautifully and starves 1 percent is not a 99 percent fabric. For a synchronized job it is exactly as fast as that unlucky 1 percent.
The wire keeps its own books
One more lesson hides in the same run, and it sets up the next post. Add up the three concurrent flows: 2.98 + 2.98 + 1.99 = 7.95 Gb/s, within 0.01 Gb/s of what the single elephant achieved alone. Three flows did not create bandwidth; they divided it. Obvious when stated, and yet the whole discipline of fabric design for AI is downstream of this arithmetic: since elephants saturate whatever they share, the game is entirely about which flows end up sharing. Classical fabrics decide that with a hash function that never met an elephant it couldn't mistake for a mouse. That failure mode has a name, I can reproduce it on my leaf-spine on demand, and it is the subject of post 2.
What changes in your head
If this world is where your network career is heading, the reframes start here:
Tail beats average. Every metric that matters is a p99 or a max, because the barrier converts your worst path into everyone's step time. Dashboards built around averages will tell you the fabric is fine while the job crawls.
Few, huge, synchronized. Capacity planning shifts from "how many flows" to "which flows collide." A fabric with a handful of simultaneous elephants is a scheduling problem wearing a networking costume.
Idle silicon is the cost function. The thing being optimized is not link utilization, it is GPU wait time. A network decision is good if expensive processors wait less, and every post in this series ultimately cashes out in that currency.
The failure modes are quiet. Nothing in the barrier run was down. Every link was up, every flow completed, and the port counters agreed: zero receive errors, zero discards, checked before and after the run. The fabric was, by every classical health check, green, and the job still ran 32 percent slower. AI fabrics fail by degrees, which is why the observability and design patterns in the coming posts exist.
Everything measured here ran on my virtual lab: Ubuntu VMs holding real ConnectX-6 silicon passed through with VFIO, SR-IOV virtual functions on the worker side and a full physical function on the far card, with an InfiniBand link between the two physical cards, and a Kubernetes cluster on a virtual leaf-spine fabric (which has its own build series). The link runs at SDR, the numbers are small, and no GPU was involved, the transfers emulate the traffic pattern, not the computation. None of that changes the two results, because both are shape, not scale: the mice-to-elephant inversion and the max-not-average barrier arithmetic hold at any line rate. When a post in this series needs scale I cannot produce, I will say so in the first paragraph, not the footnotes.
The series
Where this goes, every other day: ECMP's elephant problem, demonstrated live on my fabric. Lossless Ethernet, PFC, ECN, and DCQCN with the counters to prove it. NVLink and NVSwitch translated for network people. Rail-optimized topologies. Storage traffic on the same wires, measured. The anatomy of a GPU cluster's three networks. And an honest read on the InfiniBand versus Ethernet fight. Next up: ECMP has an elephant problem, in which my leaf-spine hashes two elephants onto the same uplink and leaves another one empty, on camera.
