On this page
Every AI data center is really two networks wearing one badge. There is the network your users touch, and there is the network your GPUs live on. They carry different traffic, fail in different ways, and are designed against different physics. Treating them as one problem is the fastest way to build a cluster that looks impressive on paper and trains slowly in practice.
This post walks through the split: what the frontend network does, what the backend network does, why the backend has rules of its own, and how to design one that keeps your accelerators busy instead of waiting.
Two networks, one data center
The cleanest way to classify traffic in an AI facility is by direction. North-south traffic enters and leaves the building: user requests, API calls, inference queries, dataset ingest, backups, management sessions. East-west traffic never leaves: GPU talking to GPU, node synchronizing with node, checkpoint streaming to storage.
The frontend network owns the first kind. The backend network owns the second. The reason to physically separate them is not tidiness, it is interference. A training job will happily saturate every link it can reach for hours at a time. Put that on the same wires as your inference API and your users feel every gradient exchange. Put your users on the GPU fabric and every noisy burst of north-south traffic shows up as jitter inside a collective operation that was supposed to complete in microseconds.
The frontend network: the service entrance
The frontend is the part that looks familiar. It is a leaf and spine Ethernet fabric connecting CPU head nodes, service machines, storage frontends, and the outside world. If you have built a data center network in the last decade, you have built this.
Its priorities, in order, are availability, security, and scale:
- Availability. Sessions must survive a device failure. That means ECMP across spines, port channels with LACP toward servers, fast failure detection with BFD, and dual-homed everything. A frontend that drops a link should lose capacity, never connectivity.
- Security. This is the only network exposed to anything untrusted. Firewalls, load balancers, segmentation, and policy all live here, because this is where the attack surface is.
- Scale. User demand grows in requests per second, not in gigabits per GPU. Moderate oversubscription is normal and correct: paying for a non-blocking fabric to carry API calls is money taken from the part of the network that actually needs it.
Latency matters on the frontend, but it matters at human scale. Tens of milliseconds decide whether an application feels responsive. Nothing about a user request breaks if it takes one millisecond longer. Hold that thought, because the backend inverts it completely.
One more property worth naming: the frontend can stretch. Because its tolerances are human tolerances, you can spread it across halls, buildings, even regions. Load balancers and anycast happily hide the distance. This is why stretching a deployment across sites is a frontend story, and, as we will see, why they are almost never a backend one.
The backend network: the GPU fabric
The backend exists for exactly one workload: accelerators exchanging tensors. During distributed training, GPUs run collective operations, all-reduce, all-gather, all-to-all, where every participant must send, receive, and combine data before anyone can continue. The job advances at the speed of the slowest exchange. There is no such thing as a partially fast collective.
That single fact drives every design rule the backend has:
- Loss is not an option. The fabric carries RDMA traffic, memory-to-memory transfers that bypass the CPU entirely. RDMA transports degrade brutally under packet loss: a single drop can stall a queue pair and drag a whole collective with it. The backend must behave like a lossless system, which Ethernet does not do by default.
- Tail latency is the metric. Average latency tells you nothing. The collective finishes when the last flow finishes, so the 99.9th percentile is the number that sets your GPU utilization. Sub-microsecond port-to-port latency and tight jitter are the target, and every queue an elephant flow can build is the enemy.
- Bandwidth is provisioned per GPU, not per server. Modern accelerators each get their own high-speed NIC, commonly 400 or 800 gigabit today. A server with eight GPUs is not one host on this network, it is eight line-rate endpoints that happen to share a chassis.
- No oversubscription. The backend is built non-blocking, one-to-one from leaf to spine. Oversubscribing a training fabric does not save money, it converts capital you already spent on GPUs into idle time.
The economics deserve to be said plainly. In an AI cluster the accelerators dominate the bill of materials, and a meaningful share of wall-clock training time is communication. Every percent of communication time the fabric shaves off is a percent of very expensive hardware handed back to compute. The frontend affects how fast users feel. The backend affects what your cluster costs per unit of training.
Making Ethernet lossless: PFC, ECN, DCQCN
If the backend runs RDMA over Converged Ethernet, RoCEv2, three mechanisms carry the lossless burden:
- PFC (Priority Flow Control) pauses a single traffic class hop by hop before a buffer overflows. It is the safety net, and it is a blunt one: pause frames propagate, and a misbehaving edge can spread head-of-line blocking upstream. PFC should be the mechanism of last resort, engaged rarely.
- ECN (Explicit Congestion Notification) marks packets as queues build instead of dropping them, giving endpoints an early, precise congestion signal.
- DCQCN ties the two together: the receiver echoes those marks back to the sender as congestion notifications, and the sending NIC paces its injection rate, so congestion is resolved at the source before PFC ever has to fire.
Getting this triple tuned, buffer thresholds, marking profiles, rate parameters, is most of the operational craft of a RoCEv2 fabric. The alternative is InfiniBand, which is lossless by construction: credit-based flow control means a sender cannot transmit into a buffer that does not exist. InfiniBand also brings a subnet manager with global visibility and, on capable hardware, in-network reduction that offloads parts of the collective into the switches themselves. The trade is cost and a second operational skill set versus the Ethernet tooling and multivendor supply chain your team already knows. Both build excellent backends. What matters is that the choice is made on lossless behavior, tail latency, and operability, not on the logo.
Frontend vs backend at a glance
| Frontend network | Backend network | |
|---|---|---|
| Primary traffic | North-south: users, APIs, inference, ingest | East-west: GPU collectives, parameter sync |
| Endpoints | CPU servers, storage frontends, appliances | GPUs, one NIC per accelerator |
| Transport | TCP/IP over Ethernet | RDMA: RoCEv2 or InfiniBand |
| Loss tolerance | Retransmission is routine | Effectively zero: lossless required |
| Latency target | Milliseconds, human-perceived | Microseconds, tail-dominated |
| Oversubscription | Acceptable, often 3:1 or more | None: non-blocking 1:1 |
| Failure goal | Sessions survive | Job completion time survives |
| Can it stretch sites? | Yes: multi-site works | Rarely: millisecond WAN latency breaks sub-microsecond synchronization |
| Cost of getting it wrong | Slower user experience | Idle GPUs, longer training, higher cost per run |
Designing the backend: rails first
Backend topology starts from an observation about the traffic itself: collective algorithms are structured. In a cluster of eight-GPU servers, GPU 0 overwhelmingly talks to GPU 0 on other servers, GPU 1 to GPU 1, and so on. Communication libraries arrange it this way on purpose. The topology that exploits it is the rail design.
Rail-only. Take sixteen servers with eight GPUs each, 128 GPUs total. Deploy eight leaf switches, one per rail. Rail switch k connects GPU k from every server. Now every same-rank exchange in the whole cluster is a single switch hop, and traffic that must cross rails rides the NVLink or equivalent interconnect inside each server rather than the network. For clusters up to a few hundred GPUs this is the simplest fabric that is also the fastest, and it needs no spine at all.
Rail-optimized. Past the port count of a single rail switch, you keep the rail structure and add a spine layer above it, a Clos fabric stitching multiple rail groups together. Same-rank traffic still lands one hop away inside its group; cross-group traffic takes the spine at full, non-blocking bandwidth. This is the shape of most large training fabrics today, scaling from hundreds to many thousands of GPUs by adding groups rather than redesigning.
Three rules travel with either variant. Keep the fabric non-blocking end to end. Spread each server's NICs so that no single leaf failure removes more than one rail from any server. And balance flows deliberately, because a handful of elephant flows on a hashed path can create exactly the hotspot the topology was built to avoid.
The takeaway
The frontend network and the backend network share cables, racks, and a room, and almost nothing else. One is judged by whether people notice it; the other is judged by whether GPUs wait on it. The frontend earns its keep with availability, security, and sensible economics. The backend earns its keep with losslessness, tail latency, and a topology that mirrors how collectives actually communicate.
The deeper lesson is that a great backend is not designed after the cluster, it is co-designed with it. The parallelism strategy, the collective algorithms, the NIC-per-GPU layout, and the switch topology are one system. Design them together and the network disappears into the training loop, which is the highest compliment a GPU fabric can receive. Design them apart and you will meet the difference on your training bill.
