Dual-Plane Backend Networks for B300-Class GPU Clusters

Once a training cluster crosses a few thousand GPUs, the backend network stops being a bandwidth problem and becomes a probability problem. Somewhere in that fabric, at any given hour, a link is flapping, an optic is dying, or a hash function is quietly stacking three elephant flows onto the same uplink. Any one of those events can stall a collective operation, and a stalled collective stalls every GPU in the job.

Dual-plane architecture is the design answer showing up in the largest production fabrics: build two identical backend networks instead of one, connect every GPU to both, and let no failure or hash collision touch more than half of anything. This post is a deep dive into how that works, sized for the current generation of hardware: eight-GPU B300-class nodes with 800 gigabit network interfaces per GPU.

The node shapes the network

Start with what a modern training node actually looks like, because the network design falls out of it. An NVIDIA HGX B300 system carries eight Blackwell Ultra GPUs on one baseboard. Inside the chassis, fifth-generation NVLink and NVSwitch give every GPU 1.8 terabytes per second of bidirectional bandwidth to its seven neighbors. Outside the chassis, each GPU gets its own ConnectX-8 network interface at up to 800 gigabits per second, and a separate pair of BlueField-3 DPUs handles storage and north-south traffic.

Notice what the board is telling you. The frontend and backend split is already physical: dedicated data processors for the service side, one dedicated high-speed NIC per accelerator for the scale-out side. The intra-node fabric is so much faster than any network link that the design goal for the external fabric is narrow and honest: carry inter-node collective traffic with no loss, minimal tail latency, and no single point whose failure takes the whole job down.

Two properties of the NIC matter for everything that follows. First, one NIC per GPU means the server is not one endpoint but eight. Second, an 800 gigabit interface in an OSFP cage can operate as two 400 gigabit ports. That second property is the physical hook on which the entire dual-plane design hangs.

What dual-plane means

A dual-plane backend is two complete, identical leaf and spine fabrics, plane A and plane B, with no links between them. Same switch models, same topology, same routing design, physically isolated forwarding domains. Every GPU NIC splits its 800 gigabits into two 400 gigabit ports: the first port cables to a leaf in plane A, the second to the equivalent leaf in plane B.

That is the whole trick, and it is worth being precise about what it is not. It is not a pair of fabrics doing active and standby: both planes carry traffic all the time. It is not multi-pathing inside one fabric: the planes share no switches, no links, and no failure domains. And it is not exotic: from any single plane's point of view, it is a completely ordinary non-blocking Clos network. Nor is it theoretical: Alibaba's HPN, the network behind its LLM training fleet, is a two-tier dual-plane design connecting fifteen thousand GPUs in a pod, documented at SIGCOMM 2024 and motivated by precisely the failure and hashing arguments above.

Three problems get solved at once.

1. Hash polarization stops mattering as much

Classic ECMP picks a path per flow by hashing the packet header. Collective operations produce a small number of enormous flows, exactly the traffic that defeats statistical load balancing. When two elephant flows hash onto the same uplink, that link congests while its neighbors idle, and the whole training step waits for the unlucky flows. This is the straggler effect, and at multi-terabit collective rates it costs real money.

Splitting the fabric into two planes halves the flow population contending in each hash domain and doubles the independent path sets. It does not repeal the birthday paradox, but combined with the load balancing techniques covered below, it removes the worst collision patterns that single large fabrics suffer.

2. Failures lose their blast radius

In a single-plane fabric, a leaf failure strands every NIC attached to it until repair. In a dual-plane fabric, the same failure removes exactly one of each affected GPU's two ports. The bond on the host fails over, traffic continues on the surviving plane at reduced bandwidth, and the training job keeps running. For a job that has been checkpointing for three weeks across thousands of GPUs, the difference between degraded and dead is the entire game.

3. Paths become predictable

Two smaller planes with fixed port mappings produce far more deterministic behavior than one huge fabric with free-form multipathing. Deterministic paths make congestion reproducible, telemetry meaningful, and capacity planning arithmetic instead of simulation.

The last row deserves emphasis: dual-plane is not double the network. The GPUs need the same total bandwidth either way. You are buying the same aggregate capacity and arranging it as two smaller, isolated, more predictable fabrics instead of one large one.

Inside each plane: rails

Within a plane, the topology follows the rail-optimized pattern that collective traffic rewards. A rail groups the same-indexed GPU across every server: rail 1 is every server's GPU 1, rail 2 is every GPU 2, and so on through rail 8. Each rail gets a leaf switch, and NIC cabling follows a fixed mapping, GPU k to rail leaf k, identically in both planes.

The payoff is hop count. Communication libraries deliberately schedule most inter-node traffic between same-ranked GPUs, so the dominant traffic pattern lands on a single leaf, one hop, no spine crossing. When a transfer must cross rails, the library moves the data over NVLink to the GPU that sits on the destination rail first, then sends it out that GPU's NIC. The in-chassis fabric absorbs the rail change; the network only ever sees rail-aligned flows.

A group is the deployment unit: eight rail leaves plus the servers they serve, replicated per plane. Scaling the cluster means adding groups and stitching them with spines, never redesigning what already works.

Keeping two planes coherent without connecting them

Dual-homing a NIC into two switches normally implies multi-chassis link aggregation, and traditional MLAG needs a peer link between the switch pair to synchronize state. In a dual-plane design that peer link is exactly the cross-plane connection you have sworn not to build: the two leaves serving one bond live in different planes. So the real engineering question of dual-plane is this one: how do two switches that must never exchange data-plane traffic both learn where every GPU lives, and prove it to the rest of their planes? Two approaches answer it.

Option 1: NIC-driven host routing

The first approach removes the switch-to-switch relationship entirely and makes the server the source of truth. The sequence:

• The server bonds its two ports, one cabled to each plane, and the NIC or bonding driver emits ARP and neighbor discovery on both uplinks simultaneously, not just on the hash-selected active port. This is the one hard requirement to verify on your NIC before committing to this design.
• Each leaf runs ARP proxy on the server-facing VLAN. It answers ARP locally, so no discovery traffic ever needs to cross planes, and the server never learns that its two ports face two different worlds.
• Each leaf converts every learned neighbor entry into a /32 (IPv4) or /128 (IPv6) host route and injects it into BGP. Reachability for every GPU propagates through each plane independently, carried by the routing protocol instead of a proprietary state-sync channel.

Steady-state forwarding is clean: when GPU A sends to GPU B, the bond hashes the flow onto one port, which selects a plane; the ingress leaf holds a /32 for B learned inside that same plane, and the packet transits leaf, spine, leaf without ever needing the other plane to exist. Failure handling is equally simple. A dead port or leaf stops the bond from using that path, the neighbor entry ages out or the BGP session drops, the /32 is withdrawn inside the affected plane only, and the other plane never notices.

The appeal is operational: no MLAG, no peer link, no inter-switch protocol at all, just ARP and BGP doing jobs they already do. The cost is the NIC-behavior dependency and a fabric-wide host-route table, one /32 per GPU port per plane, which at even the 8,192-GPU ceiling of the design below is a trivial number of routes for modern silicon.

Option 2: EVPN multi-homing

The second approach keeps a switch-side relationship but replaces the proprietary peer link with an open standard: EVPN multi-homing, defined in RFC 7432. The two leaves, one per plane, become EVPN PE devices sharing responsibility for the server-facing bond:

• Both leaves configure the bond member with the same Ethernet Segment Identifier. The shared ESI is what tells the EVPN control plane that two switches front one multi-homed device.
• The BGP EVPN session between the leaves runs over a management or auxiliary link that carries control-plane traffic only. The data planes stay dark to each other; only routes cross that wire, never tenant packets.
• The server spreads its ARP and ND across the bond as normal. Whichever leaf receives a broadcast learns the MAC and IP binding and advertises it as an EVPN MAC/IP route (type 2); each leaf then originates the corresponding /32 host route into its own plane.
• Ethernet auto-discovery routes (type 1) advertise the segment itself, which buys the two properties MLAG needed the peer link for: aliasing, so remote leaves can load-balance toward both PEs even before learning a MAC from each, and mass withdrawal, so a failed leaf retracts one type 1 route and remote devices instantly stop using the entire segment instead of waiting for thousands of per-MAC withdrawals.
• Ethernet segment routes (type 4) drive designated-forwarder election between the two leaves, so flooded traffic reaches the bond exactly once.

Failure behavior is where this option earns its complexity budget: a leaf loss converges at the speed of one route withdrawal, independent of the number of GPUs behind it. The trade is running an EVPN control plane on the backend and operating one more protocol, in exchange for multivendor interoperability and mechanics that any RFC-literate engineer can reason about without a vendor manual.

Either way, the invariant holds: the planes never exchange data-plane traffic, and every GPU is reachable through both. Pick option 1 when your NIC behavior is verified and you want the minimum moving parts; pick option 2 when standards alignment and fast mass convergence matter more than protocol count.

Load balancing past ECMP

Dual-plane halves the collision problem; modern fabrics attack the remainder with mechanisms that move traffic at finer granularity than the flow.

• Flowlet-based adaptive routing. Bursty flows have natural gaps. If a gap exceeds the path delay difference, the next burst can take a different, less congested path without arriving out of order. Switch ASICs track link utilization and queue depth and steer flowlets in hardware. NVIDIA's Spectrum-X platform and Broadcom's recent Tomahawk generations both ship variants of this.
• Telemetry-driven path selection. The fabric measures itself, per-queue occupancy from ASIC registers, nanosecond-scale path measurements from in-band telemetry, and feeds path quality into forwarding decisions, weighted multipathing steered by evidence instead of a static hash. The same telemetry becomes the operator's congestion map for free.
• Packet spraying. The blunt maximum: distribute individual packets round-robin or randomly across all equal paths, achieving near-perfect link utilization and guaranteed reordering. It is only viable because modern RDMA NICs can reorder at line rate, ConnectX-class hardware handles this, and the Ultra Ethernet effort standardizes exactly this combination of spraying transport and reordering endpoints.

These mechanisms compose with dual-plane rather than compete with it: each plane runs its own adaptive routing over its own telemetry, and the planes stay independent all the way up the stack.

A worked design: 2,048 B300 GPUs

Numbers make the architecture concrete. Take a 51.2 terabit switch as the building block for both roles, 64 ports at 800 gigabits, which the optics let you run as 128 logical ports at 400 gigabits. The cluster: 256 HGX B300 nodes, 2,048 GPUs, every GPU contributing one 400 gigabit port to each plane.

Leaves. A rail leaf uses its 128 logical ports as 64 down and 64 up, non-blocking. Sixty-four downlinks means 64 servers per group, 512 GPUs. The full cluster is exactly four groups: 32 rail leaves per plane.

Spines. Each leaf presents 64 uplinks at 400 gigabits. Thirty-two leaves times 64 uplinks is 2,048 spine-facing ports per plane; at 128 ports per spine that is 16 spines per plane, with each leaf holding a clean 4 links to every spine. Full mesh, 1:1 end to end.

Totals. Per plane: 32 leaves, 16 spines. Both planes: 96 switches for 2,048 GPUs with zero oversubscription, every GPU dual-homed, and no failure domain larger than half a NIC.

The same arithmetic scales in group increments:

The last row is the two-tier ceiling for this radix: each leaf lands exactly one link on each of 64 spines. Past 8,192 GPUs you add a third tier or accept oversubscription, and that decision deserves its own post.

Address the fabric like it is 2026: unnumbered BGP over IPv6 link-local addresses on every leaf-spine link, no per-link IP planning, /32 or /128 host routes for the GPUs, identical configuration templates for both planes. The plane is a stamp; press it twice.

What failure looks like now

Trace the event that motivates all of this. A rail leaf in plane A loses power. Sixty-four GPUs each lose one port; their bonds detect the failure and shift traffic to plane B within the bond's failover time. The collective in flight hiccups and retries; the job continues at reduced inter-node bandwidth for those sixty-four GPUs. Nobody's three-week training run dies. Repair happens on a powered-down leaf in an isolated plane, with no risk of touching the live one, and the bond re-balances when the leaf returns.

Compare the single-plane version of the same afternoon: sixty-four GPUs hard down, the job likely torn down and restarted from checkpoint across the whole cluster, and a repair performed inside the one fabric everything depends on. Same broken switch, completely different day.

The takeaway

Dual-plane backend design is what happens when you accept two facts at scale: hash-based load balancing cannot be trusted with elephant flows, and something in a large fabric is always broken. Two identical, isolated, rail-optimized planes turn both facts into manageable arithmetic, half the contention, half the blast radius, twice the predictability, at the same total port count you were going to buy anyway.

The B300 generation makes the pattern almost inevitable: the node hands you eight 800 gigabit interfaces that each split naturally into a pair of 400 gigabit ports. The architecture is sitting in the cabling diagram, waiting to be used. Build the plane once, build it boring, and build it twice.

Sources

• NVIDIA, DGX B300 system specifications: eight Blackwell Ultra GPUs, ConnectX-8 networking, BlueField-3 DPUs, fifth-generation NVLink.
• NVIDIA, ConnectX-8 SuperNIC documentation and the ThinkSystem CX-8 product guide: 800G OSFP operation as one 800G or two 400G ports.
• Qian et al., Alibaba HPN: A Data Center Network for Large Language Model Training, ACM SIGCOMM 2024 (open PDF): the production two-tier dual-plane fabric and the hash polarization measurements behind it.
• IETF, RFC 7432, BGP MPLS-Based Ethernet VPN: the EVPN multi-homing and Ethernet Segment mechanisms.
• NVIDIA, NCCL PXN deep dive: rail-aligned communication and cross-rail transfers over NVLink.
• NVIDIA, Spectrum-X networking platform: per-packet adaptive routing with endpoint reordering for RoCE.
• Broadcom, Tomahawk 5 (BCM78900 series): cognitive routing and dynamic flowlet steering in the switch ASIC.
• The Ultra Ethernet Consortium: the standardization effort around multipath packet spraying with reordering endpoints.