Scaling Ethernet for AI: how Multipath Reliable Connection (MRC) runs RDMA over best-effort Ethernet with packet spraying, SACKs and per-path health

Why RoCEv2 struggles in the largest AI clusters, and how the OCP Multipath Reliable Connection transport runs RDMA over best-effort Ethernet: packet spraying, out-of-order placement, SACK, NACK and trimming, NSCC, per-path health and SRv6.
Hand-drawn Excalidraw banner: RDMA without PFC, Multipath Reliable Connection, with packets sprayed across a spine-leaf fabric

On this page

The largest AI training clusters now wire tens of thousands of GPUs into one synchronous machine, and that machine runs only as fast as its network lets the slowest transfer finish. For years the standard way to carry RDMA over Ethernet has been RoCEv2 on a fabric made lossless with Priority Flow Control. At the scale of the newest clusters, three properties of that design turn into the limit: every flow is pinned to one path, one lost packet triggers a go-back-N burst of resends, and the lossless fabric itself spreads congestion and can deadlock. Multipath Reliable Connection (MRC), published as an open Open Compute Project specification in March 2026, rebuilds the RoCE Reliable Connection transport for ordinary, best-effort Ethernet: each packet takes a path chosen from a per-connection set of entropy values, the receiver places packets straight into memory in any order, losses are repaired selectively, congestion is paced by a window, and the sending NIC tracks the health of each path it uses.

This post is the written deep dive that goes with the film on the LevelUP I.T. YouTube channel. It goes field by field through what MRC changes, why, and what a switch still has to do, with every fact taken from the specification itself, the operators' production paper, the Ultra Ethernet 1.0 specification, the SRv6 RFCs and the peer-reviewed RoCE literature. The sources are listed at the end.

What you'll find here

  • Why collective traffic makes the network decide training speed.
  • RoCEv2 in one section, and the three places it cracks at AI scale: ECMP collisions, go-back-N, and the cost of losslessness (PFC, DCQCN, deadlock, storms).
  • What MRC is: who wrote it, what it extends, what it deliberately leaves out.
  • The multi-plane, two-tier fabric MRC was built to use.
  • Packet spraying with entropy values, out-of-order placement with a RETH in every packet.
  • SACK, NACK, probes and packet trimming, with the actual header fields and reason codes.
  • NSCC, the window-based congestion control MRC adopts from Ultra Ethernet.
  • Per-path health, static routing, structured entropy and SRv6 micro-SIDs.
  • Exactly what a switch must, should and may do.
  • What the operators measured in production, and where MRC sits next to SRD, Falcon, UET and switch adaptive routing.

This post is for paying subscribers only

Subscribe to LevelUp I.T. newsletter and stay updated.

Don't miss anything. Get all the latest posts delivered straight to your inbox. It's free!
Great! Check your inbox and click the link to confirm your subscription.
Error! Please enter a valid email address!