InfiniBand for the Ethernet Engineer.

At some point the question finds you. A job spec wants "InfiniBand or RoCE experience". A GPU cluster proposal lands with a fabric decision nobody on the team feels qualified to make. Someone says "the subnet manager restarted" in an incident call and you nod along.

If your instincts were trained on VLANs, OSPF, and QoS policy maps, InfiniBand looks like another discipline. It isn't. It moves bytes between machines without dropping them, same as your job description. What makes it worth studying is that it took the opposite position on almost every default Ethernet chose, and Ethernet has spent the last decade quietly adopting those positions one by one.

I run both in my lab: a physical InfiniBand switch with ConnectX-6 adapters on one side, and a leaf-spine Ethernet fabric on the other. So instead of explaining InfiniBand in the abstract, I'll show you every concept live, straight from the gear. My fabric is QDR-era and small, but the concepts you'll see are byte-for-byte the same ones running at 800G in a Blackwell pod.

The market picture, honestly

Worth settling up front, because it decides what kind of knowledge this is.

When Dell'Oro Group started tracking AI back-end networks in late 2023, InfiniBand carried over 80 percent of them. It earned that share: it was the only fabric with years of production history behind large synchronized training. By the first quarter of 2026 the picture had inverted. Ethernet switch sales into AI back-ends more than doubled, reaching roughly two thirds of data center switch sales in AI clusters. InfiniBand sales more than tripled in the same quarter, but Dell'Oro's own qualifier is the part to remember: much of that growth was existing estates refreshing to 800G alongside Blackwell Ultra, not new fabrics being won. Meanwhile 1.6T Ethernet is sampling with a ramp expected in the second half of 2026.

Arista's Jayshree Ullal calls Ethernet the eventual winner, and she is probably right. But listen to how the Ethernet camp makes that argument: they describe InfiniBand as the reference point, the fabric whose predictability and GPU offload they had to match. Even NVIDIA, which owns InfiniBand outright, ships a serious AI Ethernet line beside it. Nobody is arguing InfiniBand's ideas were wrong. They're arguing Ethernet can now deliver those ideas on hardware anyone can buy.

Which means the ideas are the durable part. Here they are, running.

Your vocabulary, translated

InfiniBand is a complete stack: its own physical, link, network, and transport layers, plus a software interface. Most of its nouns have an Ethernet cousin:

Here's my fabric introducing itself. One command walks the topology and prints every device with its LID:

Read it like a CDP/LLDP table: one Infiniscale-IV switch at LID 7, my storage node's HCA at LID 1, the host's ConnectX-6 at LID 2, every link negotiated to 4X QDR. The QoS plumbing is visible too. Ask the switch how service levels map to virtual lanes and it hands you the table, along with the weighted arbitration between lanes:

If you've ever mapped CoS values into egress queues and set WRR weights, you have already configured this. The difference is who did the configuring, which brings us to the first big idea.

Idea 1: the sender asks before it transmits

Ethernet's link layer is optimistic. Frames go out, buffers absorb bursts, and when a buffer is about to lose the fight you react: PFC pauses the upstream port, ECN marks packets, DCQCN walks the sender's rate down. Engineered correctly, this works. But it is a reaction chain, and every device in the path has to hold up its end of it. One box with a mismatched PFC priority or a missing ECN threshold and you get the failure modes RoCE operators tell war stories about: pause storms, head-of-line blocking, tail latencies that spike at 2 a.m. and vanish by the time you're logged in.

InfiniBand made the opposite bet. Flow control is credit-based, per virtual lane, at the link layer. A receiver continuously advertises how much buffer it has; a sender is not allowed to transmit a byte it doesn't hold credits for. There is no drop to recover from because the situation that causes a drop can't be constructed.

My switch has been forwarding RDMA traffic, multicast joins, and management sweeps since the fabric came up. Here is its error ledger for the busiest port:

Three zeros matter. PortXmitDiscards: nothing was ever thrown away. ExcessiveBufferOverrunErrors: no receiver was ever overrun. And PortXmitWait at zero says the port never spent a single tick holding data it couldn't put on the wire. That last counter is the one I'd point an Ethernet engineer at, because it's the direct read on the mechanism itself: you can watch back-pressure happening (or not happening) per port, instead of inferring it from drop deltas.

On your fabric, lossless is a state you engineer and can silently break with one config push. Here it's what the link does when you plug it in.

Idea 2: the fabric has one operator, and it's a daemon

Everything you run is distributed by instinct: switches learn addresses by watching traffic, protocols elect and converge, each box computes its own forwarding state. InfiniBand centralized all of it. A subnet manager discovers the topology, hands out every LID, computes every forwarding table, programs those SL-to-VL maps you saw above, and pushes the result into each device's management agent. Switches don't learn anything. They're told.

In my lab the SM is opensm, running on the storage node with priority 15 so it always wins mastership: you saw SMINFO_MASTER in the first screenshot. Watching its log during a fabric event is the fastest way to internalize the model. When my lab booted today, one process noticed the switch port change state, swept the subnet, rebuilt the world, and declared victory:

That SUBNET UP line is the whole philosophy in one log entry: the fabric is up because the manager says it is. Traps flow to it, multicast groups are created through it, and if it dies a standby with a copy of the state takes over. If this sounds like an SDN controller, trust the instinct. InfiniBand was running a centralized control plane in production years before anyone sold you one, and it carries the exact trade you already know from controller fabrics: beautiful determinism, plus a control plane you now have to operate, monitor, and fail over like any other critical service.

Idea 3: applications post work; adapters move memory

On Ethernet, an application writes to a socket and the kernel takes it from there: copy, segment, encapsulate, interrupt, repeat on the far side. Every byte pays a CPU toll both ways.

InfiniBand's transport was designed for hardware from day one. An application creates a queue pair (a send queue and a receive queue), posts work requests to it, and the adapter executes them, reporting completions to a completion queue. The API is called verbs, and I've walked through it on this blog before. The payoff is RDMA: one host reads or writes another host's memory with no remote CPU involvement at all.

You can see the model right in a benchmark's handshake. Look at what the two ends exchange before a single data packet flows:

Queue pair numbers, an RKey, and a raw virtual address. One detail worth pausing on: the local end reports LID 3, which appeared nowhere in the topology walk earlier. That's because my compute VM talks through an SR-IOV virtual function, and the subnet manager gives each virtual port its own LID behind the physical adapter at LID 2. My compute node is literally being handed a pointer into the storage node's RAM, plus the key that authorizes writing through it. The result on my QDR link: 30.97 Gb/s sustained, which is 97 percent of QDR's effective data rate after encoding, and 0.89 microsecond typical write latency with a 99th percentile of 0.95. Look at how close typical is to p99. That gap, not the average, is what "deterministic fabric" means, and it's the number a training job actually feels.

If you take one acronym from this section, make it RoCE: the same verbs, queue pairs, and RDMA semantics, wrapped in UDP/IP so they can ride a routed Ethernet fabric. RoCEv2 is why everything in this section is now your problem too, and why the PFC/ECN engineering from Idea 1 exists at all: RDMA transports assume the loss rate InfiniBand gave them.

Idea 4: the switch joins the computation

This is the one with no production Ethernet equivalent you can order today, and the one that keeps InfiniBand bolted into the largest training fabrics.

Distributed training leans on collective operations, above all allreduce: every GPU contributes its gradients, and every GPU needs the sum back. Do that naively and the same buffers cross the fabric over and over while GPUs burn cycles on arithmetic that isn't training. NVIDIA's SHARP moves the addition into the switch ASICs: partial results are summed as they climb the aggregation tree, so traffic shrinks at every tier and one reduced result fans back down.

My Infiniscale-IV is far too old to know this trick, so no live demo here, and I'd rather say that plainly than fake one. But sit with the design for a second, because it breaks a rule we've all internalized: the switch is supposed to forward and stay out of the payload. For exactly one workload shape, breaking that rule bought real percentage points of GPU utilization, and in-network computing is now a headline feature of every new InfiniBand generation.

Ethernet's answer: standardize the same guarantees

None of this went unnoticed. In mid-2023, AMD, Arista, Broadcom, Cisco, Eviden, HPE, Intel, Meta, and Microsoft founded the Ultra Ethernet Consortium, and on June 11, 2025 it shipped UEC Specification 1.0: more than five hundred pages defining a modernized RDMA transport, congestion control designed for AI and HPC traffic, and profiles spanning NICs, switches, optics, and cables, all deliberately multi-vendor. NVIDIA itself joined the consortium along the way.

Put UEC's feature list next to the four ideas above and the intent is unmistakable: keep the guarantees, drop the single-vendor supply chain. Add the operational gravity (your team, your tooling, your spares closet, and your muscle memory are all Ethernet) and the Dell'Oro curve explains itself.

Why the top end still buys it

InfiniBand's answer has been to keep raising the ceiling. The current Quantum-X800 platform puts 144 ports of 800 Gb/s in a switch, with SHARP v4 in-network computing, adaptive routing, and telemetry-driven congestion control, paired with ConnectX-8 SuperNICs at 800 Gb/s per adapter. The next move on both fabrics is co-packaged optics: NVIDIA's Quantum-X and Spectrum-X Photonics lines move the optical engines onto the switch package itself, cutting the power and latency cost of pluggables as clusters outgrow copper.

Notice, again, that one company is doing both. That's the clearest signal of where this ends: two fabrics, one set of ideas.

The four questions that actually decide it

When the RFP lands, the honest decision tree is short.

How synchronized is the workload? A large training run advances at the pace of its slowest transfer; that's where deterministic latency and in-network reduction pay their premium. Inference farms and mixed clusters are far more forgiving.

Is this one stack or many? Single-tenant, single-vendor, built once: a purpose-built fabric is viable. Multi-tenant or multi-vendor: the interoperable one wins by default.

Who answers the page at night? A fabric your on-call can't reason about has worse availability than its datasheet, every time. This is Ethernet's deepest advantage and it has nothing to do with silicon.

How many networks do you want to run? Storage, management, and user traffic will be Ethernet regardless. One fabric everywhere is an operational simplification someone should have to argue against.

Only the first question is about performance. That ratio is the honest summary of where the argument stands in 2026.

What transfers even if you never touch it

You may never hold an InfiniBand cable. The ideas still follow you around. Credit-based flow control is the cleanest mental model for understanding what your PFC/ECN configuration is trying to approximate, and why it hurts when it's wrong. The subnet manager is the original case study in what centralizing a control plane buys and costs. Queue pairs and verbs are already on your fabric the moment someone deploys RoCEv2. And offload keeps climbing the stack: checksums first, then transport, now arithmetic inside the switch.

Ethernet is winning the market. InfiniBand set the bar it had to clear. Learn the four ideas on either fabric and you'll be the engineer in the room who understands both sides of the RFP.

Sources

• Dell'Oro Group: Ethernet is Winning the War Against InfiniBand in AI Back-End Networks (the late-2023 share figure)
• Dell'Oro Group: Ethernet Extends Lead in AI Scale-Out Networks Despite Strong InfiniBand Rebound (all Q1 2026 figures and the refresh qualifier)
• Fierce Network: Arista's Ullal: Ethernet is the eventual winner for AI networking
• Ultra Ethernet Consortium: UEC Launches Specification 1.0 and the specification PDF
• NVIDIA: Quantum-X800 InfiniBand platform (port counts, SHARP v4, ConnectX-8 SuperNIC)
• NVIDIA: Introduction to InfiniBand whitepaper and InfiniBand QoS documentation (credit-based flow control, virtual lanes, subnet management, queue pairs)
• Live outputs in this post: captured on my lab fabric (Infiniscale-IV switch, ConnectX-6 HCAs, opensm) with infiniband-diags and perftest; commands shown verbatim in each screenshot