InfiniBand

Your AI cluster is two networks wearing one badge. One is judged by whether users notice it. The other is judged by whether GPUs wait on it. Frontend vs backend: the traffic, the rules, and how to design the GPU fabric so compute never waits.

Credit-based flow control, a daemon that runs the whole fabric, adapters that hand out pointers into remote RAM, and switches that do math. Every InfiniBand idea an Ethernet engineer should know, demonstrated live on a real InfiniBand fabric.

A firmware setting decides whether each ConnectX port wakes up speaking Ethernet or native InfiniBand. My cards run both at once, and the same verbs binaries push bytes over either. One mstconfig line to change your NIC's religion.

RDMA bypasses the kernel, so tcpdump captures nothing while the port counters swallow gigabytes. The NIC itself can mirror the port though, and from the host that owns the PF you can watch every byte your SR-IOV guests exchange.

IPoIB gives your IB fabric an IP address so ordinary sockets code just works. On a 10G link it even matches verbs on bandwidth. The CPU counter and the latency numbers tell the real story of what that convenience costs.

InfiniBand forwards nothing until one process has walked the fabric, addressed every port, and programmed every route. A deep look at opensm on a live SR-IOV fabric: sweeps, LIDs, partitions, virt_enabled, and why a VF can never be the boss.

Two ConnectX-6 ports that can do HDR. A DAC with 100G printed on the label. An InfiniBand link that trained at 10 gigabits. Your fabric's speed is written in a tiny EEPROM inside the cable, and it does not care what the label says.

A short nugget on the RDMA verbs API: the four objects every application allocates, the path a work request travels, why one-sided operations leave the remote CPU asleep, and why the same API drives RoCEv2 over Ethernet.

A hands-on walkthrough of InfiniBand multi-tenancy on real hardware: one ConnectX-6 port split into four SR-IOV virtual functions, five VMs sorted into two P_Key partitions, and the isolation proven on the wire with packet captures and hardware state.