On this page
The compatibility layer
Verbs is how InfiniBand wants to be spoken to. But your NFS mount, your Prometheus scrape, and 99% of everything else speak sockets, and rewriting the world against verbs isn't happening. IPoIB is the escape hatch: a kernel driver that presents an InfiniBand port as an ordinary network interface, encapsulating IP packets in IB datagrams so that anything speaking TCP or UDP just works over the fabric.
What it looks like
On my SR-IOV lab fabric, each node carries a child interface per partition, created off the physical IB device with the partition's pkey baked in:
Two things give away that this isn't Ethernet. The 20-byte link address is an IPoIB hardware address wrapping the port GID, and there's no MAC learning underneath: neighbor resolution maps IPs to IB address vectors, and the partition key decides reachability before IP ever gets a vote. Same subnet, same interface type, and compute-1 to compute-2 is 100% loss because the fabric's access control lives below IP (covered in the opensm post).
The honest numbers
Here's where IPoIB surprises in both directions. Same two nodes, same 10G SDR link, measured three ways:
On bandwidth, IPoIB nearly matches verbs: 7.63 vs 7.93 Gb/s, because on a link this slow the wire is the bottleneck, not the kernel. If you only benchmarked throughput you'd conclude the two paths are equivalent. They are not, and the other two columns say why. The verbs transfer used effectively zero CPU; IPoIB burned 15% of a core on the receiver just to shovel packets through TCP. And latency is a different sport entirely: 0.82 microseconds for a verbs message against roughly 8 microseconds round-trip for a warm IPoIB ping, an order-of-magnitude gap that no faster cable will close, because it's all software.
Scale the picture up and the bandwidth tie breaks too: at 100G rates, IPoIB's per-packet kernel costs become the bottleneck long before the wire does, while verbs keeps saturating the link with idle CPUs.
Where it belongs
Use IPoIB for what it is: management traffic, SSH, monitoring, the NFS mount that doesn't justify NVMe-oF, and bootstrapping things that need an IP address before RDMA connections can be established (perftest itself exchanges connection info over it). Let anything that actually cares about the fabric's performance speak verbs.
The nugget: IPoIB makes InfiniBand compatible, not fast. On a slow link it can even keep up on bandwidth, but latency and CPU always tell the truth about which path your bytes took.
