RDMA

RDMA bypasses the kernel, so tcpdump captures nothing while the port counters swallow gigabytes. The NIC itself can mirror the port though, and from the host that owns the PF you can watch every byte your SR-IOV guests exchange.

IPoIB gives your IB fabric an IP address so ordinary sockets code just works. On a 10G link it even matches verbs on bandwidth. The CPU counter and the latency numbers tell the real story of what that convenience costs.

InfiniBand forwards nothing until one process has walked the fabric, addressed every port, and programmed every route. A deep look at opensm on a live SR-IOV fabric: sweeps, LIDs, partitions, virt_enabled, and why a VF can never be the boss.

A short nugget on the RDMA verbs API: the four objects every application allocates, the path a work request travels, why one-sided operations leave the remote CPU asleep, and why the same API drives RoCEv2 over Ethernet.

A hands-on walkthrough of InfiniBand multi-tenancy on real hardware: one ConnectX-6 port split into four SR-IOV virtual functions, five VMs sorted into two P_Key partitions, and the isolation proven on the wire with packet captures and hardware state.