On this page
Sockets ask. Verbs tell.
With sockets, every send() is a negotiation with the kernel. You make a syscall, the kernel copies your buffer and does the protocol work, and the far side takes an interrupt. At 1 Gb/s nobody notices. At 100 Gb/s the kernel becomes the bottleneck, and your CPU ends up doing the network card's job.
Verbs is the answer InfiniBand shipped with from day one. It isn't a protocol. It's the API contract between an application and an RDMA-capable NIC. The kernel gets involved exactly once, at setup, to create the objects and pin the memory. After that, the application and the NIC talk to each other directly. No syscalls, no copies, and if you poll, no interrupts either.
Four objects rule the data path
Every verbs application allocates the same small set:
- Protection Domain (PD): the container. Objects in different PDs can't touch each other.
- Memory Region (MR): a buffer you register with the NIC. Registration pins the pages and hands you two keys, an
lkeyfor local use and anrkeyyou can give to a remote peer. - Queue Pair (QP): a send queue and a receive queue. This is "the connection" in RDMA, roughly what a socket is to TCP.
- Completion Queue (CQ): where the NIC reports finished work.
Life of a work request
You post a Work Request (WR) describing a buffer onto the QP, then ring a doorbell, which is a single MMIO write to the NIC. That's the entire cost on the CPU. The NIC pulls the payload straight out of your registered memory with DMA, moves it across the fabric, and the remote NIC writes it straight into memory on the other side. When your WR finishes, the NIC drops a CQE into your completion queue. The kernel never saw the byte.
Two-sided vs one-sided
SEND/RECV is two-sided and looks like classic messaging. The receiver has to pre-post a RECV WR to name the landing buffer, and both CPUs see a completion.
RDMA WRITE and RDMA READ are one-sided, and this is where verbs earns its reputation. If you hold a remote MR's rkey, your NIC can read or write that remote memory directly. The remote CPU isn't notified, isn't interrupted, and isn't involved at all. NVMe-oF and GPU collectives like NCCL and GPUDirect are built on exactly this.
Same verbs, Ethernet wire
Verbs doesn't care what's underneath it. RoCEv2 wraps the same InfiniBand transport headers in UDP/4791, so an application written against verbs runs unmodified on an Ethernet fabric. That's not a slide-deck claim. On a pair of ConnectX-6 adapters in my lab, ib_send_bw pushes 97.89 Gb/s over a 100 Gb/s Ethernet link, and the latency test comes back at 0.88 µs typical. Same verbs, same binaries, Ethernet wire. Notice the giveaway in the output below: Transport type : IB next to Link type : Ethernet. That one pairing is RoCEv2 in a nutshell.
If you're curious how one of those adapters gets carved into isolated virtual HCAs, I covered that in the previous post: One Adapter, Many Tenants.
The nugget: verbs is the reason RDMA is fast. You can swap the wire underneath, but the kernel bypass is the whole point.
