On this page
A walkthrough of InfiniBand's multi-tenancy machinery on real hardware: one physical ConnectX-6 port carved into four virtual HCAs with SR-IOV, five virtual machines sorted into two isolated partitions, and the isolation demonstrated not with a vendor diagram but with packet captures, hardware state, and commands you can run yourself. Every output here is real. If you think in VLANs, this is what the equivalent machinery looks like when it lives in the adapter instead of the switch.
Two models of network isolation
Ethernet and InfiniBand both let you slice one physical network into isolated groups. They put the enforcement in completely different places, and that single choice shapes everything downstream.
On Ethernet, isolation is a switch feature. You tag frames with a VLAN ID, switches filter on it, and the endpoint is untrusted by design: if a host injects a frame with the wrong tag, you reach for private VLANs, ACLs, or 802.1X to patch the trust gap. The host's NIC will emit whatever the operating system asks it to.
On InfiniBand, isolation is a fabric-wide contract enforced by the adapter itself. A central authority, the Subnet Manager, writes a table of Partition Keys, called P_Keys, into every port on the fabric. Every packet carries a P_Key in its transport header. Hardware checks it on receive and silently drops mismatches. The endpoint cannot opt out, because the endpoint never owned the table in the first place.
There is a second difference with no VLAN equivalent. A P_Key is sixteen bits, but only fifteen of them are the partition number. The top bit is a membership type: full or limited. Two full members can communicate. A full member and a limited member can communicate. Two limited members in the same partition, carrying the same partition number, cannot reach each other. That one bit produces hub-and-spoke topologies, a storage target reachable by every client while the clients stay dark to one another, as a property of the fabric rather than a stack of firewall rules.
Add SR-IOV and it gets more interesting. One physical adapter presents itself as several independent PCI devices. Each virtual function goes to a different virtual machine, each appears on the fabric as its own endpoint with its own GUID and LID, and each lands in whatever partition the Subnet Manager assigns. The tenant gets genuine hardware RDMA, kernel bypass and all, and still cannot cross the partition boundary, because the boundary is enforced below anything the tenant can touch.
That is the claim. The rest of this article is the work of watching it hold.
