On this page
Two ways to slice a NIC
My lab runs SR-IOV everywhere: the NIC's firmware carves a physical ConnectX-6 into virtual functions, each one a real PCI device with its own DMA queues, handed to a guest that talks to silicon directly. Readers keep asking how macvlan compares, and the honest answer starts with what macvlan is: the same idea implemented one layer up. The kernel stacks child interfaces on a parent netdev and switches frames between them by MAC address, in software.
One boundary before comparing: macvlan multiplexes by MAC address, which makes it an Ethernet-only construct. It cannot exist on my native InfiniBand ports, where there are no MACs to demux; the closest IB analog is the IPoIB pkey child interface from earlier in this series. So the demo below runs on the node's Ethernet management NIC, sitting right next to the InfiniBand VF it's being compared with.
The thirty-second setup
The first difference is ceremony. Getting my SR-IOV VFs working took firmware settings, GUID assignment with a quirky ordering bug, a subnet manager that understands virtualization, and reboots. Here is the macvlan equivalent, live on the same lab node, including the isolation modes:
That mode switch deserves a pause. Readers of the opensm post will recognize the pattern: bridge mode is full membership (children talk to each other), private mode is limited membership (children reach the uplink but never each other). The same isolation idea my IB fabric enforces in HCA silicon with pkeys, macvlan enforces in a kernel function. One is a hardware guarantee, the other is a very fast if-statement.
The numbers, honestly
57.8 Gb/s between two children looks spectacular until you notice two things. First, the price: the sender burned an entire CPU core and the receiver most of another, the same kernel tax the IPoIB post measured, just at higher volume. Second, the asterisk: child-to-child traffic in bridge mode never leaves the box, so this is a memory benchmark wearing a network costume. Traffic to anything external goes through the parent NIC at wire speed minus that same CPU tax.
The absences matter more than the numbers. A macvlan child has no PCI device behind it and no RDMA capability, ever: no verbs, no kernel bypass, no NVMe-oF, while the SR-IOV VF on the same node is a first-class RDMA device. And isolation is namespace-deep, not IOMMU-deep: the host kernel sits in every macvlan packet's path, while a VF's traffic DMAs past the host entirely (which is why capturing it needed the ibdump trick).
Choosing
SR-IOV when the workload earns hardware: RDMA, VMs with real isolation requirements, line rate with idle CPUs, or anything a tenant shouldn't share a kernel with. Its costs are VF counts provisioned in firmware config (my card currently allows exactly four, and raising that is an mstconfig change plus a firmware reset, not an ip command), pinned devices that fight live migration, and the setup ceremony. macvlan when you want many cheap L2-present endpoints and can pay CPU for convenience: containers that need real MACs on the LAN, lab services, things that move between hosts. Its costs are the kernel data path and one famous quirk: a child cannot talk to its own parent's IP address, a design decision that has confused every container operator at least once (the fix is a macvlan shim interface on the host).
They also compose beautifully, on Ethernet: pass an SR-IOV VF into a VM, then run macvlan inside the VM on the VF's netdev for that VM's containers. Hardware isolation between tenants, kernel convenience within one. (Not an option on InfiniBand VFs, whose IPoIB netdevs can't parent a macvlan; there you'd use pkey children instead.)
The nugget: SR-IOV and macvlan answer the same question, one NIC and many consumers, at different layers. Silicon children cost ceremony and give you hardware truth. Kernel children cost CPU and give you convenience. Know which one your workload is actually asking for.
