On this page
Ethernet guesses. InfiniBand plans.
An Ethernet switch learns the network by watching it: flood, learn MACs, let spanning tree fence off the loops. InfiniBand refuses to guess. Nothing forwards a single packet until one process has walked the entire fabric, assigned every port an address, computed every route, and pushed the results into every device. That process is the subnet manager, and on most Linux fabrics it's a daemon called opensm.
This post walks through what it actually does, live on my lab fabric: two ConnectX-6 ports connected back to back, with four SR-IOV virtual functions riding one of them. Five compute identities, six IB ports, one SM.
The sweep
Everything opensm knows, it learns from a sweep: walk out from its own port, hop by hop, interrogating every device it finds over the management datagram interface. From that map it assigns each end port a LID, a Local Identifier, which is the fabric's routable address. IB packets are switched on LIDs, not MACs, and nobody picks their own: the SM hands them out like a DHCP server that also owns the routing tables.
How does it interrogate anything before addresses exist? Bootstrapping is the elegant part. The SM speaks Subnet Management Packets over a reserved queue pair, QP0, using directed routing: instead of a destination address, the packet carries turn-by-turn directions ("exit port 1, then port 3"), which works on a fabric that has no addresses yet. Once LIDs are assigned, everything else, including the saquery below, talks to the SM's subnet administration database over QP1 with ordinary addressed packets. Two reserved QPs, two phases: QP0 builds the world, QP1 answers questions about it.
Sweeps come in two weights. A light sweep is a quick poll for port state changes; a heavy sweep is the full rebuild: rediscover, re-address, recompute routes, reprogram tables. Traps from devices (a port coming up, a link dropping) trigger sweeps on demand, which is why a healthy fabric's log is nearly silent and a flapping cable makes the SM narrate its life story. On a switched fabric the "compute routes" step earns its name: the SM runs a routing engine (min-hop by default, up-down and others for Clos topologies) and programs a linear forwarding table into every switch ASIC, LID by LID. My two-CA fabric makes that step trivial, but it's the same machinery that routes a 10,000-port supercomputer.
You can see the result of the sweep by asking the subnet administration database, and you can watch the sweep itself in the log:
MT4123 is the physical ConnectX-6; the four MT4124 entries are its SR-IOV virtual functions. Which raises a question the stock tooling can't answer.
The SR-IOV twist: virt_enabled
The opensm that ships in Ubuntu's repos walks physical topology only. It found my two physical ports, gave them LIDs, declared the subnet up, and the four VFs simply did not exist: no LID, no route, ports stuck down. Virtual functions appear as vPorts behind a physical port, and discovering them requires virtualization support that inbox opensm (3.3.x) doesn't have. The fix is Mellanox's opensm build with one config line, virt_enabled 2, after which the sweep enumerates vPorts and every VF gets a first-class LID. That single line is the difference between SR-IOV working and four silent ports.
Partitions: the fabric's access control
The SM's other job on this fabric is deciding who may talk to whom. Partitions are keyed by a pkey, and the file reads almost like prose:
The powerful bit is membership type. A full member talks to anyone in the partition. A limited member talks only to full members. My compute nodes are limited members of the same partition, so each reaches the storage node but never each other, enforced in the HCA silicon, not in any firewall:
Note what's absent: the member nodes have no partition config at all. The SM computes membership from the config file and programs everyone's pkey tables during the sweep. Policy lives in exactly one place.
Who gets to be SM
Any full port can theoretically run an SM, and priorities (0 to 15) decide elections when several coexist. But there's a catch worth knowing in SR-IOV setups:
Elections are how multiple SMs coexist. Every SM port carries a priority (0 to 15) and the fabric converges on one MASTER: highest priority wins, lowest port GUID breaks ties, and the losers drop to STANDBY, polling the master and ready to take over if it dies. You can watch the state machine directly: sminfo reports state 3 for MASTER, and a standby SM shows state 2. A handover is graceful (the new master re-sweeps and the fabric barely notices), which is exactly what you want from redundancy. My priority 15 with no standby is the honest homelab version: deliberate, and documented, single point of thinking.
Binding the SM's management interface is a privileged operation on the physical function, and a VF only gets it if the PF explicitly marks it trusted, which the inbox host driver on my system refuses to do. So the storage node, holding the only full physical function on the compute side of the fabric, is the SM by physics as much as by choice. It runs at priority 15 with no standby; on a two-CA lab fabric that's a fine trade, but on anything real you'd run a second SM on another physical port and let the election handle failover.
Field notes
Three things that cost me time so they don't cost you any. First: opensm belongs to native InfiniBand only; RoCE fabrics have no SM, and a forgotten opensm pointed at a dead IB port once wrote itself a 51 GB log in my lab before anyone noticed. Second: the log is the tool. A healthy fabric logs a sweep and goes quiet; a flapping port makes the SM sweep loudly and often, so tail -f /var/log/opensm.log during any cabling work. Third: packaged unit names differ between the inbox and MLNX builds, so after any install, confirm with sminfo that exactly one MASTER exists and it's the one you meant.
The takeaway: InfiniBand's determinism isn't magic, it's employment. Somewhere on your fabric a single-threaded daemon is doing all the thinking, and once you can read its sweeps, partitions, and elections, the whole fabric becomes legible.
