On this page
A 100G cable, a 10G link
Two ConnectX-6 HCAs, a half-meter DAC between them, native InfiniBand, subnet manager running, fabric up. And then the first look at the link speed: 10 gigabits. Not 100, not 200. Ten. A bandwidth test agrees, so it isn't a display quirk:
The ports are innocent
First suspect: the ports. Ask the subnet management agent what this port can actually do, and it tells you plainly. Extended speeds up to 53.125 Gbps per lane are supported and enabled. That's HDR. Times four lanes, this port is happy to run 200 Gb/s. But the active speed is 2.5 Gbps per lane, the slowest InfiniBand has ever gone:
Interrogate the cable
Every QSFP module and DAC carries a small EEPROM describing what it is and what it can carry. mstlink reads it straight off the wire:
There's the confession. The part number proudly says QSFP-100G-CU0.5M, but that 100G is its Ethernet rating. For InfiniBand, the EEPROM advertises exactly one speed: SDR. This is a cheap DAC that was never qualified for faster IB signaling, and it says so, quietly, in a field almost nobody reads.
The negotiation
InfiniBand link training is conservative by design. Before the link comes up, both ports read the cable's EEPROM, and the negotiated rate is the highest speed everyone explicitly claims to support. Not the label, not the port capability, the intersection:
That conservatism is a feature. A DAC that isn't qualified for HDR signal rates would train fine on a good day and throw symbol errors on a warm one. The EEPROM is the cable's binding promise about signal integrity, and the fabric refuses to gamble past it.
Check the cable first
The fix is boring: buy a DAC that is actually IB-rated for the speed you want (EDR and HDR DACs advertise it in the same field). The lesson is the valuable part. Before assuming anything about an InfiniBand link, ask the cable what it thinks it is:
mstlink -d <device> --show_module
This isn't even the worst case. The same lab previously ate a stranger version of this lesson: QSFP-100G-SR4 optics, which are Ethernet-only, won't train InfiniBand at all. The PHY just sits in Polling forever while you re-check every config file on both ends.
The nugget: the label on a cable is marketing. The EEPROM inside it is a contract, and InfiniBand only honors contracts.
