On this page
A traceroute between two hosts in different datacenters of my EVPN-VXLAN lab comes back as six lines. Each one is a precise statement about what a multisite fabric does to a packet: where the packet is handed to an anycast gateway, where it is wrapped in VXLAN and where it is unwrapped, which routers along the way are numbered and which are not, and where it crosses from one datacenter into the other. This post reads all six, decoded on the boxes themselves rather than inferred from the topology.
The lab is the ecloud containerlab replica: two Cumulus Linux EVPN-VXLAN datacenters, a routed VRF-lite backbone stitching them, a Palo Alto pair on the north edge, k3s clusters, and DNS/NTP service nodes. Here is the whole thing.
The trace runs left to right: from a DC1 k8s host, up through its leaf, across the two aggregation switches in the middle, down into DC2, to a k8s host on the right.
The six lines, decoded
Source is k8s-master-1 in DC1 (10.167.10.11), destination is dc2-k8s-master-1 in DC2 (10.168.10.11). This was a three-probe run (traceroute -n -w1 -q3); I have collapsed each hop below to the distinct address or addresses it returned.
root@k8s-master-1:~# traceroute -n -w1 -q3 10.168.10.11
1 10.167.10.2 0.248 ms 10.167.10.3 0.209 ms
2 192.0.0.8 0.673 ms
3 10.201.20.1 0.492 ms 10.201.20.5 0.378 ms
4 192.0.0.8 0.482 ms
5 10.168.10.3 0.791 ms
6 10.168.10.11 0.624 ms
Hop 1, the two leaves. The host's default gateway is the anycast 10.167.10.1, but the reply comes from 10.167.10.2 or .3: those are the real SVI addresses of the two k8s leaves (leaf-k8s-master-1 and -2), and the bond hashes each flow to one of them. The leaf looks up the destination /32, finds it via both DC1 border loopbacks, and VXLAN-encapsulates it into the tenant's L3VNI. The spines that carry that encapsulated packet never appear in the trace at all, because they only touch the outer header, never the inner TTL.
Hop 2, the phantom address. The border decapsulates and forwards toward the backbone over an unnumbered interface, one with no IPv4 address of its own. When the TTL expires there and the kernel has to source an ICMP "time exceeded", it has no interface address to use, so it falls back to 192.0.0.8, the dummy address reserved for exactly this in RFC 7600. It is not a device. It is the fabric telling you "an unnumbered router lives here". You see it again at hop 4 for the same reason, on the DC2 border.
Hop 3, the backbone. This is the VRF-lite DCI: the aggregation switch, forwarding between sites in the tenant's own routing table. The reply address 10.201.20.1 (or .5 from the other aggregation switch) is a real IPv4 this time, a /30 on the switch's gobgp-facing link, but the packet actually left on a different interface toward DC2. That gap is the whole subject of the next section.
Hop 5, the far leaf. Same shape as hop 1, mirrored: a DC2 k8s leaf, 10.168.10.3 (this run hashed to dc2-k8s-leaf-2; the destination is dual-homed to both DC2 k8s leaves on one EVPN-MH segment), decapsulates and hands the frame to the destination host. Hop 6 is the host.
Five routing hops. The echo replies come back with a TTL of 59, and 64 minus 59 is 5. The math agrees with the trace.
Why two hops are a non-address, and one is not
Those two 192.0.0.8 lines are not a quirk of my lab. They are the signature of how the whole fabric peers, and it is worth pulling apart, because it changes what a traceroute can and cannot tell you.
Every BGP session in this fabric is unnumbered. The routers do not have an IP address on each transit link that they peer across; they peer over the interface itself, exchange IPv6 link-local addresses, and use the extended-next-hop capability to carry IPv4 routes over that link-local session. You can see it on a leaf: the config says neighbor swp1 interface remote-as external, and the session's own endpoints are fe80:: addresses.
leaf-border-1# show bgp vrf tenant-k8s summary
Neighbor V AS ... State/PfxRcd
br-agg-sw-1(swp3.100) 4 65400 ... 9
br-agg-sw-2(swp4.100) 4 65400 ... 9
leaf-border-1:~$ ip -br addr show swp3.100
swp3.100@swp3 UP fe80::a8c1:abff:fec0:8738/64No IPv4 on the link at all. So when the TTL expires on that border and it has to send an ICMP "time exceeded", the kernel goes looking for an IPv4 source address on the outgoing interface, finds none, and falls back to 192.0.0.8, the dummy address RFC 7600 reserves for exactly this. The hop is real, the router is real, the address is a placeholder that means "an unnumbered router lives here". Both borders do it, which is why you see it at hop 2 and hop 4.
Hop 3 is the interesting counter-example. The aggregation switch is unnumbered toward the datacenters too, but it carries an IPv4 /30 on its gobgp-facing link, so it has a real address to answer from. That is why hop 3 shows 10.201.20.1 and not another 192.0.0.8. But here is the catch that trips people up: 10.201.20.1 is on swp7, the link toward the gobgp anycast controller, while the packet actually left the switch on swp5.100 toward DC2. Verified on the box: the forwarding entry points at swp5.100 and swp6.100, both unnumbered. So even the one numbered hop tells you the device but lies about the link. A router picks its ICMP source address by its own rules, not by which interface the packet used.
That is the whole tradeoff, sitting in a single traceroute.
Unnumbered is lovely to run: no per-link IPv4 planning, no /30 or /31 sprawl to allocate and document, no link addresses to leak into BGP or filter, and the leaf config is identical everywhere so you can stamp it out. The cost lands entirely on troubleshooting. Every transit hop reports the same 192.0.0.8, so a traceroute alone cannot tell you which router timed out. You get the hop count and nothing else; to turn a hop into a device you go to the boxes and read LLDP neighbors, the routing table, and the TTL math.
Numbered links flip it. Every hop answers with a real, greppable, reverse-DNS-able address you can map straight to a device and often to a specific link, so a traceroute by itself localizes a fault. The price is the address management unnumbered spares you: planning, documentation, filtering, and the security surface of live IPs on every transit link. And as hop 3 just showed, numbered does not fully rescue you either, because the reply address is the router's chosen source, not a guarantee of the egress interface.
Neither is wrong. Unnumbered optimizes for building and operating at scale; numbered optimizes for reading a path at a glance. This fabric chose unnumbered, and the price of that choice is precisely the two 192.0.0.8 lines the trace opens with. Once you know that, the non-address stops being a mystery and becomes a label: unnumbered router here, go look.
The same path, twice, in two slices
Here is the part I liked. I ran the identical trace for the other tenant, the service hosts instead of the k8s hosts, and got the same shape through the same shared backbone, the DC1 border pair, the aggregation switches, and the DC2 border pair, with each tenant on its own pair of leaves and every identifier swapped for its own:
tenant-k8s (traced) tenant-svc
ingress / egress SVI vlan110 / vlan210 vlan130 / vlan230
L2 VNI 10110 / 10210 10130 / 10230
L3VNI (DC1 / DC2) 50001 / 50101 50002 / 50102
DCI VLAN on the aggs 100 200
hop-3 reply address 10.201.20.1 10.201.1.0Two tenants sharing one backbone of borders and aggregation switches, each on its own pair of leaves at the ends, kept completely apart by VNI and VRF. Only the three middle hops are literally the same boxes; the ingress and egress leaves are separate per-tenant devices (leaf-service versus leaf-k8s-master). Even the phantom 192.0.0.8 shows up in both, because both borders are unnumbered. The only other visible difference is which dot1q subinterface the DCI rides (VLAN 100 versus 200) and which of the aggregation switch's addresses answers hop 3: the k8s trace gets the gobgp-facing 10.201.20.1, the service trace the firewall-facing 10.201.1.0.
One more thing: the firewalls are not here
Worth saying plainly, because the reply addresses can mislead you: this east-west path never touches the Palo Altos. Every next-hop at every decision point, checked in each device's forwarding table, is a leaf, border, or aggregation interface inside the tenant VRF. The firewalls sit on the north-south path only. The aggregation switch that answers hop 3 is the same box the firewalls attach to, and in the service tenant's trace hop 3 answers from the firewall-facing link itself, which makes the red herring even sharper. Either way it is source-address selection, not the packet's path.
Six lines of traceroute, and underneath them: anycast gateways, VXLAN encap and decap you cannot see, unnumbered routers announcing themselves with a reserved address, a VRF-lite backbone, and two tenants riding the same steel without ever meeting. The full per-hop audit, with every forwarding-table lookup and identity check, is in the lab repo.
Sources
- RFC 7600: assigns the IPv4 dummy address 192.0.0.8, which the Linux kernel then uses as the ICMP source when an interface has no usable address of its own
- RFC 8365: A Network Virtualization Overlay Solution Using Ethernet VPN
- RFC 9014: Interconnect Solution for EVPN Overlay Networks
- ecloud-containerlab: the two end-to-end trace audits (tenant-k8s and tenant-svc, every hop decoded with live forwarding-table output)
