EVPN MAC Mobility, Watched Live: RFC 7432 Sequence Numbers Across a Cisco and Juniper Multi-DC Fabric

A MAC address is supposed to live in exactly one place. This week I made one pack up and move between two datacenters: same address, same IP, first behind a Cisco Nexus leaf in DC1, then behind a Juniper vJunos leaf in DC2, across a multisite EVPN interconnect. EVPN has a precise little mechanism for exactly that moment, a sequence number riding on the type-2 route, and I wanted to watch it tick on real gear instead of taking the spec's word for it. So here it is: one deliberately mobile MAC, two vendors, two datacenters, the sequence number climbing 0, 1, 2 across the two vendor CLIs, and the exact BGP packets that carried each move over the DCI.

Tested on: a multivendor DCI lab in containerlab on an AMD EPYC host. DC1 is a Cisco Nexus 9000v VXLAN fabric, DC2 is a Juniper vJunos-switch fabric, and a vJunosEvolved border gateway plus a vJunos-switch core stitch the two sites together. VLAN 10 is stretched across both datacenters by the multisite design, which makes this the perfect mobility playground: the demo needs zero fabric changes. The standard is RFC 7432 (section 7.7 defines the MAC Mobility extended community, section 15 the procedures), with RFC 9014 covering the datacenter interconnect model.

A MAC you can pick up and carry

The mobile endpoint is a macvlan with a locally administered MAC, created on the server attached to DC1's first leaf. The server ports are plain access ports in the stretched VLAN 10, so there is nothing to prepare:

root@s1:~# ip link add mob0 link eth1 address 02:ec:de:00:00:88 type macvlan mode bridge
root@s1:~# ip link set mob0 up
root@s1:~# ip addr add 172.16.10.88/24 dev mob0
root@s1:~# arping -c3 -I mob0 172.16.10.254

A few frames later the Cisco leaf holds it as a local MAC, and here is the number this whole post is about, starting at zero:

dc1-leaf1# show l2route evpn mac evi 10 mac 02ec.de00.0088 detail
Topology    Mac Address    Prod   Flags              Seq No     Next-Hops
----------- -------------- ------ ------------------ ---------- ----------
10          02ec.de00.0088 Local  L,                 0          Eth1/3
            Route Resolution Type: Regular
            Forwarding State: Resolved
            Sent To: BGP

Its BGP advertisement carries route targets and the VXLAN encapsulation, and nothing else. A brand-new MAC has no mobility history, so per RFC 7432 there is no MAC Mobility community at all on the first advertisement:

dc1-leaf1# show bgp l2vpn evpn 02ec.de00.0088
Route Distinguisher: 192.0.2.11:32777    (L2VNI 10010)
BGP routing table entry for [2]:[0]:[0]:[48]:[02ec.de00.0088]:[0]:[0.0.0.0]/216, version 332
  Path type: local, path is valid, is best path, no labeled nexthop
  AS-Path: NONE, path locally originated
      Extcommunity: RT:65500:10010 ENCAP:8

And the far side already knows about it. On the Juniper leaf in the other datacenter, the MAC arrives through the border gateway's re-origination, and Junos states the sequence in plain words:

admin@dc2-leaf1> show evpn database mac-address 02:ec:de:00:00:88 extensive
VN Identifier: 10010, MAC address: 02:ec:de:00:00:88
  Source: 00:00:00:00:00:00:00:00:22:00, Rank: 1, Status: Active
    Remote origin: 192.0.2.24
    Mobility sequence number: 0 (minimum origin address 192.0.2.24)
    IP address: 172.16.10.88

Dataplane check before touching anything: a ping from a second DC1 server, and one from a DC2 server that crosses the entire interconnect:

root@s2:~# ping -c3 172.16.10.88
rtt min/avg/max/mdev = 3.027/3.787/4.526/0.612 ms
root@s3:~# ping -c3 172.16.10.88
rtt min/avg/max/mdev = 6.528/7.195/8.418/0.865 ms

The move: one MAC leaves Cisco, arrives at Juniper

Delete the endpoint in DC1, recreate the identical MAC and IP on the server behind DC2's first leaf, and push a few frames:

root@s1:~# ip link del mob0

root@s3:~# ip link add mob0 link eth1 address 02:ec:de:00:00:88 type macvlan mode bridge
root@s3:~# ip link set mob0 up
root@s3:~# ip addr add 172.16.10.88/24 dev mob0
root@s3:~# arping -c3 -I mob0 172.16.10.254

This is the moment section 15 of RFC 7432 describes: a PE learns locally a MAC that another PE is currently advertising, so it must re-advertise with a sequence number one higher. The Juniper leaf does exactly that, and even starts a mobility history for the address:

admin@dc2-leaf1> show evpn database mac-address 02:ec:de:00:00:88 extensive
VN Identifier: 10010, MAC address: 02:ec:de:00:00:88
  Source: ge-0/0/2.0, Rank: 1, Status: Active
    Local origin: ge-0/0/2.0
    Mobility sequence number: 1 (minimum origin address 192.0.2.21)
    IP address: 172.16.10.88

Back in DC1, the Cisco leaf has yielded: the entry flips from Local to BGP-learned, the sequence follows, and the next hop now points at the border gateway's multisite VIP, the exit toward the far site (note the RS flag, remote site, on the entry):

dc1-leaf1# show l2route evpn mac evi 10 mac 02ec.de00.0088 detail
Topology    Mac Address    Prod   Flags              Seq No     Next-Hops
----------- -------------- ------ ------------------ ---------- ----------
10          02ec.de00.0088 BGP    SplRcv             1          192.0.2.114 (Label: 10010)(Flags: RS)
            Route Resolution Type: Regular
            Forwarding State: Resolved (PeerID: 3)
            Sent To: L2FM

And NX-OS spells out the extended community on the received route, with the multisite Site-ID alongside it:

dc1-leaf1# show bgp l2vpn evpn 02ec.de00.0088
Route Distinguisher: 1:10010
  Path type: external, path is valid, is best path, no labeled nexthop
    192.0.2.114 (metric 0) from 192.0.2.102 (192.0.2.102)
      Extcommunity: RT:10:10 RT:65500:10010 ENCAP:8 Site-ID:0:0
          MAC Mobility Sequence:00:1

The dataplane follows the sequence number. The same ping from the DC1 server now crosses the interconnect to DC2:

root@s2:~# ping -c4 172.16.10.88
rtt min/avg/max/mdev = 5.346/6.224/7.462/0.770 ms

The move, on the wire

The CLI is a rendering. The packet is the truth. I left tcpdump running on the DCI link between DC1's border gateway and the core for the whole exercise (tcpdump -i eth3 -w dci.pcap port 179 in the gateway's network namespace) and decoded it in Wireshark. Three packets tell the entire story.

The initial advertisement leaving DC1, before any move. Route targets, VNI, no mobility community, because there is nothing to say yet:

Frame 3:  192.0.2.14 -> 192.0.2.250   (DC1 gateway to core)
Border Gateway Protocol - UPDATE Message
    EVPN NLRI: MAC Advertisement Route
      Route Type: MAC Advertisement Route (2)
      ESI: 03:00:00:00:00:00:01:00:03:09
      MAC Address: 02:ec:de:00:00:88
      IPv4 address: 172.16.10.88
      MPLS Label 1: 625
      MPLS Label 2: 656
    Route Target: 10:10
    Route Target: 500:500
    Route Target: 65500:500
    Route Target: 65500:10010
    (no MAC Mobility extended community)

The move, arriving back from the core after the MAC surfaced in DC2. This frame actually carries three UPDATE messages (the Info column in the screenshot shows all three): the withdrawal of the old DC1 route rides in the same packet as the new advertisement. Here is the advertisement; the Ethernet segment identifier ending in 22:00 is the interconnect ES the gateways use, the same source the Junos database shows:

Frame 15:  192.0.2.250 -> 192.0.2.14   (core to DC1 gateway)
Border Gateway Protocol - UPDATE Message
    EVPN NLRI: MAC Advertisement Route
      Route Type: MAC Advertisement Route (2)
      ESI: 00:00:00:00:00:00:00:00:22:00
      MAC Address: 02:ec:de:00:00:88
    Route Target: 10:10
    Route Target: 500:500
    MAC Mobility: Movable MAC [Transitive EVPN]
      .... ...0 = Sticky/Static MAC: No
      Sequence number: 1

And the return, leaving DC1 again after I moved the endpoint home:

Frame 29:  192.0.2.14 -> 192.0.2.250   (DC1 gateway to core)
Border Gateway Protocol - UPDATE Message
    Route Target: 10:10
    Route Target: 65500:10010
    MAC Mobility: Movable MAC [Transitive EVPN]
      .... ...0 = Sticky/Static MAC: No
      Sequence number: 2
    EVPN NLRI: MAC Advertisement Route
      Route Type: MAC Advertisement Route (2)
      MAC Address: 02:ec:de:00:00:88

Worth noticing in passing: the MAC Mobility community is transitive, and it survives everything the interconnect does to the route. The border gateways rewrite next hops and route distinguishers when they re-originate between sites, the route crosses two vendors and an eBGP mesh, and the sequence number arrives intact. That is what makes multi-DC mobility work at all.

Move it back, watch it count

The return trip, for the proof that this is a counter and not a coincidence:

root@s3:~# ip link del mob0
root@s1:~# ip link add mob0 link eth1 address 02:ec:de:00:00:88 type macvlan mode bridge
root@s1:~# ip link set mob0 up
root@s1:~# ip addr add 172.16.10.88/24 dev mob0
root@s1:~# arping -c3 -I mob0 172.16.10.254

dc1-leaf1# show l2route evpn mac evi 10 mac 02ec.de00.0088 detail
Topology    Mac Address    Prod   Flags              Seq No     Next-Hops
----------- -------------- ------ ------------------ ---------- ----------
10          02ec.de00.0088 Local  L,                 2          Eth1/3

Sequence 0, 1, 2 across two moves between datacenters, and a ping from the DC2 server finds the address back across the interconnect:

root@s3:~# ping -c4 172.16.10.88
rtt min/avg/max/mdev = 5.120/5.688/6.105/0.368 ms

Both vendors also implement the other half of section 15: duplicate MAC detection. Bounce an address back and forth fast enough (five moves in 180 seconds is a common default) and the fabric freezes it instead of playing ping-pong. Two moves is a demo; fifty is an incident.

The checklist

If you want to run this on your own fabric:

• You need the same L2 domain at both attachment points. In a multisite design a stretched VLAN gives you that for free; inside one site, the VNI must exist on both VTEPs.
• A macvlan with a locally administered MAC is the cheapest mobile host you will ever build. Set the MAC at creation time so the fabric sees exactly one clean learn.
• Judge the move by three things: the losing leaf flipping from local to remote, the sequence number incrementing on the winner, and the MAC Mobility community in the type-2 as received by a third party.
• Capture BGP on the interconnect while you do it. The single UPDATE carrying the incremented sequence is the whole mechanism in one packet, and it proves the community rides through re-origination untouched.
• Keep the move count low; duplicate detection freezes a MAC that bounces too often, by design.

Sources

• RFC 7432: BGP MPLS-Based Ethernet VPN (section 7.7, MAC Mobility extended community; section 15, MAC mobility procedures)
• RFC 9014: Interconnect Solution for Ethernet VPN Overlay Networks
• RFC 8365: A Network Virtualization Overlay Solution Using Ethernet VPN
• Cisco NX-OS: VXLAN EVPN Multi-Site
• Juniper: EVPN MAC Mobility
• The lab: Cisco multisite to Juniper VXLAN stitching (aninchat's interop topology, run from my fork with vJunos 26.2R1.7)