On this page
The problem with the classic Active/Active
Most firewall HA designs lean on layer 2 tricks: a floating IP, gratuitous ARP on failover, and a prayer that every switch in the path honors the MAC move quickly. It works until it doesn't, and when it doesn't, you're debugging ARP caches at 2 AM. There's a cleaner shape available if your firewalls sit next to switches that speak BGP: stop treating the pair as one virtual device wearing a shared IP, and start treating them as two ordinary routers that happen to sync state.
That's how my Palo Alto pair runs. Both firewalls forward traffic all the time, no floating IP on the inside path, no ARP games, and failover is a routing event instead of a layer 2 scramble.
The design in one paragraph
Each firewall bundles two interfaces into an LACP aggregate toward the border pair and runs eBGP (AS 65300) against both leaves (AS 65205). Each advertises a default route it learned from its upstream interface. The border leaves run maximum-paths in the tenant VRF, so the fabric's default route is ECMP across both firewalls: flows hash across the pair and both boxes earn their power draw. HA still exists underneath, but for different jobs: HA1 carries control, HA2 syncs the session table so either box can pick up the other's flows, and HA3 handles the packets that arrive at the firewall that doesn't own the session, because with ECMP on both sides, asymmetry isn't an accident, it's the design. Session ownership goes to the first packet's firewall; setup load is spread by IP modulo.
Here is the configuration on both sides of that peering. On the N9K side, three lines carry the whole design: advertise l2vpn evpn turns the fabric's EVPN routes into ordinary BGP advertisements toward the firewalls, maximum-paths 8 makes the two firewall next-hops coexist, and timers 1 3 is the failover story we'll get to:
What the fabric shares, and what it hears back
The route exchange is where this design connects to everything behind it. The workloads live on a VXLAN/EVPN fabric; their subnets and the Kubernetes LoadBalancer VIP pool travel the fabric as EVPN type-5 routes and arrive at the border leaves. The border leaves re-advertise them to the firewalls as plain eBGP, and you can read the topology right out of the AS path: every prefix says 65100 (the cluster nodes) then the leaf pair it entered through:
The asymmetry of that exchange is the security model in miniature. The firewalls learn everything about the fabric because they need to route inbound traffic to published services and steer return paths. The fabric learns almost nothing back: a default route (the way out) and one directly attached segment. Externally, north of the firewalls, no routing protocol runs at all: outbound traffic is source-NATed to the firewalls' own upstream addresses, and inbound reaches published services through destination NAT on floating addresses, so the upstream network never needs a route to any internal prefix. Routing stops at the firewall on purpose; NAT and published VIPs carry the conversation from there.
Why not Active/Passive?
Fair question, because A/P is genuinely simpler and most shops should run it. Three things pushed this design to A/A. Capacity: with ECMP, both boxes carry production traffic, so you know your failover capacity is real because you're using it every day; an idle standby is a promise nobody has tested. Failure blast radius: losing one of two active paths degrades to half capacity instantly, with no role transition, no preempt decision, no state promotion. And honesty about asymmetry: once the fabric ECMPs toward the pair and the upstream does its own path selection, packets will arrive at the wrong box sometimes no matter what you do. A/A with HA3 packet forwarding makes that a designed-for case instead of an outage: the non-owner firewall forwards the packet over HA3 to the session owner, inspection happens exactly once, and nobody drops anything.
NAT is where the A/P and A/A worlds differ most, and it's worth naming here because it shapes the design. In Active/Passive, NAT is a non-question: one box forwards, so it owns the source-NAT pool, answers the destination-NAT addresses, and holds every translation; on failover the standby inherits all of it (sessions included, thanks to HA2) and carries on with the same addresses. In Active/Active both boxes translate simultaneously, and that splits NAT down the middle: source NAT must become per-device, because two dataplanes cannot share one port-allocation pool, so each firewall NATs outbound traffic to its own upstream address. Destination NAT stays pair-wide, because rewriting a published address to an internal one is stateless and either box can do it for new sessions. Outbound flows therefore leave with two different public identities depending on which firewall ECMP picked, which is invisible to almost everything but matters the day something whitelists your egress IP: whitelist both. The full mechanics, including the commit errors that enforce this split, are the next post.
Session ownership deserves one more sentence, because two knobs control it. The session owner (who inspects the flow) is the firewall that saw the first packet; the session setup device (who does the initial work) is spread by IP modulo. Between them, load distributes without either box needing to agree with the switches about hashing, and HA2 keeps both session tables identical so a mid-flow failover has somewhere to land.
The part that didn't work: 90 seconds of silence
The first failover test was humbling. Kill a firewall and the affected flows black-holed for tens of seconds. The reason is BGP's dirty secret: by default, a dead neighbor is detected by the hold timer, and the default hold timer is 90 seconds. The leaves happily forwarded into a corpse until the timer expired.
The textbook fix is BFD, and the textbook is wrong for this environment: the N9Ks here would not run it reliably. The workable fix was aggressive BGP timers on the firewall-to-leaf sessions: keepalive 1, hold 3, configured on both sides. Two gotchas earned their place in my notes. NX-OS won't renegotiate timers on an established session, so the sessions must bounce after the change. And Palo Alto HA has a testing trap: an admin suspend keeps BGP politely alive, so it never triggers fast convergence. If you test failover with request high-availability state suspend, you'll conclude your tuning did nothing. Test by shutting the firewall's port-channel on the leaves instead.
What failure actually looks like: three regimes
I re-ran the failover live for this post: three continuous pings from a workload through the fabric, then firewall-a's bundle shut on both leaves. The results are more interesting than a single number:
The whole story in fifteen seconds: ECMP through both firewalls, fw-a dies and traffic reroutes without a drop, then the failback rehash.
Regime one: link-down failure. When the bundle dropped, the leaves tore the BGP session down with the link and withdrew the route in milliseconds. ECMP shrank from two next-hops to one, and not a single ping was lost. The hold timer never entered the conversation.
Regime two: silent death. If the box dies while its links stay up (a hung dataplane, a hypervisor freeze), nothing tears the session down for you, and the hold timer is the whole story. That's the case the 1/3 timers exist for: measured at roughly 5 seconds of loss when we tuned it, down from 30 to 48 seconds with defaults.
Worth pausing on regime one, because it's the quiet win of the whole design. Nothing was configured to make that happen: no BFD, no fast-hello protocol, no tuning. A directly connected eBGP session dies with its link, and route withdrawal plus ECMP shrink is as fast as the control plane can program hardware. The expensive machinery is only for failures the link state can't tell you about.
Regime three, the surprise: recovery. When firewall-a's bundle came back and its BGP sessions re-established, the route rejoined ECMP and one of my three flows rehashed onto the returning box mid-stream, dropping 25 packets over 5.4 seconds while the new path settled. Failback is not free. If your change window cares about that, bring the box back with its BGP session administratively damped and release it deliberately.
The takeaway
The nugget inside the deep dive: Active/Active firewalls get dramatically simpler when you let routing do the failover. Both boxes forward, ECMP spreads the load, link-down converges in milliseconds for free, and the one genuinely hard case (silent death) is handled by three seconds of hold timer. The complexity doesn't vanish, it moves: into NAT rules that must be written per-device and a config-sync feature with sharp boundaries. That's the next post.
