Replicating a Two-DC EVPN Datacenter in Containerlab

I have a two-datacenter EVPN-VXLAN lab that I care about: 20 Cumulus Linux switches across two leaf-spine fabrics and a routed backbone, a Palo Alto Active/Active firewall pair, two Kubernetes clusters running Cilium, a host-based GoBGP anycast controller, DNS, and external clients. It lived in EVE-NG, and it worked. It also lived in exactly one place, held together by disk images and memory.

So I rebuilt it in containerlab. Not a cut-down version: all 39 nodes and 75 links, and every switch, firewall, host and cluster configures itself on clab deploy. Today the whole thing is a public git repo. Clone it on a fresh host, build the images once, and from there twenty hands-off minutes separate clab deploy from an external client curling an anycast VIP by DNS name, through the firewall pair, into whichever datacenter the control plane picked.

What containerlab buys you (and what it does not)

Let's kill one myth first: this is not a resource saving. Cumulus VX has no working container-native path for the 5.x releases, so every switch is still a full VM booting the same qcow2 it booted in EVE-NG. The supported path is containerlab's nvidia_cumulusvx kind, which wraps the VM in a container using vrnetlab. Same deal for the PA-VMs. You need KVM and a decent host either way.

What you actually gain is that the lab becomes text. The topology is a YAML file generated from the real link map. Every switch config, firewall config, and host bootstrap script sits in git next to it. The mgmt network has static IPs instead of DHCP addresses that drift every boot. And the whole thing rebuilds the same way every time with one command (VX has exactly one known flake, a sticky ES bond, and the repo ships a watchdog for it). When a node misbehaves, you stop debugging state and start debugging code.

The topology is the config

A generator script holds the authoritative link map, captured live from the EVE-NG lab with the EVE API, none of it inferred. It emits the containerlab YAML: switches point at their startup-config, hosts run bind-mounted bootstrap scripts, and the Kubernetes nodes carry their entire identity as environment variables.

That env: block is doing a lot of work. The k8s host image has a PID-1 entrypoint that reads it, builds the dual-homed LACP bond, sets the fabric default route, and launches k3s in the right role. The first master then installs Cilium with the BGP control plane, deploys the demo app, and creates every anycast VIP. Nobody types anything.

Making first boot do everything

The catch: almost none of this works out of the box. The interesting engineering lives in a pair of vrnetlab launcher patches that ship in the repo.

Cumulus VX. The stock launcher never applies startup-config at all, and it carries a bug where no VX node ever reports healthy (a bytes-versus-string comparison that can never be true). The patched launcher replays the nv-set config over the serial-side SSH once switchd is up. For the EVPN multihoming leaves it goes further: VX only converges Ethernet Segment bonds after a reboot, so the launcher reboots the leaf once and re-applies evpn mh redirect-off, all inside first boot.

PAN-OS 12. Three separate fights. qemu64 lacks x86-64-v2, so PAN-OS 12 cannot even boot without QEMU_CPU: host, and at vrnetlab's default 2 vCPU and 6 GB it drops into the maintenance recovery tool; the working shape is 8 vCPU and 16 GB. Set-format config replay fails silently because PAN validates each line as it arrives and the export is not in dependency order, so the launcher imports a full sanitized XML config through the API instead. And jumbo-frame mode only activates after a reboot, so the bootstrap is two-phase: enable jumbo, commit, reboot, wait out the auto-commit, then load and commit the real config. About 19 minutes per firewall, fully unattended.

Kubernetes in a privileged container is the kind/k3d problem: the PID-1 entrypoint evacuates the cgroup-v2 root before containerd starts, or no pod ever schedules. Everything is airgapped on purpose. The nodes default-route into the fabric, not the internet, so every image k3s, Cilium and the demo app need is baked into the host image.

Proof, not promises

(And yes, k8s-woker-1: the typo shipped with the original lab, and faithful means faithful.)

The acceptance test was brutal on purpose: destroy the lab, delete the clone, git clone from the repo, run the build scripts, deploy, and touch nothing. The fabric converges in about 4 minutes, both clusters go Ready around minute 12, the firewalls finish near minute 19. Then the control-plane check: GoBGP health-probes both datacenters and steers each VIP by policy, and the client reaches the app by name through the firewall DNAT.

That traceroute is my favorite artifact in the lab. A DC1 pod traces to a DC2 node and you can read the whole east-west path: a Cilium hop on the source node, the DC1 leaf SVI (10.167.10.2), border, backbone aggregation, DC2 leaf, DC2 node. The two 192.0.0.8 entries are the borders: those links run BGP unnumbered with no IPv4 address, so the kernel answers traceroute with the reserved IPv4 dummy address. No firewall in the path, exactly as designed. North-south goes through the PA pair; east-west rides the backbone.

The clone test earned its keep, too. The first run deadlocked both firewalls at the password prompt: vrnetlab's telnetlib-style console wrapper discards unmatched data on timeout, so a prompt split across two read windows can never match. Three earlier deploys had simply won that race. The fix (nudge the console after 20 stalled polls) is in the repo, and the failure would never have surfaced without rebuilding from scratch.

What bit me, so it does not bite you

• Docker's default route. The mgmt interface gets a metric-0 default that silently beats the fabric route. Cross-subnet replies leave via mgmt and die. The entrypoint deletes it and makes the fabric bond the only default.
• Partial redeploys lie. On VM kinds, recreating one node makes containerlab recreate its link peers, and a partially reconciled deploy can leave broken veths that no reboot fixes. The lab rebuilds in 20 minutes; a clean destroy and deploy is always cheaper than surgery.
• Grey failures are invisible to the VM. Cut a container veth and the guest NIC stays up, so link monitoring never fires. My Active/Active firewall drill blackholed until the whole box was killed. Path monitoring, not link monitoring, is the answer.

Try it

The repo is public. It needs a KVM host with real headroom (the 22 VMs want roughly 48 cores and 128 GB), docker, containerlab, a vrnetlab checkout, and two images you bring yourself: the Cumulus VX 5.12 qcow2 (NVIDIA no longer distributes it) and a PA-VM 12.1.2 from your support account. After that:

git clone https://github.com/aramidetosin/ecloud-containerlab.git && cd ecloud-containerlab
mkdir images        # drop the two qcow2s here
# host prerequisites (bonding module, the shared bridge): step 1 of the README
python3 cvx_startup_config.patch.py <vrnetlab>/nvidia/cumulus-vx/docker/launch.py
python3 pan_panos12.patch.py       <vrnetlab>/paloalto/pan/docker/launch.py
VR=<vrnetlab> ./build_images.sh && k8s/build-k8s-image.sh
sudo containerlab deploy -t ecloud.clab.yml

One build session and twenty hands-off minutes later, you own a two-datacenter anycast platform that fits in a git repo. The next post uses it to walk through every EVPN route type with live output, including the two types most labs never get to see.

Sources

• containerlab documentation
• hellt/vrnetlab (VM-in-container launchers)
• NVIDIA Cumulus Linux documentation (EVPN multihoming reference)
• Cilium BGP control plane
• GoBGP