On this page
Part 6 of the seven-part series Building an Agentic NOC. Each part builds on the ones before it; the full list is at the end.
The architecture is built. This post shows what it does when you run it against live infrastructure, and every block of output below is captured from a real run, not composed for the post. That is the whole point: a system that touches production has to be judged on what it actually did, so here is what it actually did.
A request through the full stack
Start with the simplest thing: ask the fabric a question and watch it travel down both protocol layers. The supervisor receives the request, routes it over A2A to the fabric specialist, and the specialist drives its read-only MCP tools, which run real commands against a real switch. The ten-step ladder walks from the physical layer up to border egress and reports each rung; the tenth rung, end-to-end reachability, runs from a host, so this device run ends at rung nine.
$ python3 -m agents.supervisor "run the ladder on cluster-leaf-1"
routed to: fabric-agent
[fabric-agent] run_ladder(device=cluster-leaf-1, peer_vtep=10.0.1.3)
VERDICT: healthy
rung1 physical PASS all 7 fabric links up
rung2 underlay_bgp PASS 2/2 underlay sessions up
rung3 vtep_reachability PASS VTEP 10.0.1.3 reachable in underlay
rung4 ecmp PASS 2 equal-cost path(s) to 10.0.1.3
rung5 mtu PASS 9000B DF-bit to 10.0.1.3 passes
rung6 nve_tunnels PASS 12 VTEPs, 2 VNIs
rung7 evpn_control PASS 2/2 EVPN sessions up
rung8 tenant_vrf PASS VRF tenant-1 is Up
rung9 border_egress PASS default route present in tenant-1 via 2 path(s)That output crossed two protocol boundaries. The supervisor spoke A2A to the specialist over HTTP, the specialist spoke MCP to its tool server over stdio, and the tool server ran nine sets of read-only commands over SSH against a live NX-OS switch, then the structured result flowed back up the same path. This is the MCP-under-A2A stack from the earlier posts, doing real work. The command goes straight through the supervisor's deterministic routing, which is why it prints the routing decision. The ./noc wrapper sends the same request through the LLM planner instead, which we look at next.
The planner, fanning out in natural language
Ask the whole stack a plain-English question and the local planner takes over. It asks a model running in Ollama on the same machine which specialists to consult, fans those out concurrently, and synthesises one answer. No hosted API, no credits, and the model is never the source of a fact.
$ ./noc "give me a full health check of the whole stack"
=== planner mode: llm:ollama (model qwen3.5:35b) ===
plan:
-> fabric-agent: run-ladder on cluster-leaf-1
-> edge-agent: check fw-a health
-> k8s-agent: is the cluster healthy
=== answer ===
Stack Health Verdict: HEALTHY
All specialist agents report successful checks across the stack, no failures:
Fabric (cluster-leaf-1): all physical links, underlay BGP, VTEP reachability,
ECMP, MTU, NVE tunnels, EVPN control, and tenant VRF operational.
Edge (fw-a): HA active/active healthy; 2 eBGP sessions established; 4 interfaces up.
Kubernetes: all 6 nodes ready; Cilium BGP advertising the VIP prefix;
gateways and LB services programmed.
--- raw specialist output (ground truth) ---
[fabric-agent] run-ladder on cluster-leaf-1
[fabric-agent] run_ladder(device=cluster-leaf-1, peer_vtep=10.0.1.3)
VERDICT: healthy
rung1 physical PASS all 7 fabric links up
rung2 underlay_bgp PASS 2/2 underlay sessions up
rung3 vtep_reachability PASS VTEP 10.0.1.3 reachable in underlay
rung4 ecmp PASS 2 equal-cost path(s) to 10.0.1.3
rung5 mtu PASS 9000B DF-bit to 10.0.1.3 passes
rung6 nve_tunnels PASS 12 VTEPs, 2 VNIs
rung7 evpn_control PASS 2/2 EVPN sessions up
rung8 tenant_vrf PASS VRF tenant-1 is Up
rung9 border_egress PASS default route present in tenant-1 via 2 path(s)
[edge-agent] check fw-a health
[edge-agent] run_edge_checks(device=fw-a)
VERDICT: healthy
ha PASS A/A healthy: local active-primary, peer active-secondary
bgp PASS 2/2 eBGP sessions Established
interfaces PASS all 4 routed interfaces up
[k8s-agent] is the cluster healthy
[k8s-agent] run_k8s_checks()
VERDICT: healthy
nodes PASS 6/6 nodes Ready
cilium-bgp PASS all 6 sessions up, advertising 1 VIP prefix(es): 192.168.200.0/32
gateways PASS all 1 gateways programmed, 1 LB service(s) addressedThat is one real run, and it is worth being honest about what will and will not reproduce. The plan and the prose are produced by the model, so the exact wording, and even which specialists it chooses to consult, vary from run to run: this run checked one firewall, another run might check both. What does not vary is where the facts come from, and you do not have to take that on faith. Beneath the synthesis, the run prints the verbatim tool output the summary was built from: the ground-truth block above. Every claim in that verdict, the active/active HA state, the two eBGP sessions, the six ready nodes, the advertised VIP, is a line in that block, produced by a specialist's read-only tool call against live infrastructure, not by the model. The model routed and it wrote. The numbers came from the tools, not the model, and appear verbatim in the ground-truth block beneath, which is exactly why its varying prose is safe to trust.
Closing the loop: detect, remediate, verify
Reading is half a NOC. The other half is fixing things, and this is where the guarded write path earns its narrowness. We ran a closed-loop test: inject a fault by administratively shutting one redundant link, let the read-only agent detect it, let the guarded path repair it, and verify. Thirty times, one after another, across every device role.
OK [ 1/30] cluster-leaf-1 Ethernet1/1 -> spine-1 detect=True remediate=True verify=True
!! [ 3/30] cluster-leaf-2 Ethernet1/1 -> spine-1 detect=False remediate=True verify=True
!! [ 6/30] cluster-leaf-3 Ethernet1/2 -> spine-2 detect=False remediate=False verify=True
...
=== 30 faults: detected 28/30, remediated 29/30, verified restored 29/30 ===Twenty-seven of thirty were perfect on all three counts, and the whole fabric was verified fully restored at the end. The three anomalies are the interesting part, and none of them is a remediation error. They all fell on two switches during SSH stalls on the loaded simulator, and in every case the guardrails behaved correctly. Fault three: the agent's detection read got stale during a stall, but remediation's own precondition check found the port down and fixed it anyway. Fault six is the one to sit with: the fault injection itself stalled, so the port never actually went down, and the guarded path, checking state before acting, correctly refused to touch a healthy port. The audit log recorded twenty-nine no shutdown actions in that run and zero wrong ones. A remediation system that refuses to act on a port that is already up is exactly the remediation system you want near production.
Autonomy: the monitor self-healing
The closed loop above was driven by a test harness. The monitor drives the same loop on a timer, with no human involved. Inject a fault while it is running and the log tells the story on its own.
10:01:55 FAULT_DETECTED cluster-leaf-1 Ethernet1/1 down (Administratively down) -> spine-1
10:02:03 REMEDIATION_APPLIED cluster-leaf-1 Ethernet1/1 no shutdown
10:02:05 FAULT_CLEARED cluster-leaf-1 Ethernet1/1 recoveredTen seconds, no operator. The monitor detected the down link, the guarded path re-enabled it, and the next cycle confirmed recovery.
Autonomy done responsibly: escalation, not endless fixing
Self-healing is dangerous if it is naive, because a genuinely broken link that keeps failing would get repaired forever, hiding a real fault. So we forced a port to flap, repeatedly shutting it faster than the monitor could keep it up. After two repairs inside the window, the monitor stopped.
10:03:25 REMEDIATION_APPLIED cluster-leaf-1 Ethernet1/1 no shutdown
10:03:41 REMEDIATION_APPLIED cluster-leaf-1 Ethernet1/1 no shutdown
10:03:49 ESCALATION cluster-leaf-1 Ethernet1/1 2 remediations in 300s -- stop auto-fixing, page a humanIt repaired the link twice, recognised the flap, and escalated rather than fight it. For a short demo the flap guard was lowered to two repairs in three hundred seconds with NOC_MONITOR_FLAP_THRESHOLD and NOC_MONITOR_FLAP_WINDOW; the shipped defaults are three repairs in six hundred seconds. That single behaviour is the difference between an autonomous system you can trust near production and one you cannot.
Watching all three domains, and refusing to fake health
Run the monitor across fabric, edge, and cluster together and it reports a clean baseline, then handles a real problem honestly.
10:38:28 MONITOR_START domains=fabric+edge+k8s, every 60s, mode=DETECT+ALERT ONLY
10:39:48 HEALTHY baseline clean across fabric+edge+k8s
10:41:48 DEVICE_UNREACHABLE fabric service-leaf-1 could not read this cycle (not treated as healthy)
10:42:44 MONITOR_STOP 3 cyclesThe middle line matters more than it looks. On the third cycle a switch stalled and could not be read. The monitor did not shrug and call it healthy, it logged the device as unreachable and explicitly did not count it as clean. Silence is not success, and a monitor that treats an unreadable device as a healthy one is worse than no monitor at all. Alert-worthy events, including that one, are pushed to Telegram, so the operator learns about a fault or an escalation on their phone, not by reading a log.
The spine underneath all of it
Every result above rests on the same foundation, the one from the build post: the tool refuses to pass along a value it can prove is impossible. The counter that read eighteen quintillion errors never reached the agent, because the gate stopped it. The leaf-sourced ping that would have shown false loss was never run, because the tool knows about the anycast gateway. The forty-nine phantom down ports were never reported, because the physical check derives its link set from LLDP. Those three lies, defused in the tool, are why the healthy verdicts above are trustworthy and the fault verdicts are real.
As for scale, a single request through the deterministic path returns in seconds, a k8s health sweep of thirty questions averaged just over five seconds each with zero errors, and a full tri-domain monitor cycle takes around eighty seconds of sequential SSH. This is a homelab running virtual switches, so those numbers are about correctness and behaviour, not throughput, and the honest framing matters: the point of this system is that its answers can be trusted, not that it is fast.
One post remains. Everything you have seen here was validated, and the final post is about how, because the claim that a monitoring system can be trusted is only as good as the tests behind it. We built one hundred and sixty-two of them, and one whole category exists because a stress test caught a bug we did not know we had.
Building an Agentic NOC, the full series:
