On this page
Part 5 of the seven-part series Building an Agentic NOC. Each part builds on the ones before it; the full list is at the end.
The last four posts were groundwork. You have both protocols in your hands and you know how they emerged. Now we build the real system: a read-only network operations centre where a supervisor agent speaks A2A to three specialists, and each specialist speaks MCP to the live infrastructure of its domain, a VXLAN and EVPN fabric of twelve NX-OS switches, a Palo Alto firewall pair, and a Kubernetes cluster. This post is the architecture and the load-bearing decisions, with the real code for the parts that matter. It is not a "clone this repo" post, partly because the repo carries a live lab's addressing and partly because the useful thing to take away is the pattern, which you apply to your own gear.
One rule governs the whole design, and if you remember nothing else, remember this: the guardrails live in the tool, not in the model's judgement. An agent is only as safe as the worst thing its tools will let it do. We build tools that cannot do the wrong thing, and then it does not matter how the model reasons.
The shape
The system is five parts, layered exactly the way the ecosystem converged.
noc-agent/
mcp-fabric/ read-only MCP server for the 12 switches
mcp-edge/ read-only MCP server for the firewall pair
mcp-k8s/ read-only MCP server for the cluster
a2a-noc/ supervisor + 3 A2A specialists + an LLM planner
remediation/ the one guarded write path
monitor/ the autonomous poller + Telegram alerting
noc one entry pointReading bottom up: three MCP servers expose read-only tools over each domain. Three A2A specialists each wrap one MCP server. A supervisor discovers the specialists by their Agent Cards and routes work. An LLM planner sits above the supervisor for natural language. A separate, tiny guarded-write path handles the one kind of remediation we allow. An autonomous monitor drives the whole thing on a timer. Every layer is read-only except the remediation path, which is deliberately isolated and narrow.
Layer one: read-only MCP servers, enforced in the transport
Each MCP server talks to its domain over SSH or an API and exposes typed tools. The critical design choice is that read-only is not a convention or a code review rule, it is enforced in the transport before any command is sent. For the fabric server, which drives NX-OS over SSH, the transport will only ever emit a show, ping, or traceroute, and refuses anything containing a state-changing token.
# fabric/transport.py -- the read-only guarantee
_ALLOWED = re.compile(r"^\s*(show|ping|traceroute)\s", re.IGNORECASE)
_FORBIDDEN = re.compile(
r"\b(conf|configure|clear|reload|write|copy|delete|erase|install|"
r"shutdown|no\s|boot|format|debug)\b", re.IGNORECASE)
def assert_read_only(command: str) -> None:
if not _ALLOWED.match(command):
raise NotReadOnly(f"refused: not a read-only verb: {command!r}")
if _FORBIDDEN.search(command):
raise NotReadOnly(f"refused: state-changing token: {command!r}")This is what "guardrails in the tool" means concretely. A prompt injection in device output, a confused model, a bug in a higher layer, none of them can turn this server into something that writes to a switch, because the string never reaches SSH if it is not provably a read. The edge and cluster servers apply the same principle: PAN-OS is restricted to show, and Kubernetes is restricted to kubectl get/describe/version/top plus the read-only Cilium views.
On top of that transport, each tool is a small typed function, exactly the pattern from the MCP post, scaled up. The fabric server exposes a ten-step troubleshooting ladder, one tool per layer from physical up through BGP, VXLAN tunnels, EVPN, and end-to-end reachability, plus a tool that walks the ladder and stops at the lowest broken rung, because a symptom three layers up is not the fault.
The finding that shaped everything: infrastructure lies
Here is the part that justifies the entire "guardrails in the tool" philosophy, and it is worth dwelling on because it is not obvious until it bites you. Three separate times, a completely standard command returned a confidently wrong answer on a perfectly healthy fabric. An agent that trusted those answers would have raised false alarms or, worse, taken action on them.
The first was interface counters. A standard show interface counters errors reported receive errors in the range of eighteen quintillion, a value near two to the sixty-fourth power. The errors were actually zero. The virtual switch was corrupting the counter with an additive splice, and the corruption moved between devices between reads, so no static "trust this device" list could save you. The fix is a gate in the tool that refuses to emit any counter at or above two to the fortieth, because that magnitude is not traffic, it is corruption.
# gate.py -- refuse values physics says are impossible
GATE = 1 << 40 # 1.1e12; a real 20s counter delta is ~1e8
def check(field, raw):
val = int(raw)
if val >= GATE:
return Counter(field, val, trusted=False,
reason="platform counter splice, not traffic")
return Counter(field, val, trusted=True)The second lie was reachability. A ping from a leaf switch inside the tenant VRF returned one hundred percent loss on a healthy fabric, because the gateway address is an anycast address shared by every leaf, so replies land on whichever leaf is nearest the responder rather than the originator. The tool refuses to run that test from a leaf and runs it from a host instead. The third lie was link state. A naive physical check reported forty-nine ports down, because the virtual switches present sixty-four ports and only the cabled ones are up. The fix is to derive the real link set from LLDP, the neighbours the device actually sees, and check only those.
None of these is the model hallucinating. In every case the infrastructure handed over a wrong number, and no amount of careful reasoning downstream recovers from bad input. The only defence that works is a tool that refuses to pass along a value it can prove is impossible. That is the whole argument for typed, validating tools over handing a model a raw shell, and it is the spine of the series.
Layer two: A2A specialists and a supervisor
Each specialist is an A2A server and an MCP client at the same time. It publishes an Agent Card whose skills are grounded in the tools its MCP server actually implements, and when it receives a task it drives those tools. Crucially, the specialist spawns its MCP server as a real subprocess and speaks MCP to it, rather than importing its functions, because collapsing the two protocols into one process would make the architecture a fiction. The seam is real.
The supervisor is an A2A client. It fetches each specialist's Agent Card, then routes an incoming request to the right specialist by domain. Routing scores requests by distinct keyword matches rather than first match, a detail that came directly from a stress test which caught the naive version misrouting. The word "bgp" for instance is deliberately not a routing keyword for any domain, because it means fabric underlay BGP, PAN-OS BGP, and Cilium BGP all at once, so it cannot discriminate.
Two of the three specialists began as honest stubs. Before the firewall and cluster tool servers existed, those specialists published truthful Agent Cards that advertised no working skills and returned an explicit "not implemented" for any request. Because the supervisor routes on advertised capability, a stub is never handed work it cannot do, and the system degrades honestly instead of pretending. That is a small decision that matters a lot: vaporware in an Agent Card is a lie the whole system will act on.
The brain: a local-first LLM planner
Above the supervisor sits a planner that takes natural language, asks an LLM which specialists to consult and what to ask each, fans those out concurrently, and synthesises one answer. It prefers a local model running in Ollama on the same machine, and only falls back to a hosted model if no local one is available, so the default path costs nothing and keeps infrastructure data local. Without any model it degrades to deterministic keyword routing and says so.
The planner's design rule is the same rule in a new place: the LLM plans and writes, it never produces facts. Every status, number, and device name in the final answer comes from a specialist's read-only tool output, which comes from live infrastructure. The model is a router and a writer, not a source of truth. This is the counter-gate principle applied one layer up: keep the model away from being the authority on what is real.
The one write path, kept tiny on purpose
Everything above is read-only. Remediation needs to write, so it lives alone in its own module behind an allowlist narrow enough to express exactly one intent: re-enable a fabric link that was administratively shut but should be up. It can emit precisely one command, and there is no code path that can emit anything else.
# guarded_write.py -- the only thing that can change a switch
def decide(device, port):
if port not in baseline_links(device):
return Decision(allowed=False, reason="not a baselined fabric link")
state = port_admin_state(device, port) # a read-only show
if state == "up":
return Decision(allowed=False, reason="already up; nothing to do")
if state == "Link not connected":
return Decision(allowed=False, reason="physical fault; no shutdown won't fix a cable")
if state != "Administratively down":
return Decision(allowed=False, reason=f"unexpected state {state!r}")
return Decision(allowed=True, command=f"configure terminal ; interface {port} ; no shutdown")It only acts on a known fabric link, only when that link is administratively down rather than physically broken, and the only command it can build is no shutdown. It cannot shut a port, cannot touch routing, cannot reconfigure anything. Dry-run is the default and every action is audited. This is the same law as the read-only servers: the guardrail is in the tool, so a caller's intentions, good or bad, cannot exceed what the tool allows.
Making it autonomous: the monitor
The top layer is a poller that runs on a timer and probes all three domains each cycle. It is edge-triggered, so a new fault pages, a still-open fault does not re-alert, and a recovered fault pages the recovery. For the fabric, and only if auto-remediation is explicitly enabled, an administratively-shut link is repaired through the guarded path. Firewall and cluster faults are always alert-only, because no guarded write path exists for them and the monitor does not pretend otherwise.
Three safety behaviours make autonomy responsible rather than reckless. Auto-remediation defaults off, because autonomous write access is a decision you make deliberately. A port that is repaired and fails again is flapping, so after a threshold the monitor stops fixing it and escalates to a human rather than fighting it forever. And a device it cannot read this cycle is logged as unreachable and never counted as healthy, because silence is not success. Alert-worthy events are pushed to Telegram; everything is written to a structured log.
One entry point
All of it runs from a single script, with configuration in a gitignored .env file rather than scattered environment variables.
./noc "is the whole stack healthy" # one request through the planner
./noc --monitor # the autonomous poller
./noc --test # every offline test suiteHow to rebuild this for your own network
The pattern transfers even though the addressing does not. Start with one read-only MCP server over your most important domain, and put the read-only guarantee in its transport on day one, not later. Write your tools against captured real output, and be suspicious of every value your infrastructure returns, because the three lies above are not unique to a simulator, real gear has its own, from stale ARP to counters that wrap. Wrap the server in an A2A specialist with a truthful Agent Card. Add a supervisor only when you have more than one specialist. Add an LLM planner only when you want natural language, and keep it away from producing facts. Add a write path last, alone, and make it so narrow it can only do the one safe thing. Add the monitor when detection is solid, and make silence loud.
Next post shows what this actually does when you run it, with the real output of the supervisor reaching a live switch, the planner fanning out, the guarded loop closing, and the monitor self-healing. Then the final post shows how the whole thing is tested, because a monitoring system you cannot trust is worse than none.
Building an Agentic NOC, the full series:
