On this page
Part 7 of the seven-part series Building an Agentic NOC. Each part builds on the ones before it; the full list is at the end.
A monitoring system you cannot trust is worse than no monitoring system, because it replaces honest uncertainty with false confidence. So the last question in this series is the most important one: how do you earn the right to trust a system that watches your network and, sometimes, changes it? The answer is one hundred and sixty-two offline tests across four suites, a live fault-injection campaign on real hardware, and one governing principle: test against reality, never against your own assumptions about it. This post walks through how, because the claim that this system can be trusted is only as good as the evidence behind it.
$ ./noc --test
== mcp-fabric offline == 30 passed, 0 failed (parsers)
21 passed, 0 failed (fault detection)
== a2a-noc offline == 85 passed, 0 failed (routing, cards, planner)
== monitor offline == 26 passed, 0 failed (state machine, alerting)Parsers tested against what the device really returns
The first and largest trap in this whole project was not the network, it was the output format. NX-OS emits JSON, but it emits a single row as a bare object and multiple rows as a list, under the same key, so a parser that assumes a list crashes on a single-peer device. Field names mix hyphens and underscores across commands. One command returns a flat object with no wrapper at all. If you write your parsers against a tidy example from the documentation, they will break on the first real device.
So the parser tests do not use invented data. They run against JSON captured verbatim from the live fabric and committed alongside the tests. Thirty of them assert that the parsers extract the right values from real output, including the dict-versus-list case that bit us, the inconsistent field naming, and the wrapperless command. The rule is simple and it is the difference between tests that pass and code that works: your test fixtures must be real output, not your idea of what the output looks like.
Detecting faults without breaking live gear
A checker that only ever returns healthy is worthless, because it has never been shown to notice a problem. But you cannot break a live fabric twenty different ways just to test your detection. The resolution is to take the real captured JSON and mutate it into known-broken states, then assert that each check fails correctly. Twenty-one tests do this: a peer flipped from Established to Idle must produce a degraded verdict, an interface that the baseline says should be up but reads down must produce a failure that names the peer on the other end, a tenant VRF that vanishes must fail. These prove the checks detect the shape of a fault, against real data, without touching a switch.
The part you cannot mock: live fault injection
Mutated JSON proves the parser reacts correctly, but it does not prove the device behaves the way you expect during a real fault, so at some point you have to break real hardware. We did, in a controlled way: administratively shut one redundant link, thirty times across every device role, and drove the full detect, remediate, verify loop against it, restoring the fabric completely at the end. That campaign, shown in the last post, is the test that mutated data cannot replace, and it is where the guarded write path proved it would refuse to act on a port that was already up. Offline tests prove the logic, the live campaign proves the behaviour, and you need both.
The state machine, and the subtle safety cases
The monitor has twenty-six tests, and the interesting ones are not the happy path. They are the safety corners that are easy to get wrong and dangerous when you do. A fault that persists across cycles must alert once, not every cycle, or the operator learns to ignore it. A port that is repaired and fails again must escalate after a threshold rather than be repaired forever. A firewall or cluster fault must be alert-only even when auto-remediation is on, because no guarded write path exists for those domains. And the one that matters most: a device that could not be read this cycle must not have its open faults cleared, because a stall is not a recovery. Each of these is a test, because each of them is a way the system could quietly lie to you.
The tests that found bugs we did not know we had
The best evidence that a test suite is doing its job is that it catches something you did not write it to catch. Two examples from this build are worth telling honestly, because they are the reason to bother.
The first was a stress test. We fired sixty questions through the agents to see if the routing held under load, and it did not: the cluster questions scored eighteen out of thirty. The stress test had found a real routing bug. The supervisor matched keywords with word boundaries, so the pattern for "node" never matched the plural "nodes", and several perfectly reasonable questions routed nowhere. We had never noticed because our own examples happened to use the singular. The fix was to match on prefixes, and the score went to thirty out of thirty. A stress test that only ever confirmed what we already believed would have taught us nothing.
The second was a bug in a test harness itself, which is a useful humility. The fabric stress run reported eight errors, and for a moment that looked like agent failures. It was not. Our harness flagged any reply containing the word "error" as a failure, and the physical check's evidence legitimately contains the phrase "aggregate input/output error counters not evaluated". The agents were correct every time; the measurement was wrong. The lesson, which cost an hour, is that your test oracle can be the thing that is broken, and you should suspect it too.
The worst bug, and why it drove the design
There is one class of bug that is worse than any wrong answer, and it shaped the whole system. Early on, the ladder tool reported a device as healthy while one of its rungs had actually timed out and never run. It downgraded its verdict on a failure and on a degraded result, but an unchecked rung, an outright unknown, slid through as if it had passed. A device that was unreachable on every rung would have come back healthy. That is the worst possible bug in a monitoring tool, because it manufactures confidence it has not earned, and no amount of careful reasoning downstream can recover from a tool that confidently reports health it never established.
The fix was to make unknown a first-class outcome that forces an inconclusive verdict, and then to encode that rule as tests everywhere it applies: the ladder, the monitor, the multi-domain poller. Silence is not success. A tool that cannot check something must say so loudly, never quietly assume the best. If you take one engineering principle from this entire series, take that one, because it generalises far beyond agents and networks.
Where this leaves us
Seven posts ago this was two protocols and a diagram. MCP is the vertical axis from an agent to its tools, A2A is the horizontal axis across agents, ACP converged into A2A, and the two survivors compose into a clean two-layer stack. We built a minimal server and a minimal agent to see the JSON-RPC each one speaks, traced how the protocols emerged and consolidated, then built a real read-only network operations centre on top of them: three MCP servers over a live fabric, firewall pair, and cluster, three A2A specialists and a supervisor, a local-first planner that routes and writes but never invents facts, one tiny guarded write path, and an autonomous monitor that self-heals, escalates, and refuses to fake health.
The thread through all of it is a single idea, and it is not really about AI. Put the guardrail in the tool, not in the judgement of whatever is calling it. Refuse to emit a value you can prove is impossible. Refuse to act on a port that is already up. Refuse to call an unreadable device healthy. An agent, a model, or a script is only ever as safe and as honest as the worst thing its tools will let it do, so make the tools incapable of the wrong thing, and then it does not matter how cleverly, or how carelessly, they are driven. That is how you let software touch your network and still sleep at night.
Building an Agentic NOC, the full series:
