There’s a line from Richard Cook’s treatise on complex system failure that I keep coming back to, and it has nothing to do with AI: maintenance and production are usually at cross purposes. If you shut everything down every time you have to change a light bulb, you’re never going to get anything done.
That’s it. That’s the whole problem with the agent stacks I review every week, described decades before anyone shipped a tool-calling loop.
Cook was writing about patient safety, not software. But the observation travels. Every organization running a complex system faces the same standoff: the thing that keeps the system alive is the thing that stops it from doing its job. And in a space where “ship weekly” is treated as a virtue, guess which side of that standoff usually wins.
The failure mode nobody puts in the demo
When I evaluate an agent platform, the pitch is always about capability. Look how many tools it can call. Look how it chains steps. Look how it recovers from a bad API response. Fine. What I almost never see is an honest accounting of what happens when three of those steps degrade at once.
Complex systems don’t fail because one component breaks. They fail through cascades — small faults that individually mean nothing, lining up in an order nobody modeled. A stale credential, a rate limit, a retry policy that’s slightly too aggressive, and suddenly your “autonomous” workflow is hammering a production endpoint with garbage.
The uncomfortable part is that each of those pieces was working as designed. That’s the signature of cascading failure. Nothing was broken. Everything was fine. The system still went down.
Why agent stacks are unusually good at this
Agent architectures have a structural problem: every new integration adds failure paths to a system you thought you understood. One perspective I found on Cook’s work framed this as increased entropy plus a lack of training on the new surfaces — and that pairing is exactly what I see in practice.
Teams bolt a new connector onto an agent, the happy path works, and nobody develops a mental model of the failure behavior. Six weeks later something misfires and the on-call engineer is reading a trace they’ve never seen before, in a component they didn’t build, from a vendor whose status page says all systems operational.
The stacks I’d actually trust share a few unglamorous traits:
- They can tell you what the system did, not just what it decided. Traces over summaries.
- They fail closed on ambiguity instead of guessing and continuing.
- They let you update one component without a full stop-the-world deployment.
- They treat permissions as a first-class concern rather than a config file you fill out once.
That last one matters more every year. Security vulnerabilities are a standing feature of complex systems, not an occasional visitor. A large share of the updates you’re tempted to skip are patches for attack vectors that didn’t exist last quarter. Digital threats move fast; defenses have to move at the same speed or the gap becomes the story.
The maintenance tax is real, and you’re already paying it
Which brings us back to the light bulb. Cook’s tension doesn’t resolve. You can’t maintain your way to zero downtime and you can’t ship your way out of decay. What you can do is stop pretending the choice isn’t being made.
Most teams I talk to are making it by default — deferring updates because the agent is “working,” accumulating security debt on a system whose failure modes they can’t enumerate. That’s not a strategy. That’s a bet that the cascade won’t line up this quarter.
Proactive maintenance sounds like the kind of phrase that shows up in a vendor deck, which is a shame, because in 2026 it’s just arithmetic. The cost of scheduled, boring, partial updates is knowable. The cost of an unscheduled cascade in a system with production write access is not.
What I’m grading for now
My review criteria have shifted. Capability benchmarks tell me what a tool does on a good day, and good days are not where systems die. I’m more interested in whether a platform can be patched without a full teardown, whether its failure behavior is documented rather than discovered, and whether the vendor talks about degradation with any specificity at all.
If a company can’t describe how its agent behaves when a dependency goes sideways, that’s not a gap in their docs. It’s a gap in their understanding of the thing they built — and you’ll be the one who finds it.
Cook’s treatise is nearly a generation old and available openly under a Creative Commons licence. It’s short. Read it before your next architecture review. It will make you worse at being impressed by demos, which is the point.
🕒 Published: