Architecture

Designing agents that fail safely

Autonomy is easy to demo and hard to trust. An AI agent that can call your tools can also call them wrong. The real question when building AI agents isn't "will the agent make a mistake?" — the answer is always yes — it's whether that mistake is contained, visible, and reversible. That's exactly what "failing safely" means, and it's an architectural design problem before it's a language-model problem. In this article we share the principles we apply at Grid when designing AI agents that are ready for production, not just for demos.Why the demo succeeds and production failsIn a demo, the team controls everything: the inputs are clean, the script is rehearsed, and the audience is forgiving. In production, the agent meets inputs nobody has seen before, systems that change without notice, and users who probe the edges of the system — sometimes in good faith, sometimes not. The gap between those two environments is where incidents are born.Successful teams don't try to build an agent that "never makes mistakes" — an impossible goal with probabilistic models. Instead, they build infrastructure that makes the cost of any single mistake small and contained. That shift in mindset — from preventing failure to containing it — is the foundation of everything that follows.Start from permissions, not promptsMost agent failures we see in real projects trace back to one cause: broader access than necessary. It's easy to hand an agent a full API key and write "never delete any data" in the prompt — but a prompt is not a guarantee, it's a wish. The model can be misdirected through prompt injection, misread the context, or simply hallucinate.The rule we hold to: scope every agent to the smallest set of tools and data it needs for its task, expressed as explicit system-level permissions — narrow OAuth scopes, restricted keys, and read-only database roles wherever that is enough.An agent should never be able to do something your permission model didn't already allow — regardless of what the prompt says.The advantage of this approach is that it's auditable: you can know precisely what the agent could do in the worst case by reading the permissions, not by guessing at model behavior.Design for blast radius: put humans at the dangerous edgesNot every step needs human approval — an agent that asks permission for every read is useless, and users learn to click "approve" without thinking, which is more dangerous than having no approval at all. The answer is to classify actions by blast radius:Low-impact actions — reading data, drafting content, proposing a plan: run autonomously without stopping.Medium-impact actions — sending an internal email, editing a single record: execute with an immediate notification and an undo path.High-impact actions — deleting data, moving money, messaging customers: always stop for explicit human approval.With this classification you get an agent that is fast in 90% of its work and careful in the 10% that deserves care. Most importantly: when the model misjudges how risky an action is, the safety net lives in the classification layer itself, not in the model's good intentions.Make every decision traceableWhen something goes wrong in production, "the model decided that" is not an answer that satisfies a client or an auditor. Every agent run should emit a complete, reviewable trace: the original inputs, the plan the agent formed, every tool call with its parameters and result, and the final output — all tied to a single run ID.That log is not an operational luxury; it's what turns an incident from a mystery into a lesson. Without it you know something went wrong, but not why, and you can't prevent it from happening again. And with regulators in sensitive sectors, an audit trail is table stakes, not a bonus feature.Make reversibility the first plan, not the emergency planA reversible action can be delegated to an agent with far more confidence than a final one. So we redesign the tools themselves before handing them to the agent: delete becomes restorable archiving, send becomes scheduling with a cancellation window, and edit keeps the previous version. The wider the "reversible" surface in your system, the more you can safely automate — without raising your risk ceiling.Evaluate before launch, monitor afterManual testing is not enough for a non-deterministic system. Before launch, we build every agent an evaluation suite of real cases drawn from the actual work, plus adversarial cases designed to break it: ambiguous inputs, prompt-injection attempts, and rare edge cases. Any change to the prompts or the model runs through this suite before it ships.After launch, monitoring continues on samples of real traffic: task success rates, approval-stop rates, and anomalous patterns in tool calls. A model that worked well last month can change behavior with a new update — continuous monitoring is what catches that before your customers do.A checklist before launching any agentDoes the agent hold the narrowest possible permissions for its task?Are actions classified by blast radius, with mandatory human approval for the sensitive ones?Does every run produce a complete audit trail you can go back to?Are dangerous actions reversible, or deferred with a cancellation window?Is there an evaluation suite that runs before every change, and continuous monitoring after release?The bottom lineA trustworthy agent is not one that never makes mistakes — it's one whose mistakes are contained by permissions, visible in the logs, and reversible by design. These are not limits on what the agent can do; they are precisely what lets you widen its permissions later with confidence, one step at a time.At Grid we build AI agents on these principles from day one: minimal permissions, approvals at the dangerous edges, complete audit trails, and continuous evaluation. If you're planning to bring agents into your business systems, get in touch — we'll help you start safely without sacrificing speed.

6 min read · 20 Jul 2026