MHMunir Hassan

Leadership · AI · Architecture

Human-in-the-loop is not a security boundary

MHMunir Hassan//8 min read

In nine seconds, an AI coding agent deleted a company’s production database, along with the backups stored beside it. It was working on a routine task in staging.

The common response is to add a human: make the agent ask before it does anything risky. But human approval is a decision control. It improves the judgment behind an action. It doesn’t limit what an agent is able to do. That job belongs to permissions, and in both incidents below, the permissions were the real failure.

My argument is simple: use the system to decide what an agent can do, and use people for the judgment the system can’t make.

What actually happened

SaaStr and Replit, July 2025. During an explicit code freeze, Replit’s agent deleted the production database behind SaaStr founder Jason Lemkin’s project, after he’d told it repeatedly not to make changes. The telling part is Replit’s fix. It didn’t promise a better-behaved agent. It began automatically separating development and production databases, “to prevent this categorically”, in its CEO’s words.

PocketOS, April 2026. A Cursor agent working in staging hit a credential mismatch. It found an API token, made a single API call to the infrastructure provider, and deleted the storage volume holding the production database. The backups lived on the same volume, so they went too. The company ran on a three-month-old backup for about 30 hours. One rule the agent cited afterwards said never to run destructive git commands. Deleting a volume isn’t a git command.

Two lessons follow:

  • An instruction in a prompt is a request, not a guardrail. And a list of forbidden actions only covers the actions someone thought to list.
  • The environment didn’t define the risk. The credential did. The task was in staging. The token could reach production.

Generating, executing, authorising

“Using AI” bundles three capabilities with very different risk:

  1. Generating. The agent writes code or a pull request. Nothing changes until something acts on it, so your existing review and CI still apply.
  2. Executing. The agent calls tools and APIs that change real systems. Its risk is whatever its identity is allowed to do, not what the task needs.
  3. Authorising. Deciding whether an action is allowed. This must never sit with the agent. If an agent decides its own action is permitted, by interpreting a rule in its prompt or by holding a credential that makes approval optional, you don’t have a control. You have a suggestion. The same goes for changing its own limits: an agent must not be able to grant itself a role, create a new token, or edit the policy and configuration that govern it. Otherwise every other control is one step away from being bypassed.

That leads to one principle:

Don’t give an agent a capability that is only safe if it correctly interprets and obeys its own instructions.

There’s a second reason this matters. Agents read untrusted input all day: READMEs, tickets, web pages, tool output. Any of it can contain instructions. A rule in a prompt can be overridden by other text in the prompt. A permission can’t.

In practice, on Azure for example:

  • The agent gets its own identity, not a developer’s token or a shared admin key, and its service connection has no role on the production resource group.
  • High-impact deployments need an approval check in the pipeline that the agent can’t satisfy or edit, and approvers can’t approve runs they triggered themselves.
  • Critical data carries a delete lock the agent’s identity can’t remove, and backups live in a different account, so whatever can delete the data can’t also delete the way back.

None of that depends on what the agent understood. It holds even when the agent is wrong.

Prompt rule versus system limit Left: an agent on a staging task uses an API token that reaches production and deletes the production database and its backups; only a prompt rule stood in the way. Right: the agent's identity is scoped to staging, its path to production is blocked because it has no role there, production changes go through a pipeline approval a human gives, and backups live in a separate account. LIMIT IN THE PROMPT What happened at PocketOS AI agent routine task in staging Prompt rule: no destructive git commands A request, not a guardrail API token it found reaches production Production database and its backups Deleted in one API call Safety depended on the agent obeying its own rules. LIMIT IN THE SYSTEM The same mistake, designed for Agent identity scoped to its job Staging allowed blocked: no role Production delete lock on data Pipeline approval a person, not the agent Backups separate account Holds even when the agent is wrong.
Same agent, same mistake. On the left, safety depends on the agent obeying a rule. On the right, the system limits what it can reach.

Four questions, not four tiers

Sorting agent actions into tiers by type is too neat. A staging deploy isn’t safe if staging shares credentials with production, and not every production change needs a person. Ask four questions instead:

  • Impact: what’s the worst credible consequence?
  • Scope: which systems, customers or records could it touch?
  • Reversibility: can it be reliably undone, and has anyone tested that?
  • Authority: what could the agent’s credentials do, not just what this task needs?

Least privilege limits how much can go wrong. It doesn’t make an action correct: an agent can still update 50,000 records when it meant to update 50, entirely within its permissions. So where impact is severe, prevent it with validation and execution limits, such as a cap on how many records one run can change. Where it’s moderate, detect it quickly with monitoring. For everything, make sure you can recover, with restores you’ve actually tested.

Whether a person needs to decide depends on three things: the impact, how uncertain the change is, and how mature your automated checks are. Rollback lowers the stakes, but it doesn’t remove the need for judgment. You can roll back a change after it has already charged customers twice. A routine dependency bump and a first-of-its-kind data migration carry very different uncertainty, even with identical permissions.

Action Risk Main controls Human decision?
Deploy to staging Low, if isolated from production Separate identity with no production reach No
Deploy a customer-facing change Medium, usually reversible Pipeline gate, canary, automated rollback Yes, until automated checks are proven
Change production infrastructure High, partly reversible Reviewed infrastructure-as-code plan, scoped role Yes, enforced by the pipeline
Delete production data or backups Severe, may be irreversible Agent identity can’t do it; isolated backups Not the agent’s call at all

Both incidents happened in the last row. Neither needed a better approval step. They needed an agent that couldn’t reach that row.

Applying it to my own work

I’m designing automation for our environment refresh: back up the production content database, restore it into a lower environment, deploy the branch, publish content and rebuild indexes. Today every step is manual.

The first design question wasn’t “where does a human approve?” It was “what can the orchestrator’s identity trigger?” The refresh reads from production. If the identity that starts the backup pipeline can start any other pipeline, a refresh tool becomes a production change tool with extra steps. So it gets an identity that can run the refresh pipelines and nothing else.

The four questions also removed a manual step. Publishing into a lower environment felt like something a person should do, but no customer sees it and a bad run is fixed by rerunning the refresh. So it’s automated. The person moves to the end: a quick check that the environment actually works before testers start on it, until the automated smoke checks have earned trust. That’s a judgment about fitness for use, not a safety gate.

What the human is actually for

Permissions answer can this happen? They can’t answer should it? A change can pass every test, sit well within the agent’s permissions, and still be wrong in a way only someone who knows how production behaves would spot.

That’s the job, and it’s why my rule for anyone approving AI work is: don’t ship anything you can’t define and can’t explain. If you can’t say what a change does, what it touches and how to undo it, you’re not approving it. You’re just passing it along. And if the honest answer to “how do we undo it?” is “we can’t”, that action belongs behind a permission, not an approval.

The question to ask about every agent

Every agent should have a named owner who can answer one question:

What’s the worst thing this agent’s credentials allow, and what, outside the model, stops it?

If the answer is “it knows not to”, that’s a request, not a guardrail.