If you know the steps, you don't need an agent
An AI agent picks up a failing browser test, reads the code, makes a change and reports back:
Fixed.
What does that mean? The build passes? The test passes? Every other test still passes? The page still works for a keyboard user? The change does what the business actually asked for?
Those are very different levels of confidence. Before we get to how to tell them apart, there’s an earlier question most teams skip: should an agent have been doing this at all?
Agents are being added to problems that were never uncertain
Agents are easy to add now, so they’re turning up in places where the work was never in doubt. A pipeline that runs tests and posts a report becomes “an agent that monitors quality”. A release checklist becomes “an agent that manages releases”.
Anthropic’s guide to building effective agents draws the line I find most useful. A workflow follows a path someone defined in advance. An agent decides its own next step as it goes. Their advice is to use the simplest option that works, and only add an agent when the flexibility is worth the cost.
That cost is easy to underestimate. A workflow does the same thing every time. You can test it, read it, audit it, and when it fails, the log tells you which step broke. An agent adds judgement, and with it, variance: two runs on the same input can take different paths. Sometimes that’s exactly what you need. When the steps are already known, it’s risk you’ve added for nothing.
The rule I use
Automate the predictable. Use AI for the uncertain. Keep people for the accountable.
The question that sorts every step is simple: do we already know what should happen here?
Here’s how it plays out on a typical deployment with browser tests:
| Step | Do we know the steps? | Who does it |
|---|---|---|
| Deployment finishes, tests start | Yes | Pipeline |
| Run browser and accessibility tests | Yes | Pipeline |
| Collect results, screenshots and logs | Yes | Pipeline |
| Work out why an unfamiliar failure happened: the app, the test or the data? | No | Agent, worth trying |
| Propose the smallest change that fixes it | No | Agent, worth trying |
| Decide the change is safe to merge | It’s a judgement with consequences | A person |
Notice where the agent sits. It’s not running the process. It’s at the one point where today’s process stops and says an engineer needs to look at this.
An example where an agent adds nothing
Take an environment refresh: copy a known dataset into a test environment, deploy a branch, publish content and rebuild the search indexes. In many teams it’s still done by hand, step by step.
It’s tempting to describe this as “an agent that refreshes environments”. It shouldn’t be one. Every step is known, the order is known, and the definition of success is known. That’s a pipeline. Adding an agent would turn a process that works the same way every time into one that probably works the same way every time, and in a pipeline that moves data between environments, “probably” is a regression.
The interesting part for AI is the edge of that process. When a refresh fails in a way nobody has seen before, someone has to read the logs, compare them with the last good run and work out what changed. That’s investigation, and it’s where an agent can genuinely save time.
Where an agent earns its place
Take a failing accessibility test after a deployment. The pipeline has already done the deterministic work: it ran the scan, captured the violation, the screenshot, the page and the build that introduced it. That’s the signal and the evidence.
Today, the next step is a person reading a report, creating a ticket and starting to dig. An agent can start that digging immediately: find the component that renders the failing element, check how the same problem was solved elsewhere in the codebase, and prepare a draft pull request with the smallest change that fixes it.
What it shouldn’t do is change production, merge its own work or decide it’s done. The pull request is the boundary. The agent’s change goes through exactly the same checks, reviews and approvals as anyone else’s. AI shortens the distance between the signal and the investigation; it doesn’t skip the system that decides what ships.
The failure mode to design for
Back to “Fixed.”
When a coding agent is told to make a failing test pass, the cheapest way to succeed is often to change the test, not the code. Loosen an assertion. Add a wait. Skip the check that’s failing. The test goes green, the report says fixed, and the problem is still there.
It isn’t malice; it’s optimisation. The agent did what it was asked, against the only measure it was given. Which leads to the most important design rule in this whole area:
The checks that judge a change must be checks the change can’t edit.
In practice:
- Protect the tests. Changes to test files need a person’s review, even when the rest of the pull request is routine. Most repositories can enforce this with code owners on the test folders.
- Run the agreed tests, not the changed ones. Verify the change against the test suite from the main branch, so a pull request can’t quietly redefine what “passing” means.
- Define “fixed” before the agent starts. “The accessibility violation on the checkout page is gone, no new violations appear, and the existing browser tests still pass” is checkable. “Fix the test” isn’t.
The agent can produce the change. The engineering system has to produce the evidence.
The line moves, in one direction
This boundary isn’t fixed. When an agent has investigated the same kind of failure ten times and the fix is always the same, that’s no longer uncertainty. It’s a known step that hasn’t been written down yet. The right move is to turn it into a rule, a check or a pipeline step, and take the agent out of that loop.
Used well, agents work like scouts. They explore the unknown parts of a process, and the parts they make known get handed back to deterministic automation. Over time the uncertain zone gets smaller and the predictable zone gets bigger, which is exactly the direction you want a delivery system to move.
The question to ask first
Before adding an agent to any process, ask one question: do we know the steps?
If yes, write the pipeline. It’ll be cheaper, faster and easier to trust.
If no, an agent may be the right tool, as long as it works inside a system that checks its work, with tests it can’t change and a person deciding what ships.
Generating code is getting cheaper. Knowing which problems need judgement, and building the system that proves the answer is right, is the part that’s getting more valuable.