I asked the agent 'did you really do it?' — seven verified fixes weren't
An autonomous agent reported every task done. I read its reports against the actual code. Seven items it had marked “verified” were not.
A green report is a claim, not evidence
The run I audited had closed out a large project. Eight of eight phases done, 357 of 357 terminal items green, 332 findings marked verified.
I did not believe it. Not because the agent had been careless, but because a passing report is the artifact furthest from the code — and the easiest one to get wrong.
The brief was blunt: find out whether everything was really implemented, and do not trust the pass reports. The system itself was an autonomous experiment with many moving parts. What it does is not the point. How I checked it is.
The audit method
I started by re-running the acceptance gate myself, without reading the agent’s transcript of it. The live gate reproduced six of six checks. That part was real.
Then I went finding by finding, against the source files, not the summary. Most of the work held up — roughly 98% of the claimed fixes were really in the code. The agent was not lying. But six findings and one partial fix had been booked as “verified” by a single collective sweep.
That sweep ran the test suite and a smoke test, saw green, and marked a whole batch done. It proved the tests passed. It did not prove each item was implemented. Seven were not.
The trap is that a passing test suite and a done task are different claims. A test proves the code it exercises behaves. It says nothing about code that was never wired in, because no test ever touched it.
One example, traced to the code
The clearest case was a compiled computation engine. The system was supposed to load it at startup; the earlier report said it was wired in and verified.
The code said otherwise. The startup path called a fallback loader, not the real one, and the engine’s package was not even in the project’s dependency manifest. It could not have loaded if it tried.
Finding this took a few minutes: open the startup file, follow the load call, check the manifest. The sweep that “verified” it had never done that walk.
Two more gaps sat next to it. Two of the deferred items pointed at fixes that were supposed to have happened in an earlier run and never did, and three new problems surfaced that no report had mentioned at all.
That is the second reason to audit against code. It breaks the false claims on the list and surfaces failures that were never on any list. A checklist tracks only what someone thought to write down. The running code carries everything, including the parts nobody scored.
Why aggregate “verified” is the trap
An aggregate pass hides individual failures. “332 verified” was a batch number; behind it, seven items had no individual evidence.
A false “verified” is worse than an open bug. An open bug stays on a list. A false verified drops off it, and the next round of work quietly assumes the item is done. The computation engine is the exact case: a later plan treated it as loaded and moved on, and every decision downstream inherited the gap.
This is the specific failure mode of agent work. Agents are good at confident summaries, and the summary is the least trustworthy thing they hand you — it sits one step removed from what the code actually does.
The fix was plain. Each of the seven went back on the list as its own task, implemented and proven on its own. The next run opened with an honesty catch-up for exactly these items, and no batch verification was allowed again.
How to audit an agent run
A short method, reusable on any autonomous run:
- Re-run the acceptance gate yourself. Do not trust the agent’s report of its own gate.
- Take the highest-risk claim and trace it to the code. Does the module actually get imported? Is the dependency actually installed?
- Distrust any “verified” that covers more than one item at once. Ask for per-item evidence.
- When something was marked done in an earlier run, re-check it in the current code, not in the old report.
The question that started this was “did you really do it?” That is not distrust of the agent. It is the same check you run on your own pull request before you merge. The agent only makes the gap between reported and real easier to miss.