Agent Loop: Reliability & Verification · Part 7 of 7

Testing AI Agents With Independent Agents

Abstract illustration of independent agents following separate paths through shared evidence

“Ask another agent to review it” is becoming a standard answer to the reliability problem. Sometimes it helps. Sometimes it produces several confident copies of the same mistake.

The difference is methodology.

In the research behind this series, independent agents included Codex, Claude, OpenCode, and local-model review runs. They were useful not because a particular brand is automatically objective, but because separate agents can expose different failure boundaries—if the experiment prevents them from sharing an unexamined conclusion.

Independence is not a label. It is a property of the path from input evidence to output judgment.

Independent reviewers following separate paths through the same evidence

What independence should mean

For an agent reviewer, independence has at least four parts:

  • the reviewer receives the same bounded evidence bundle rather than an evolving shared conversation;
  • the reviewer has no access to another reviewer’s conclusion before producing its own;
  • the reviewer’s tool calls and outputs are recorded separately;
  • the scoring rubric is applied after the outputs are collected.

Two agents in the same long-running chat are not independent merely because they have different names. Two agents that both read an earlier reviewer’s summary are not independent evidence of the summary’s correctness.

The goal is not to create an artificial contest. It is to make disagreements inspectable.

A practical test layout

I use a five-stage layout.

1. Freeze the question

Write the target claims before running reviewers. Give each claim an ID. Do not let the reviewer redefine the question while answering it.

For the Axum case, the claims included a purported list of removed types, an assurance that no code changes were needed, and an exact TypedHeader import path. Negative controls were included as well: claims that the supplied evidence directly supported.

2. Freeze the evidence

Build a small evidence bundle from primary sources. Preserve source identity, version, excerpt boundaries, and hashes. The reviewer should see the exact text being evaluated—not a citation that it must retrieve again.

This prevents a reviewer from quietly changing the experiment by finding a newer page, a different version, or a familiar answer from prior knowledge.

3. Separate roles

One agent can collect evidence. Another can draft. A third can review the draft. But the reviewer should receive the evidence bundle and target answer as explicit inputs, with instructions not to retrieve or rely on prior knowledge.

In the Context Mode investigation, the agent that explored the source-resolution behavior was not treated as proof that its proposed upstream contract had been accepted. In the ExecutionJournal work, the code review status ACCEPT was reported as a review result for the test change, not as a universal guarantee about recovery. The later failure-boundary audit reported 1,312 passed across the defined matrix, but that remains evidence about the tested checkout—not a universal reliability guarantee.

4. Add negative and insufficiency controls

A useful reviewer must do more than find errors. It must leave supported claims alone and abstain when the evidence is incomplete.

The Axum fixture included an insufficiency control asking whether the evidence proved the exact TypedHeader import path. The safe answer was abstention. A reviewer that “corrects” the target by inventing the path has failed even if it detected that the original answer was risky.

5. Score the contract in pieces

Record detection, classification, evidence attribution, uncertainty, correction quality, and false-positive controls separately. Keep parse errors, timeouts, and truncated generations outside the reasoning score.

The current review runner also validates the response envelope before scoring it. Exact finding/control IDs, required fields, non-empty uncertainty, and explicit abstention for insufficiency are part of the test contract. A response that is valid JSON but violates that schema is a protocol failure, not a reasoning score.

This makes disagreements useful. One agent may detect F1 but call it “unsupported” instead of “contradiction.” Another may classify it correctly but cite the wrong excerpt. Those are different engineering problems.

What multiple agents can reveal

Independent agents are particularly useful for finding:

  • claims that sound plausible but are not licensed by the evidence;
  • source-version confusion;
  • overconfident exact API details;
  • retrieval strategies that repeatedly return the same text;
  • corrections that introduce a new unsupported assertion;
  • rubric weaknesses that binary pass/fail scoring hides.

They are less useful when the task is underspecified, the evidence is mutable, or the runs share hidden state. More outputs then create the illusion of consensus without increasing information.

Repeated behavior must also be counted honestly. If a local model returns five byte-identical reviews, that is one observed behavior repeated five times. It is not five independent votes.

The limits

This methodology does not produce a universal reliability percentage. A handful of agents and a single evidence bundle cannot rank models generally. Different agents may also share training data, familiar conventions, or the same blind spot.

Independence reduces correlated failure; it does not eliminate it.

Nor does a reviewer replace tests. The crash-recovery article in this series rests on an executable two-process regression, not on an agent saying that recovery “looks correct.” The Context Mode article rests on a real MCP differential reproduction, not on a reviewer’s confidence about an error contract.

The strongest workflow combines independent judgment with executable or protocol-level checks wherever possible.

A small operating checklist

Before trusting an independent-agent review, ask:

  1. Did every reviewer see the same immutable evidence?
  2. Were unsupported claims and negative controls included?
  3. Were the reviewers prevented from seeing each other’s conclusions?
  4. Were retrieval, parsing, and reasoning failures separated?
  5. Can every correction be traced to the evidence, or does it abstain?
  6. Were duplicate outputs counted as repeated behavior rather than independent confirmation?

If the answer to these questions is yes, multiple agents can be a powerful diagnostic instrument. If not, they may only make an unsupported conclusion sound more social.

The point of independent agents is not to manufacture consensus. It is to create several auditable paths through the same evidence—and to learn exactly where those paths diverge.

Related: Evidence-to-Claim Verification: Why LLM Reviewers Still Fail, The Crash Window, and A Successful Search That Never Searched.

Tags: AI Agents, Testing, Evaluation, LLM, Software Engineering

← Back to all posts