How to Check an AI Judge Before Trusting Its Test Review

TL;DR: A model can review a generated test and still miss the reason it would pass when the feature is broken. Start with a small set of examples whose expected reviews a person has documented, ask the judge to evaluate them without those labels, and inspect the dangerous disagreements: cases the judge marks “accept for human review” that your reviewer marks “request changes” or “blocked by missing evidence.” Use the judge’s decision as an aid to review, not as proof that the application works.

An AI assistant writes a browser test for a checkout button. A second model says the test looks good. The team merges it, then a customer discovers that clicking the button never saved the order.

The test may have verified that a confirmation appeared without checking whether the order actually existed afterward. The judge saw plausible test code and gave it a good review. Neither result established the outcome the customer needed.

That scenario is illustrative, not a result from a Samelogic experiment. It shows a useful distinction: a test checks an application, while a judge checks an answer or artifact against criteria. The judge can help a person review a test, but its confidence is not an independent observation of the application.

Why AI Test Review Needs Its Own Check

In its 2026 State of AI in Quality Engineering report, TestMu AI surveyed members of its worldwide community online between late July and early October 2026. Each response describes the respondent's own team; respondents were not a sample of TestMu customers. The sample leaned toward organizations based in India and toward SaaS companies. The report was published November 2.

The vendor reports that 32% of teams that evaluate AI output use LLM-as-judge. In a separate comparison, 29% of teams using it said they regularly merged AI output they did not fully understand, compared with 15% of “other teams,” which the report does not further define. These are self-reported responses and an association, not evidence that model judging caused the gap. The report does not establish that these respondents were all judging generated browser tests, specify subgroup sizes for the comparison, or measure judge accuracy or test-review quality.

The useful question is not whether an AI judge is good or bad in general. It is whether your judge can identify the review mistakes that matter for your tests, and what happens when it cannot.

The judge-review method is our recommendation, informed by the survey above. This is source-backed analysis, not an executed experiment. That distinction refers to the judge-review method, not the product QA described later.

Separate the Test, Its Oracle and the Judge

Three checks often get compressed into one conversation:

• The test runs a sequence and makes assertions about what happened.

• The test oracle defines the expected result and how to compare it with observed evidence. For an order, it might require one saved order and compare that expectation with an independently read record, not merely the message shown after a click.

• The judge evaluates the test or its output against instructions. It can identify a missing assertion or explain a suspicious result, but a text-only review cannot by itself verify that an order was saved.

A judge can be useful even when it does not execute code. It might flag that a test only checks the confirmation message, ask for a saved-record check, or point out that the wrong account could have been updated. But those are review suggestions until someone checks the relevant evidence.

Build a Small Review Set Before Delegating

Choose a browser workflow your team understands. Write a short requirement and a set of candidate tests with different strengths and mistakes. Have a qualified reviewer document the expected assessment and the reason before showing those examples to the judge.

For a checkout example, the requirement might be: an authorized user places one order, the order remains after reload, and no duplicate is created. Use synthetic accounts and data in an authorized test environment.

These are designed examples, not measured results or a complete checkout test suite. The expected labels depend on the stated requirement and evidence. If a reviewer has not checked whether a proposed independent read is available and trustworthy, record the uncertainty and missing evidence alongside the provisional label; use “blocked by missing evidence” when the assessment cannot be made without it. Do not treat provisional labels as ground truth.

Include more than obvious failures. Add examples that pass for the wrong reason, incomplete requirements, environment failures, and genuinely sound tests. Record the browser, data and permission assumptions that affect each assessment. Remove secrets and personal data from prompts, logs and review materials before sending them to a judge. Keep sensitive customer information out of the set unless its use is explicitly authorized.

Ask the Judge to Explain the Decision

Give the judge the same requirement, candidate test and relevant evidence the reviewer assessed, without the expected label. For example, provide evidence that the order readback is unavailable before expecting the judge to identify that constraint. Ask for a structured answer:

1. Decision: accept for human review, request changes, or blocked by missing evidence.

2. Evidence: the exact assertion or result that supports the decision.

3. Missing checks: requirements the test does not establish.

4. Uncertainty: information the judge cannot infer from the supplied material.

5. Next action: the smallest check a person should make.

Keep the prompt, model and settings with the results so a later comparison is meaningful. A polished explanation is not enough: verify that the cited assertion actually exists and addresses the requirement. If the judge says a test verifies saved state, find the line that does so and check whether it reads an independent source.

Do not let the judge invent execution evidence. If it has only source code, its answer is a review of source code. If it has a real execution log, it can discuss what that log shows; it still cannot infer a saved order from a log that never checks one.

Treat False Approval as a Separate Risk

Compare each judge decision with the documented human assessment. Do not hide disagreements inside one overall agreement percentage.

Here, a false approval is the judge selecting “accept for human review” when the reviewer selected “request changes” or “blocked by missing evidence.” Keep those two kinds separate: the first may include any reason the reviewer requests changes, not just an unchecked requirement; the second reflects missing evidence. Also record the reverse, where the judge requests changes to a sound test. That can waste review time, but it is a different problem from allowing a weak test through.

For each case, retain the expected label, judge decision, supporting evidence, uncertainty and reviewer resolution. Treat the judge's “accept for human review” as an approval. Divide the number of judge approvals of reviewer “request changes” cases by the total number of cases the reviewer marked “request changes.” Separately, divide the number of judge approvals of reviewer “blocked by missing evidence” cases by the total number the reviewer marked blocked. Exclude cases with provisional or disputed reviewer labels from these rates until resolved, and report their count and reasons separately. When either denominator is zero, its rate is not calculable. Report both numerators and denominators alongside any rates. Record all nine combinations of the three reviewer labels and three judge decisions to expose other disagreements, such as a judge blocking a sound test. Keep provisional or disputed labels visibly separate in that record too, and record invalid or missing judge decisions separately rather than forcing them into a label. A handful of examples is a diagnostic starting point, not a reliable production estimate.

A second reviewer should revisit disputed labels when the evidence is ambiguous. If the requirement itself was unclear, fix the requirement and relabel deliberately. Do not silently alter an expected answer to make a judge look more accurate.

Decide What the Judge Is Allowed to Do

Before connecting a judge to a pull-request workflow, agree on its authority:

• Advisory: it posts review suggestions for a person to inspect.

• Escalation: missing evidence or disagreement sends the test to a named reviewer.

• Release decision: a person or a separately validated process decides whether the required checks passed.

A low-risk internal trial may tolerate some unnecessary requests for human review. A judge approving a payment or access-control test without the required evidence has a different consequence. Set the allowed error and escalation path for the workflow rather than adopting a generic confidence threshold.

Recheck the review set when the model, prompt, test framework or requirements change. Add newly discovered mistakes as separate examples, and keep a portion of reviewed cases aside when possible so tuning does not become the only evaluation. If a case requires a real browser run or a read-only order lookup, the team must actually perform that check before calling it verified.

What Changes When Agents Review More Work

If agents generate and review more test changes, the review itself becomes a workflow that needs an owner. A useful record includes the requirement, candidate test, evidence supplied to the judge, decision, disagreement and human resolution. Without those boundaries, a later reviewer cannot tell a model's interpretation from an observed application outcome.

Start now with one high-consequence workflow and a few reviewed examples. Assign someone to own the labels and decide which disagreements always require a person. Expand only after the team understands where the judge helps and where it misses important evidence.

Bring Browser Context to the Human Review

When a generated test or judge response raises a question about a browser bug, Samelogic describes a deliberately started recording workflow intended to provide context about browser steps and affected elements. In an October 31 controlled fake-data test of a 1.10.1 development build, the first action’s captured evidence incorrectly showed the second action’s failure. A November 3 live retest was blocked by test-tool access restrictions, and no corrected build has been verified. We have not established whether the released extension is affected. Do not rely on captured evidence or playback alone to attribute a result to an action; independently check the action and its outcome. The product page says input values are masked by default, but other page content can still appear; it advises reviewing the recording before sharing. As a precaution, remove sensitive content by an appropriate means before sharing; if you cannot do so safely, do not share the recording. The page also says anyone with a public link can open a public recording, and making it private again turns off public-link access. Samelogic is not a model judge or an independent order-state oracle, and a recording alone does not establish that a fix worked. Explore the browser bug reproduction workflow and decide which independent outcome check your reviewer still needs.

Sources

• TestMu AI, State of AI in Quality Engineering 2026, published November 2, 2026. The survey and its stated methodology are the source for the reported comparison; the worked review design above is our recommendation.