How do you know when an AI judge is worth trusting?
An AI judge disagreed with human reviewers. I designed a feedback loop around approved criteria, fair replay, human calibration, and visible evaluation coverage.
The initial problem
In a feedback-to-delivery system, an LLM judge scored work much more harshly than human reviewers did. An evaluation score looked authoritative, but the scales were different. The disagreement did not tell me which evaluator was correct.
I needed a way to make those scores useful for decisions about the agent. Treating every low score as poor work would hide the possibility that the evaluation itself was wrong or incomplete.
How I approached it
I treated the disagreement as a measurement problem and started with acceptance: what observable change should appear in the output or code diff? Those criteria need human approval before they can serve as the standard. Draft criteria remain drafts.
I then considered how to replay a request fairly. The generation prompt needs a time boundary before the first agent output. Otherwise, a replay could give the agent knowledge of the answer that was unavailable during the original request. Human corrections and final implementations belong in scoring, rather than in the generation prompt.
The benchmark also needed corrected and unsuccessful cases alongside accepted ones. Testing only successful work would create an easier problem than the operation actually faces.
How I solved it
I documented a loop connecting production feedback with offline replay. It pairs human labels with judge results so that their agreement can be inspected, and records versions, generation environments, and repeated runs for each comparison.
When scales differ, rank correlation can help show whether humans and the judge order results similarly. Sample size and review coverage still matter: a small set of paired ratings cannot establish reliability. The judge may become useful for tracking trends while humans retain individual acceptance decisions.
I also kept unknown results separate from failures. A skipped verification or missing rating is unmeasured. Recording it as failure would turn an environment limitation into a claim about code quality. Recording evaluation coverage makes it easier to see whether only the easy examples received ratings.
What the evidence supports
This is an evaluation process, not evidence of a production quality lift. A single run is too fragile for a confident version comparison, and a small calibration sample cannot establish a trustworthy judge.
The documented resolution is a clearer basis for the next decision: approved criteria, replay without answer leakage, paired human labels, repeated runs, and visible coverage. Those records help distinguish a weak agent from a failed evaluation harness.