A score implies a measurement. Run the instrument twice before you believe it.
Somewhere in your GTM stack, an AI is scoring something. A lead gets a fit number. A reply gets classified interested or not interested. A draft gets graded against the brand bar. A call gets a quality rating. Each of those numbers reads as a measurement, and decisions route on them automatically.
Magnetiz runs judges like these across its own fleet, and on 2026-09-09 we put one of them under the simplest test an instrument can face. Same draft, same rubric, nine runs. Consecutive runs on identical input returned opposite verdicts. Across all nine, four said publishable and five said kill. The judge also reports a confidence figure, and it ranged from 0.64 to 0.76 with no relationship to which verdict it accompanied.
Nothing was broken. This is what large language models do, and the discipline that AI engineering built around it has mostly not crossed into GTM. Here is the version that fits your stack.
Where are LLM judges already inside your GTM motion?
Anywhere an AI produces a score, a grade, a label, or a pass.
Lead scoring and account fit. Reply sentiment sorting the inbox into interested and everything else. Content gates that hold a draft below a quality bar. Call scoring, deal risk flags, routing rules that key off any of these. If a model's judgment feeds a threshold, you are running a judge.
Most of these arrive inside other products, so nobody on the team thinks of them as instruments the company now owns. The vendor's UI shows a number, and a number looks finished.
Why does a single verdict feel like a measurement?
Because GTM inherited its instincts from dashboards, where the same query returns the same number.
Revenue reporting trained everyone to treat a displayed figure as a fact. Query it again, same answer. LLM judges break that inheritance quietly. The figure displays with the same font and the same decimal point, and underneath it is one sample from a distribution.
The decimal is the trap. A 7.4 looks like it was measured to a tenth of a point. Our nine-run test says the honest error bar on a single run can span the entire distance between ship and kill.
How do you test your own scorer?
Ten minutes. One item, five runs, count the agreement.
Take one item your stack scored this week, ideally one that landed near a threshold. Run the same scorer on the identical input five times, nothing changed between runs. Count how many runs agree on the action the score triggers.
Five out of five, and the instrument deserves more trust than this piece implies. Three out of five, and every past decision that leaned on one run near the boundary was a coin flip wearing a decimal. You cannot know which case you are in without the test, and almost nobody has run it.
What do you do when the verdicts split?
Run three, take the majority, and record all three.
Majority-of-three is the operating pattern our fleet moved to after the measurement. It cuts variance at triple the inference cost, which for judge calls is usually pennies against the cost of a wrong kill or a bad-fit meeting.
Recording all three matters as much as the vote. A three-to-zero and a two-to-one are different facts. The unanimous verdict acts automatically. The split gets flagged and lands with a human, because majority voting narrows the noise without making the instrument precise.
Which part of the output can you actually trust?
The findings that reproduce, more than the verdict label.
When our judge flags the same specific defect in two or three runs, that defect is real, whatever the overall verdicts said. A flaw named once across three runs is noise, and acting on it wastes a rewrite. The label wobbles. The reproduced reasons hold still.
The same logic scales up. One draft judged nine times told us less than four different drafts all failing in the same direction, which is the aggregate pattern that actually set our content direction that week. Direction across many items is the strong signal. Any single verdict near a boundary is the weak one.
Do rating scales make this worse?
A 1-to-10 scale invites the instrument to be confidently vague. Binary axes force it to commit.
Magnetiz scores evals as binary checks per axis. Does the draft make an attackable claim, yes or no. Does the lead match the segment, yes or no. Each axis reproduces or fails on its own, so you learn which parts of the judgment are stable instead of watching one blended number wobble.
A threshold applied to a blended decimal, pass at 7.0 and fail at 6.9, is spurious precision stacked on an unreproduced score. The gate feels rigorous in the exact proportion that it is not.
The pattern across the discipline
None of this says stop using AI judges. Our fleet runs on them, and the alternative at GTM volume is a human reading everything. The change is smaller and cheaper. Treat every scorer as an instrument on probation. Test it for reproducibility before its threshold acts on anything, vote it when it wobbles, escalate its splits, and trust its repeated findings over its labels.
A score you cannot reproduce is an opinion with a decimal point.
Frequently asked questions
How reliable are LLM-as-judge scores?
Less reliable than their formatting suggests, and the spread is measurable. In a nine-run Magnetiz test on identical input, a content judge returned publishable four times and kill five times, with reported confidence between 0.64 and 0.76 either way. Reliability varies by task and rubric, so the honest answer for your stack comes from running your own scorer several times on identical input and counting agreement.
How do I test an AI scoring system for consistency?
Pick one recently scored item, ideally near a decision threshold, and run the scorer on the identical input five times with nothing changed. Count how many runs agree on the action the score would trigger. High agreement earns the instrument trust. Frequent disagreement means single runs near the boundary have been deciding by coin flip, and the system needs majority voting before its scores act on anything.
What is majority voting for LLM judges?
Running the same judge three times on the same input and letting the majority verdict stand. It reduces variance at triple the inference cost, which is usually trivial next to the cost of a wrong decision. Record all three verdicts rather than just the winner, treat unanimous results as actionable, and route two-to-one splits to a human, since majority voting narrows noise without making the judge precise.
Should AI evals use binary scoring or a 1-10 scale?
Binary checks per axis reproduce better and localize failures. A blended 1-to-10 score hides which part of the judgment wobbled, and a pass threshold written to one decimal implies precision the instrument has not demonstrated. Magnetiz scores each axis as a yes or no, so a re-run shows exactly which axes are stable, and the axes that flip get rewritten or removed from the rubric.
Can I trust AI lead scoring?
Trust it after it passes a reproducibility test, and trust its patterns before its individual numbers. Re-score a handful of accounts on identical input and check whether the scores would route each account the same way twice. Aggregate direction, like a segment consistently scoring above another, is far more robust than any single account's number sitting near a threshold.
When is a single AI verdict still enough?
When the margin is wide or the stakes are low. A verdict far from the threshold survives instrument noise, and a low-cost reversible action does not warrant triple-running. Near a boundary, on a kill decision, or on anything expensive to undo, one run is one sample, and sampling again costs pennies against deciding wrong.