Judge the Fact Before You State It
An LLM's attention is cheap and a human's is scarce. Spend the cheap one to protect the scarce one.
When an AI assistant states a fact you’ll act on — a date, a number, a claim about what someone said — the cost of it being wrong is borne by you, later, often in front of other people. The model is confidently fluent whether or not it’s right, and fluency is exactly what makes a wrong fact dangerous.
The cheap fix I keep coming back to: before shipping a load-bearing factual answer, route the draft through a separate model whose only job is to judge it against the actual sources. Not the model that wrote the answer grading its own work — a different pass, pointed at the evidence, returning a verdict: pass, revise, or block.
The economics are the whole point. A human’s attention is the scarce resource; a judge model’s attention is nearly free. So substitute the cheap one for the expensive one. Instead of me carefully fact-checking every claim, a judge checks it in a minute and only escalates the ones that don’t hold up. That substitution is the architecture.
A few things that made it work in practice: the claim you submit to the judge and the claim you actually ship have to contain the same facts — audit “Thursday the 26th” and then ship “the 26th” and you never checked the Thursday. And an infrastructure failure in the judge is not a verdict; “the judge didn’t run” means retry, never “looks fine.”
The line for what needs judging: would you repeat it externally, or base a decision on it? If yes, judge it. If it’s reasoning or opinion, label it as such and move on.