Quick conclusion
Between late August and this week I ran three pre-registered experiments to find a cheap model that could fact-check my drafts. Thirty-three API calls, $0.29558, two candidate checkers, six more models in a side test. I selected exactly zero of them.
The pass line was set before the first run: recall of at least 0.75 and precision of at least 0.5, averaged over three cases. The best recall I ever measured was 0.389. Both candidates failed the line twice, in August and again in September, so I am closing this line.
That sounds like a dead end, and the cost result is one. The useful part is what the three rounds did to each other. Round two looked like a triumph until round three explained it, and round three only happened because I asked why the triumph had no variance in it.
What I was actually trying to do
The goal was never research. It was routing.
I write with an expensive model and I check with the same expensive model, which means the checking work competes with the writing work for the same budget. Back in July I wrote that cheap models are cheap, but reviewing their output is not. The obvious next move was to flip the roles: let the cheap model do the checking, where the output is short and structured, and keep the expensive one for the part that needs a voice.
In August I tested that directly. I gave two cheap models a draft and the research notes it was written from, told them to find the claims the notes did not support, and scored them against an answer key I had written in advance. That one became The Second Model Marked the Fiction as Sourced, and near the end it left a hypothesis:
The fix is not a different model. It is a different question: not “does the source contain a corresponding passage” but “does this match what the source says, including reversals and wrong numbers.”
I also wrote, in the same article, that I had not run that test yet. This article is me running it.
Two things are worth naming before the numbers. First, every experiment was pre-registered: the scoring rules and the answer key were frozen in a file before any call was made, and in rounds one and three I wrote down my predictions as well. None of it was changed afterward. Second, rounds one and three use the same two checkers, deepseek/deepseek-v4-pro and google/gemini-3.7-flash, so those two rounds compare directly. Round two used a different set of six cheap models, and neither candidate was among them.
Round 1: the checker missed almost everything
August 23, 2026. Three drafts, each written by a different cheap model from the same source notes, each carrying errors I had cataloged by hand. Ten cataloged errors in total. Two checkers, one pass per draft.
Recall came out at 0.333 for the first checker and 0.167 for the second, against a line of 0.75. Precision was 0.278 and 0.333.
The number that stuck with me was not the average. It was one specific item. A draft had put an incident on the wrong day: it said the day before a July 4 entry, and the source notes dated it July 1. The checker wrote the correct original date in its own evidence field, and then marked the claim as sourced. It had the discrepancy in its hands and filed it as a match.
My reading at the time was that the question was wrong. I had asked “is there a corresponding statement in the source?”, which a plausible-looking sentence can pass. I had not asked “does the value, the direction, and the condition all match?” That reading is what round two was supposed to test.
Round 2: I rebuilt the test and everybody got a perfect score
September 20, 2026. New test rig, six cheap models, a stricter question, a three-way verdict instead of yes-or-no, and structured output I could score automatically. I extracted the claims myself and handed them over one by one, so the model no longer had to find them first.
Result: 204 of 204 judgments correct, across 18 runs. No misses, no false alarms, no schema failures, no empty responses. Every model passed everything, including the hardest case, where the draft had been rewritten to be internally consistent so that only a comparison against the source could catch it. The cheapest model in the set quoted the source value back at me correctly. Across the whole round, the most expensive model in the set cost about 35 times the cheapest, and produced no better output.
For about an hour I wrote this up as a confirmation of the hypothesis from August. Then I looked at the shape of the result. Six models, six perfect scores, no variance at all. A test where the cheapest and the most expensive option are indistinguishable is not measuring the thing it was built to rank. It saturated.
And I had changed six things at once: the question, the verdict scale, the output format, the extraction step, the material being checked, and the models themselves. Any of them could own the result. I retracted the confirmation the same day.
Round 3: change the question, and only the question
So I ran the round I should have run second. Same inputs as August, same drafts, same answer key, same two checkers, same scorer. The only edit was the checker’s instruction file: the new question, the three verdict labels, and a list of the error types to look for. Against the August file the diff deleted six lines, all of them belonging to the old question and the old two-way verdict. Everything else stayed byte for byte as it was, down to a wording slip I left in place so that I would not add another variable.
I wrote down seven predictions before calling the API, and then ran six passes for $0.10340.
| Checker | Recall, August | Recall, September | Precision, August | Precision, September |
|---|---|---|---|---|
deepseek/deepseek-v4-pro | 0.333 | 0.389 | 0.278 | 0.711 |
google/gemini-3.7-flash | 0.167 | 0.278 | 0.333 | 0.583 |
Recall moved by 0.056 and 0.111, against a line of 0.75. Both checkers failed again, by roughly the same distance as before.
Precision is the other half of the table, and it more than doubled for both. The number of items the checkers flagged as wrong went from 9 to 18, and the number of false positives among them went up by one.
So the perfect score in round two was not the question. It was the difficulty. Handing over pre-extracted claims and comparing two nearly identical documents is a much easier job than searching a source for something that is not there, and that is where the 204 out of 204 came from. That was the reading I had pre-registered, and the control run confirmed it.
I also have to be honest about the limit of this round. My new instructions did two things at once: they changed the question, and they listed the types of error to look for. The precision gain cannot be attributed to the question alone, because I did not separate those two edits. I got five of my seven predictions right. The miss that matters is the one I would have staked the round on: I predicted precision would fall as the checkers flagged more items, and instead it more than doubled.
What the numbers say, and what they do not
Averages hid the interesting behavior. Three per-cell results are worth more than the table above.
The date case came back. The same date discrepancy that both checkers waved through in August was now correctly rejected by gemini-3.7-flash, which wrote that the source says July 1 and the draft does not. And it was waved through again by deepseek-v4-pro, which quoted the correct source date in its evidence field and then marked the claim as matching. Same instructions, same verdict scale, same explicit slot for date mismatches. One model used it, the other repeated August exactly. Instrument design did not guarantee the fix. The model still mattered, in the one place where I had expected the instrument to take over.
Volume did not predict accuracy. One cell went from 30 flagged items to 49 and still detected none of the three planted errors. Another went the other way, from 32 items down to 20, and its detections fell from four out of four to two out of four. The new question changes which sentences the checker puts on the table. It does not reliably put more of the right ones there.
And one failure type is invisible to both metrics. In one pass, a checker quoted a sentence as coming from the source, and marked the claim as supported. The sentence exists only in the draft. Precision measures the quality of the rejections, so a fabricated confirmation is not counted anywhere. I did not find that one by scoring. I found it by reading the raw output.
What I cannot say from this: that these models have comparable fact-checking ability in general. I measured one narrow band of one task, with one pass per cell.
Note added October 29, 2026. Two details about how I counted. The ten cataloged errors include one item whose source passage contradicts itself, and one that was outside the checker’s scope. Removing the first leaves deepseek’s recall at 0.389 and moves gemini’s from 0.278 to 0.333; neither is near 0.75. And under my pre-registered rule, borderline flags counted as non-matches in precision, which is why I wrote that false positives went up by one. If they counted as matches, precision would be about 0.83 and 0.67. The verdict does not change.
The part I got wrong
Three experiments. Thirty-three runs. $0.29558. Zero models selected.
Look at what rounds two and three actually optimized. Round two rebuilt the question, the output contract, and the extraction step. Round three isolated one of those edits. Neither of them was choosing a model. They were improving the test. The routing decision I started with, which model gets the checking work, was no closer after round three than it had been in July.
The tell was in how I justified round three. I had a real methodological reason, and I used a different one when I wrote the proposal: that the article would be much stronger with a control arm. That is a reason to run an experiment for the blog, not a reason to run it for the workflow. When the stated benefit of the next measurement is a better writeup, the measurement has stopped feeding a decision.
There is a version of this loop that is healthy. Measure, change one thing, measure again is exactly how you get an answer worth having, and I would not give that up. What went missing was the checkpoint: which operating decision is still waiting on this number, and is it still waiting? Mine had quietly been answered in August. Both checkers failed the line, the line was set in advance, and I kept going.
I am not counting the spend as a win, either. $0.29558 is small, but in separate work this year I lost $0.54 and $0.0967 by mechanically re-running calls that had come back empty, instead of stopping to find out why they were empty. The cheap part of cheap models is the invoice.
What I would still hand to a cheap model
The line I now draw is not about price. It is about whether the answer can be checked by something other than me.
Checking whether a draft’s claims survive contact with the source is judgment work, and price tier did not move it. Where the job was easy enough to be scored automatically, a 35x price gap between six models produced identical results. Where the job was hard, both candidates failed the same line twice. Formatting, tagging, converting Markdown to HTML, generating file names, splitting a table: for those, the output can be diffed, validated, or compiled. If it is wrong, something other than my attention notices.
I have not measured that band. I want to be exact about this, because it would be easy to end an article like this with a confident-sounding boundary. Every number in this piece comes from the checking task. The claim that cheap models are fine for mechanically verifiable work is, right now, a hypothesis with no runs behind it. That is the next thing worth measuring, and it needs its own pass line written down first.
FAQ
Does this mean cheap models are bad at fact-checking?
It means two specific models failed a specific pass line twice, on a task that required finding unsupported claims in a draft against a source. In the easier band, where I extracted the claims first, six cheap models scored perfectly. The task boundary matters more than the price tier.
Was your prompt just bad?
That is what round three tested. I held the inputs and the answer key fixed and changed only the checker’s instructions: the question, the verdict labels, and the list of error types. Recall moved by 0.056 and 0.111. If a better prompt is the answer, it is not this better prompt.
Why not run more repetitions?
I should, for a ranking. But the gap here is not a measurement-noise gap. The best recall I measured was 0.389 against a line of 0.75, and both models missed the line in both rounds. Repetitions would sharpen an estimate that is already far from the decision boundary.
What did the whole thing cost?
$0.29558 across 33 runs. The two rounds I ran under this project’s approved ceiling of $0.50 came to $0.16540 of that, and the August round was budgeted separately. The scoring, the answer keys, and the reading of raw output were mine, and that is where the actual cost of this project was.
Next steps
The checker line is closed. Both candidates failed the pass line twice, and I am not running a third variation of the same question.
Two things stay open. The first is the band I have not measured: work whose output can be verified mechanically, with a pass line written before the first call. The second is the question underneath all of this, which I never actually asked. Routing through a paid aggregator has to beat what I already pay for, and after three rounds I still do not know whether it does. That is a cost question, not a quality question, and it deserves its own experiment.
If you are running a loop like this, the cheapest thing you can add is a line at the top of the plan naming the decision that is waiting on the result. Then read it again before each round. Mine would have stopped me one round earlier.