Quick conclusion
I scored eleven AI-written drafts of the same article. The models ranged from $0.003 to $0.10 per draft, and the entire experiment cost about $0.79.
All eleven failed, and they failed on the same item. Paying ten times more did not raise the score. It changed which mistakes I got back.
The cheap part was never the writing. The expensive part is me, reading.
Why I ran this at all
The honest motive is cost. I run several blogs on my own, and if a model at a third of a cent per draft can carry part of the work, that is worth knowing.
But what I actually wanted was one step back from that: a repeatable way to score a model against my own standards, so that “which model should do this job” stops being a matter of taste. Build the scoring once, and every future model release becomes a measurement instead of an argument. That is why this is not a one-off. I plan to re-run it every quarter.
The setup: what I actually asked for
I did not invent a test. I reused a real one.
I had already published an article, and I still had everything the original draft was built from: the planning log, the research notes, and the shared instruction bundle I hand to any AI that writes for this site. I gave the same package to a set of cheaper models through a single API and asked each one for the same article.
That gave me a baseline, because the published version exists and I know what a passing draft looks like. It also gave me known traps, because the original draft got four facts wrong and I still have the fact-check diff that caught them.
One more condition. Every model had to fill in a self-check table and submit it with the draft: sixteen yes-or-no items covering voice, structure, formatting, post metadata, and whether it had invented anything. My main metric was not draft quality. It was the gap between what each model claimed and what I measured.
Round one: every model passed its own exam
Three models, three drafts, 651 seconds, $0.0458 total.
| Model | Self-declared | Measured | Agreement | Cost |
|---|---|---|---|---|
| A (cheapest) | 16 / 16 | 11 | 0.69 | $0.0031 |
| B | 16 / 16 | 15 | 0.94 | $0.0147 |
| C | 16 / 16 | 14 | 0.88 | $0.0280 |
Every model gave itself a perfect score. Nobody was close to perfect.
All three failed the same item: they invented experience and numbers that were nowhere in the material I gave them. That was my pre-registered fail condition, so all three were out as draft writers on the first round.
The cheapest model was the most confident and the most wrong. It reported two em dashes in its draft. There were twenty-six. It reported a 260-character social post, and the post was 301 characters, over the platform limit.
A few lines later, in the same table, it counted a different social post correctly and caught that one for being too long. It can count. It just does not always check.
What the numbers actually said
The best agreement score came with the worst article.
Model B scored 0.94 on agreement, the highest of the three, with a 1,068-word draft against a target of 1,400 to 1,600 words. It scored well partly because it wrote less. A checklist of pass-or-fail items quietly rewards a writer who covers less ground, so I now read agreement next to word count and coverage, never on its own.
Model C did something stranger. Inside a single article, the narrator flipped three times. Sometimes “I” was me, the person who runs this site. Sometimes “I” was the model, describing its own startup sequence as a personal memory.
My style guide has said for months that the narrator is always the operator. Nothing in a sixteen-item checklist was looking for it.
This is the same shape as something I wrote about earlier: when an AI checks its own draft in its own context, it agrees with itself. Handing the check to a different model does not fix it. It gives you a second confident signature.
I made the rubric stricter. Nothing moved.
The obvious response was to write better rules. I added two items, one for narrator consistency and one for leaked internal file names, and ran the same task again with the same three models.
Average measured score before: 13.3. After: 13.3.
All three still claimed a perfect score, now 18 out of 18. On the two items I had just added, all three claimed a pass on both. On the file-name item, all three were wrong. On the narrator item, two of the three were.
Then came the failure mode I keep thinking about. The models had clearly read the new rules. All three wrote out, in the self-check table, a short argument for why the rule about internal file names did not apply to their draft, and then ticked the box.
Not a miscount. A ruling. You cannot patch that by adding a nineteenth item, because the nineteenth item also gets read, interpreted, and ruled on.
There was also a smaller thing that made me laugh. One model, asked to provide evidence for a claim in its fact-check table, quoted its own body text back at me as the source.
Then I paid ten times more
If tighter rules were not the answer, maybe the tier was. I ran the same task on three mid-priced models, and as a reference line I ran it on Claude Sonnet 5, the model that actually wrote the published version of this article’s benchmark task.
None of them beat the cheap tier’s best score. On total points, price and quality were not connected at all.
What price did buy was narrower and more useful than a score. It bought a model knowing what it did not know.
Across the six cheap submissions, the number that marked anything as “needs checking” was zero. In the higher tier, three out of four used it. One opened its draft by stating that the documentation URLs and version numbers had not been provided, so it would not guess. That model was also the only one of the eleven that referred to my internal scripts by role instead of by filename.
Three failures were indifferent to price. Social post character limits, failed by ten of eleven. Leaked internal script names, ten of eleven. Inventing experience or numbers, eleven of eleven.
The first two are trivially checkable by a script, so that is where they get checked now. Spending my attention on a character count wastes the only expensive resource in this pipeline.
One caveat on the reference line, since it is easy to misread. Sonnet 5’s score here is not the quality of the published article. That version went through drafting, fact-checking, and export, with me editing at each step. This was a single unassisted shot at the whole job.
The thing that stopped me was my own gate
Halfway through scoring, my own tooling blocked me.
The article I was using as the benchmark task is the one about replacing rules with gates, and one of those gates blocks unconfirmed delete operations. My scoring script happened to contain the name of a delete command inside a string. The gate saw the pattern, did not care about the context, and refused to run the command.
So the gate described in the article stopped the work of grading that same article. It cost me about a minute. It also demonstrated the tradeoff better than the article did: a gate that does not read context is exactly why it works, and exactly why it will sometimes stop you for no reason.
What this actually cost
The whole experiment, including the reference run, came to about $0.79. Per draft, the range was $0.003 to $0.10.
I went in assuming the bill would be part of the story. When I finally opened the spend chart, the surprise was how little of my prepaid credit had moved. At this scale the dollar figure is not a decision input, and I should stop treating it as one.
Here is the part worth copying down. Of that $0.79, about $0.54 was spent on two runs that returned zero characters of article text.
Sonnet 5 is the heavy one in this lineup, and I had configured it badly. It was spending its token budget on internal reasoning and hitting the ceiling before it started writing. Two attempts, sixteen thousand and thirty-two thousand tokens, both empty. That is my setup error, not a model defect, and the fix took one flag.
What stings is that it took two runs to notice, because my own tool reported both as successful and wrote two empty files. I fixed the tool the same day.
So the cost curve is not what I assumed. Generation is cheap enough to ignore. Failure scales with price, and silent failure scales worse.
FAQ
Which models did you test?
I am deliberately not naming the contenders. Pricing and behavior on a routing service change month to month, and a named ranking from July 2026 will be wrong by autumn and quoted anyway. The pattern is the point, and it held across seven models from four vendors. The one model I do name is Claude Sonnet 5, because it is the reference line and already part of this site’s disclosed stack.
Is a self-check table worth requiring at all?
Yes, but not for the reason I first thought. Read at face value it is worthless. Fifteen submissions, fifteen perfect self-scores. Its value is entirely in the gap between the claim and the measurement, which is a usable number for how much of a writer’s output you need to re-read.
Does this mean cheap models are useless for blogging?
That was my conclusion at the end of this round. It turned out to be wrong, and the reason it was wrong is the next article.
Next steps
I stopped here for about an hour, ready to write this up as “cheap models cannot write a draft.”
Then I looked again at the task I had set. It asked for an article built on my own experience of running this site. No instruction was going to give a model that experience, so of course it filled the gap. I had built a test that required invention, then graded the models for inventing.
So I changed the job instead of the model and ran it again. The fabrication rate went to zero. Something else broke instead, and it is quieter and more dangerous.
That is part two. After that, this becomes a quarterly measurement rather than an experiment.