I write a lot of my drafts with AI, and for a long time the scariest part wasn’t the writing. It was hitting publish. An AI will hand you a paragraph that reads beautifully, cites a real source, and is quietly, confidently wrong.
I tried the obvious fixes first: a smarter model, a longer “be accurate and cite everything” prompt, asking the AI to double-check its own work. They all helped a little. None of them was the thing.
The thing turned out to be almost stupidly simple: change the context. Don’t make the AI check its own draft in the same chat where it wrote it. Hand the finished draft to a fresh context — a new chat, or a separate agent that never watched the writing happen — and ask it to verify each claim from scratch, straight from the source. That combination catches errors the original chat swears aren’t there.
Here’s why it works, where it doesn’t, and how I actually run it.
Why same-context self-checks fail
When you ask an AI to fact-check the draft it just wrote, you’re asking it to audit its own assumptions using the same assumptions. If it decided halfway through that a source said X, that belief is now sitting in the conversation. Ask “is this accurate?” and it re-reads its own confident claim, recognizes it as something it already “knows,” and waves it through.
I call it verified-by-vibes. The source is real. The sentence sounds careful. The check comes back clean. Nothing was actually re-derived from the source — the AI just agreed with its earlier self.
This isn’t a model being lazy. It’s the predictable result of checking work inside the context that produced it. The buried assumption never gets challenged because, from inside that chat, it isn’t an assumption anymore. It’s settled.
The fix: switch the context
So I stopped asking the writing context to grade itself. Instead, the finished draft goes to a clean slate.
- Minimum version (anyone can do this): open a brand-new chat. Paste only the draft — not the research conversation, not the back-and-forth that produced it. Say something like: “Check each factual claim against a primary source. Flag anything the source doesn’t actually support.” Because this context never saw the draft get built, it has no assumptions to protect. It reads the claims cold.
- The heavier version I use: a separate agent runs the audit with no memory of the writing session, re-checking claims straight from primary sources. Same principle, more horsepower — but the new chat trick gets you most of the way there for free.
One honest caveat: part of what’s working here is simply that the second pass is forced to re-open the source instead of trusting the draft. You could argue that’s the real lever, not the fresh context. I think they’re two halves of the same move — the separation is what keeps the re-check honest, because a context with no stake in the draft has no reason to skip the source and nod along. A fresh context has its own blind spot, too: it doesn’t know the article’s intent, so it can flag things that were fine on purpose. That’s a feature here. You want a reader who isn’t in on it.
A real catch
One real catch from my finance blog (in Japanese). A draft explained an idea using a clean “three-axis framework” and attributed that framing to a government source. My in-chat fact-check passed it with zero issues — the source was real, the numbers matched, so it felt verified.
Then I ran the same finished draft through a fresh-context audit. It pulled up the actual source text and found something the in-chat pass had glossed over: the framework itself is a perfectly legitimate, commonly-used way to explain the idea — but the source never laid it out in those three labeled parts. The draft had presented its own tidy structure as if the source had structured it that way. Not a fake fact. A real framework, pinned to an authority that didn’t actually phrase it like that.
The in-chat checker found nothing, because at the fact level there was nothing wrong. The fresh context caught the thing one level up: an attribution the source didn’t support.
That’s the whole argument in one example. Nothing about the model changed between the two checks. What changed was that the second reader had no investment in the draft being right — and went back to the source to prove it.
(And yes — I’ll admit the irony. The first draft of this very post overstated that same catch, calling it a “fabricated framework” until a fresh-context pass made me walk it back to what actually happened. The method caught the article about the method.)
Where I draw the line
I don’t do this for every paragraph of every post — it would be exhausting and it’s overkill for a casual update. I reach for the fresh-context audit when the cost of being wrong goes up:
- The draft leans on specific numbers (percentages, dates, figures).
- It’s a topic where an error actually hurts the reader — money, health, anything they might act on.
- It’s a foundational post other articles will link back to and inherit from.
- It was written unattended, on a pipeline where no human read every line as it went.
That last one surprised me. The less a human is watching, the more the independent check earns its keep — automation is exactly where verified-by-vibes slips through unnoticed.
The failure that taught me the rule
I’ll be honest about how I learned to take this seriously, because I learned it by getting it wrong.
On one article, my notes proudly recorded that the draft had been through an independent audit. It hadn’t. I’d labeled the step as done and quietly skipped actually running it. The draft looked finished, the record said “audited,” and it nearly shipped on that false comfort. It only got caught on review, when someone went back over the record and noticed the claim didn’t match what had actually been done.
The lesson stuck harder than any clean success would have: a step you name but don’t run is worse than no step at all. It manufactures exactly the false confidence the audit exists to destroy. “Supposed to be checked” reads, at a glance, identical to “checked.”
So I changed the rule. No more “audit-equivalent,” no more checking off a step I didn’t perform. The independent pass is now a real, separate action from the start, and the record has to match what actually ran. If the audit didn’t happen, the draft doesn’t get to claim it did.
Try it on your next draft
You don’t need an agent fleet to use any of this. Next time AI writes you a draft, do one thing: open a new chat, paste only the draft, and ask it to verify the claims against primary sources. Watch what a context with no stake in the answer finds.
The upgrade was never a better brain. It was a second pair of eyes that hadn’t already made up its mind.
If you want the bigger picture of how this fits together, here’s how I run multiple blogs solo with AI, how I draft in English as a non-native speaker, and the publishing flow this check sits inside.