Quick conclusion
I added a new check to my writing process this month, one meant to catch problems that fact-checking alone won’t catch: leftover process notes, hedges that don’t match the evidence, sentences that don’t connect, paragraphs that just read a table back out loud. I ran it for the first time this week, against real output from two models. It flagged the clean, publishable drafts. It said nothing about the broken ones. Silence from a check like this doesn’t mean the writing is fine. It might mean the check can’t see the specific way it broke.
The setup
Everything I’ve published here for the last five posts has been about facts: is the number real, is the source real, did a model invent something it never saw. I built and ran checks for exactly that, and I’ve written about what those checks caught and missed, including a gate nobody ended up walking through.
This month I added a different kind of check, one that doesn’t ask whether a fact is true. It asks whether the writing itself is honest with the reader: does it leak how it was made, does it state something as certain when it isn’t, does one sentence actually follow from the last, does a paragraph give the reader anything to use. I call it the meaning gate, to keep it separate from the fact-checking gates I already had.
By the time I ran this, I had two posts ready for a first check. Zero of them had gone through it yet. That’s the honest starting point for this post: a brand new check, still untested against real output, about to get its first real run.
I ran it against output from two current models, GLM-5.3 and Gemini 3.8 Flash. Neither is here because it’s cheap. GLM-5.3 is, in fact, the most expensive model in the GLM family by output price. My last five posts already covered what happens when you route work to inexpensive models. This one is about whether a different kind of check can tell working writing from broken writing, regardless of what it cost to produce.
Calibrating the instrument before trusting it
Before pointing the gate at anything a model wrote, I pointed it at a fake draft I built myself, with nine known problems planted in nine sentences, on purpose, so I would know the right answer before the gate gave me one.
Four of those problems used words the gate’s English rules already know to look for, things like “TODO” and “I still need to fact-check.” It caught all four.
The other five didn’t use any of those words: a passage describing how many times a draft got rewritten, vague hedging with no reason for it, a paragraph that just reads a table back in prose, a sentence that draws a conclusion the one before it doesn’t support, and a passive sentence that hides who actually made a decision. Same five categories the gate is supposed to cover, just phrased without the specific vocabulary it was built around. It caught zero of five.
That told me something before I’d even started the real test: the gate’s English rules only have explicit coverage for two of its five intended categories, and even those two are partial word lists, not real detection. The other three don’t exist yet on the English side at all.
The test I got wrong first
My first attempt at the real test was unfair, and I want to say that plainly rather than bury it. I asked both models to write a finished section with specific numbers, gave them no source material, and didn’t tell them not to invent anything. Under those conditions a model has exactly one option: make something up. Both did, and I initially wrote that down as a finding about the models. It wasn’t. It was a finding about my test.
I redid it the way I should have the first time: give the model the same notes a human writer would get, and tell it plainly to mark anything the notes don’t cover instead of guessing. That’s the actual protocol I hold myself to when I draft, and it’s the only fair way to test whether a model can follow it too.
What the fair test found
With real notes and permission to say “I don’t have that,” both models stopped inventing numbers entirely. Zero unsupported figures from either one, down from nine and six in the unfair version. Gemini 3.8 Flash even wrote, plainly, that it had no track record to show me on something it didn’t have data for. That’s the honest-experimenter standard I hold myself to, and a model met it without being asked twice.
Then I ran the gate on those same two clean drafts, and on a second pair written with no source notes at all, just the instruction not to invent anything and to flag the gaps instead.
| Output | Draft quality | Gate flags | Verification marker used |
|---|---|---|---|
| Gemini 3.8 Flash, given real notes | Clean, publishable | 4 | 0 |
| GLM-5.3, given real notes | Clean, one misread | 2 | 0 |
| Gemini 3.8 Flash, no notes | Incomplete, placeholders left in | 0 | 5 |
| GLM-5.3, no notes | Broken, invented details anyway | 0 | 2 |
The two clean drafts got flagged. The two broken ones did not.
Why the clean drafts set off the alarm
Every flag on the clean drafts turned out to be a false alarm, and the reason was the same each time. Both models had to describe the test itself as part of writing about it, which meant quoting phrases like “TODO” or “there is no evidence” while explaining what those phrases were being tested for. The gate can’t tell the difference between a model using a phrase and a model quoting a phrase. Using my own test material as an example made a clean draft look like it had the exact problem that material was designed to represent.
Why the broken drafts stayed quiet
This is the part that matters more, because a false alarm just wastes a few minutes and a false all-clear does the opposite. Without notes to work from, Gemini 3.8 Flash left five separate points where it correctly admitted it didn’t have enough to say something. That’s not a failure. That’s the fallback working exactly as designed. GLM-5.3, worse off, used that same fallback only twice, embedded it into ungrammatical sentences, and invented a specific memory of an experiment it never ran to fill the rest of the gaps.
The gate’s score for both outputs: zero flags.
The one word the gate can’t read
My drafting process runs in Japanese behind the scenes even though every post here is written in English, and the specific marker it requires when material runs out is a Japanese word: 要確認, pronounced yo-kakunin, meaning roughly “needs verification.” It’s not a nice-to-have. It’s the one word the entire fallback depends on.
The gate’s English rules never look for it, because they were never given any Japanese vocabulary to check, on the reasoning that an English blog shouldn’t need Japanese pattern matching. That reasoning is sound for everything else the gate checks. It just happens to be wrong for the one word my own process is built around using when a draft is honestly incomplete. The exact marker my workflow requires when a model runs out of material is invisible to the tool meant to catch drafts that ran out of material.
Lessons learned
- A quiet result from a check like this isn’t proof the writing is clean. It might just mean the way it broke doesn’t match any pattern the check knows to look for.
- False alarms and blind spots can come from the same structural gap. Mine came from one checklist ported unevenly from one language to another.
- Before trusting a benchmark, check whether the test itself is fair. My first attempt at this one wasn’t, and the fix wasn’t a better model. It was better notes.
- I ran each version of this test once. I haven’t repeated any of it, and I’m not treating one run per condition as more certain than it is.
FAQ
Does this mean automated content checks are a waste of time?
No. The same gate caught every problem I planted using vocabulary it already knew, instantly and correctly. The danger isn’t that it’s useless. It’s trusting silence from it as proof of a clean draft, when silence can also mean a blind spot.
Is this specific to GLM-5.3 or Gemini 3.8 Flash?
No. Both models behaved well once given a fair test, and the gap that let broken output through has nothing to do with either one. It’s in the checklist’s own coverage, not in what either model wrote.
Why not just add the missing word to the English rules?
That would fix this one instance. It wouldn’t fix the bigger issue underneath it, which is that the English side of this checklist only has explicit rules for two of its five intended categories in the first place. A single word patch would hide a wider gap instead of closing it.
What did this actually cost to test?
The fair version of the test, both models combined, ran about two cents in API fees. Finding out a new check has a blind spot is cheap. Trusting it before finding out would not have been.
Next steps
Widening the English side of this checklist is on my list, but I’m not treating a wider word list as the fix on its own. This test already showed that word lists are exactly what let two real problems through unflagged. The next test I actually want to run is having someone, or something, read a finished post with none of the backstory I’ve just given you, and see if the same problems still slip past unnoticed. Until I’ve run that, I’m treating every quiet result from this gate as “not flagged,” not as “clean.”