The claim up front: a language model's confidence carries no information about whether it's correct, so you cannot verify its answers by reading them more carefully — you verify by routing each answer through a fixed checklist and reserving real scrutiny for the claims that would actually cost you something if they were wrong. Re-reading a fluent paragraph for errors is the wrong tool, because fluency is exactly what the model is optimized to produce. This guide gives you a triage rule for deciding how hard to check, a five-step verification routine, a table of failure modes and the tell that exposes each one, and the mistakes that make people either trust too much or waste hours double-checking trivia.
By the end you'll be able to look at any AI answer and decide, in about ten seconds, whether it needs a glance, a source check, or a full independent rebuild — and know exactly how to run each.
TL;DR
- Confidence ≠ correctness. A model's fluency and certainty are produced by the same process whether the answer is right or wrong. Tone is not evidence. (See [What Is an AI Hallucination](what-is-an-ai-hallucination.md).)
- Triage before you verify. Score each claim on cost-if-wrong × checkability. Most answers need a glance; a few need a full rebuild. Spend effort accordingly.
- Verify the claim, not the prose. Re-reading the answer can't catch a confident fabrication — only an outside source or an independent recomputation can.
- Know the five failure modes — fabricated specifics, stale facts, plausible math, misattributed quotes, and silent scope drift — each has a reliable tell.
- Ground the model in your own sources to cut the verification load before the answer is even generated.
Why "read it carefully" doesn't work
A language model predicts the most plausible next words given everything before them. It is trained to produce text that reads like a correct, confident answer — which means a fabricated citation and a real one are generated by the identical mechanism and come out equally polished. (For the underlying mechanics, see [How Large Language Models Work](how-large-language-models-work.md).)
This breaks the instinct most people rely on. In human writing, hesitation, vagueness, and hedging often leak through when the writer is unsure. In model output, that signal is gone: the model will state a wrong date, a non-existent court case, or a miscalculated total in exactly the same assured register as a verified fact. You are not a worse proofreader than you think — you are using proofreading on a problem that proofreading cannot solve. Verification has to come from outside the answer.
Step 1: Triage — decide how hard to check
Verifying everything is as much a failure as verifying nothing; one wastes your time, the other wastes your credibility. Score each claim on two axes:
| Axis | Question | Low | High |
|---|---|---|---|
| Cost if wrong | What happens if I act on this and it's false? | Mild annoyance | Money, reputation, safety, a published error |
| Checkability | How fast can I confirm it? | One search / one recomputation | Needs expertise or unavailable data |
The two axes give you a routing rule:
| Cost if wrong | Verification level | What you actually do |
|---|---|---|
| Low | Glance | Sanity-check it reads sensibly; move on |
| Medium | Source-check | Confirm against one independent source |
| High | Rebuild | Reconstruct the answer yourself from primary sources or first principles |
The decision rule: the higher the cost of being wrong, the less the model's answer counts as evidence at all. For a high-stakes claim, treat the AI output as a draft hypothesis, not a finding.
Step 2: Separate claims from framing
Before checking anything, split the answer into its load-bearing claims. A typical AI response mixes three things:
- Verifiable facts — dates, numbers, names, quotes, citations. These are checkable and are where fabrication hides.
- Reasoning steps — "because X, therefore Y." Checkable by following the logic, independent of the facts.
- Framing and hedging — "generally," "it's often said," "many experts believe." Not claims; don't waste verification on them, but notice when they're doing the work of smuggling an unsupported assertion past you.
Worked example. The answer: "The Sharpe ratio, introduced by William Sharpe in 1966, is generally considered the best single measure of risk-adjusted return; a value above 1.0 is widely regarded as good." Splitting it: fact — Sharpe, 1966 (checkable, and correct); framing — "generally considered the best" and "widely regarded as good" (opinion dressed as consensus — the real content is a rule of thumb, not a law). The verification load is one date and a recognition that the qualitative claims are softer than they sound.
Step 3: Match the failure mode to its tell
Hallucinations are not random — they cluster into a handful of patterns, and each has a reliable tell that tells you where to look before you look.
| Failure mode | What it looks like | The tell | How to check |
|---|---|---|---|
| Fabricated specifics | A precise citation, URL, case name, or statistic that doesn't exist | Suspiciously exact and convenient; "perfect" supporting source | Search for the source directly; if it doesn't resolve, it's invented |
| Stale facts | Confidently states something true as of the model's training, now outdated | Present-tense claims about prices, leaders, "latest" anything | Check the live/current value against a primary source |
| Plausible math | Arithmetic or unit conversions that look right and aren't | Multi-step calculation presented without the steps | Recompute independently; never trust an unshown total |
| Misattributed quotes | A real-sounding quote pinned to the wrong person or invented whole | Quote is too on-the-nose for the point being made | Search the exact quoted string in quotation marks |
| Silent scope drift | Answers a narrower or broader question than you asked | The answer is fluent but doesn't quite fit your case | Re-read your question, then ask "did it answer this one?" |
The pattern across all five: the tell points you at the claim's weakest joint so you don't have to check the whole answer uniformly. A precise statistic and a vague generalization carry very different fabrication risk; check the precise one.
Step 4: Run the right check
Each verification level has a concrete routine. Don't improvise — improvised checking is how a fabricated citation survives because you "kind of remember reading something like it."
Glance (low cost). Read for internal contradiction and obvious impossibility. Does a number in paragraph three contradict one in paragraph one? Does a total exceed the sum of its parts? This catches sloppy errors in seconds and nothing more — that's the point.
Source-check (medium cost). Find one independent source and confirm the load-bearing fact. "Independent" is the operative word: asking the same model "are you sure?" is not a check — it will often double down or, worse, flip a correct answer to placate you. A second source means a different system or a primary document.
Rebuild (high cost). Reconstruct the answer without leaning on the model's version. For a calculation, redo it from the inputs. For a factual claim, go to the primary source — the filing, the statute, the paper, the documentation. For an argument, write the reasoning yourself and see if you reach the same conclusion. If your rebuild and the model's answer agree, you have real confidence; if they diverge, you've found exactly the thing verification exists to catch.
Step 5: Reduce the load at the source
The cheapest claim to verify is one the model never had to invent. Grounding — giving the model the source material and asking it to answer only from that — shrinks the verification surface from "everything the model has ever absorbed" to "the documents in front of it."
Worked example. Asking a model "what's our refund policy?" cold invites a plausible-sounding invention. Pasting in your actual policy document and asking "according to this, what's the refund window?" turns the task from recall (high fabrication risk) into extraction (low risk, and trivially checkable against the text you provided). The same move applies to your own notes: when the source of truth is a document you control, verification collapses to "is this actually in the source?" — a question you can answer by reading one paragraph instead of auditing the entire web.
Common mistakes
- Treating confidence as evidence. The single most expensive error. A fluent, assured tone is the model's default output, not a signal of correctness.
- Asking the model to check itself. "Are you sure?" measures agreeableness, not accuracy. Verification must come from outside the system that produced the answer.
- Verifying uniformly. Checking a low-stakes aside as hard as a high-stakes number burns the time and attention you needed for the claim that mattered.
- Stopping at plausibility. "That sounds right" is the feeling a hallucination is engineered to produce. Plausibility is the start of verification, never the end.
- Forgetting the clock. A model's facts are frozen at training time; anything that changes — prices, office-holders, "current" anything — is stale by default and needs a live check.
- Checking the prose, not the claim. Re-reading the answer for errors can't catch what the answer is confidently asserting. Only an outside source can.
Summary + next step
Verification isn't a posture of distrust — it's a routing problem. Triage each answer by what it would cost you to be wrong, split it into checkable claims, match each suspect claim to its failure mode and the tell that exposes it, and run the check at the level the stakes deserve. Most answers earn a glance; the few that don't are exactly the ones worth the rebuild. And the highest-leverage move happens before generation: ground the model in sources you trust so there's less to verify in the first place.
That last point is where your notes do real work. When you keep your own facts, decisions, and sources in one place you control, the model answers from your material instead of from its memory — and an answer you can check against a note you wrote is one you barely have to verify at all. To go deeper on why models invent in the first place, read [What Is an AI Hallucination](what-is-an-ai-hallucination.md); for the mechanism underneath it, [How Large Language Models Work](how-large-language-models-work.md).