To fact-check AI output without drowning in it, stop trying to verify every sentence. Pull out the claims, rank them by what breaks if they're wrong, and check the ones your piece rests on against the original source, because neither a tidy citation nor the model's own "yes, I'm sure" proves anything.
The cost of skipping that step keeps climbing. A public database of court decisions involving AI-hallucinated material had logged 2,125 of them at the time of writing, and 1,193 of those came from this year alone (AI Hallucination Cases database). Earlier this year KPMG pulled a published report over apparent AI hallucinations, after organizations it described called its claims about their AI use "either untrue or misleading" (TechCrunch).
Hallucinations haven't gone away. They've become harder to spot, because the prose around them is fluent and the fake references look like real ones. Below you'll find the claim types that fail most, why self-checking falls short, a risk-tiered workflow with copy-ready prompts, and an honest look at grounding tools like Gemini Notebook (formerly NotebookLM), Perplexity and Consensus.
Fake citations are now a courtroom genre
Lawyers learned this first, and in public. Damien Charlotin's database records decisions where a court or tribunal found, or implied, that a party relied on hallucinated content, usually invented case law. It doesn't try to count every fake citation ever filed, only the ones a court or tribunal addressed in writing. Even with that narrow filter, the curve is steep.
The 2026 bar covers only part of the year and has already passed 2025. People representing themselves account for 1,227 of the tracked decisions, but lawyers account for 840, and the list includes 33 where the hallucinated material came from a judge. The United States dominates with 1,459 decisions, followed by Canada with 223 and Australia with 112.
The case that started it all is still the best lesson. In Mata v. Avianca in 2023, a lawyer in a New York federal case asked ChatGPT whether one of the cases it had given him was real. It said yes. Pressed again, it said the case "does indeed exist and can be found on legal research databases such as Westlaw and LexisNexis" (CNN). The case didn't exist, and the court fined the lawyers and their firm $5,000. The lawyer said in an affidavit that he "was unaware of the possibility that its content could be false."
The bills have grown since. In Couvrette v. Wisnovsky, a federal court in Oregon found fifteen fabricated citations and seven misquotes across the plaintiffs' briefs. It struck the briefs, dismissed the claims with prejudice and imposed $110,204 in sanctions and costs, the biggest total for a US case in the database. If you use AI in legal work, our guide to AI for legal work covers the safer setups.
Courts are only the most visible arena. Deloitte agreed to a partial refund on a $440,000 Australian government review that cited nonexistent reports by university professors and included a made-up reference to a real court decision. The firm later disclosed it had used a tool chain built on Azure OpenAI's GPT-4o (The Guardian). EY withdrew a report on loyalty programs this year that appeared to include fake footnotes, according to the same TechCrunch story that covered KPMG. And in 2025 the Chicago Sun-Times printed a syndicated summer reading list on which 10 of the 15 books were invented, NPR reported.
Where AI output goes wrong
Models rarely misspell words anymore, and OpenAI's researchers explain why the errors moved elsewhere. Spelling follows patterns that a large model learns, but "arbitrary low-frequency facts, like a pet's birthday, cannot be predicted from patterns alone and hence lead to hallucinations" (OpenAI). When they asked a widely used chatbot for the PhD dissertation title of one of the paper's own authors, it gave three different answers. None was right.
That gives you a map. Errors cluster in a handful of claim types, and the riskiest are the ones dressed up as proof.
Citations sit at the top because they fail in two ways. Some are invented outright: when the Tow Center at Columbia ran 1,600 queries through eight AI search tools in 2025, asking each to identify the source of a news excerpt, the tools collectively got more than 60% wrong, and Grok 3 sent 154 of its 200 citations to error pages (Tow Center). Others are real but cited for something they don't say. Charlotin's database counts 894 decisions involving misrepresented material, next to 1,752 involving fabrication. A working link doesn't catch that second kind.
Quotes come next. The database lists 572 decisions with false quotes, and OpenAI's own help center names "fabricated quotes, studies, citations or references to non-existent sources" among ChatGPT's known failure modes (OpenAI Help). Numbers fail more quietly: a real figure from the wrong year, a percentage presented as percentage points, a regional price passed off as a global one.
Recency is its own trap. Perplexity's developer docs say hallucination is most likely when information sits behind a login or paywall, when searches return "related but non-matching results", or "when very recent information isn't indexed yet" (Perplexity docs). A 2025 BBC-led study of more than 3,000 news answers from ChatGPT, Microsoft Copilot, Google Gemini and Perplexity found major accuracy problems, including outdated information, in 20% of them.
Then there's tone. In that Tow Center test, ChatGPT misidentified 134 articles but flagged uncertainty only 15 times in 200 responses, and paid tiers of some tools answered more questions correctly while also giving more confidently wrong answers. OpenAI's help page puts it plainly: "The model may express high confidence even in incorrect answers." In AI output, a confident tone tells you almost nothing about accuracy.
Why asking the model to double-check falls short
The obvious shortcut is to paste the draft back in and ask whether it's accurate. Mata v. Avianca already showed how that goes, and research backs it up. A paper presented at ICLR 2024 found that models "struggle to self-correct their responses without external feedback, and at times, their performance even degrades after self-correction" (Huang et al.).
The cause is in the training. OpenAI's researchers argue that "standard training and evaluation procedures reward guessing over acknowledging uncertainty," much as a multiple-choice exam rewards a blind guess over a blank. They also concede that accuracy "will never reach 100%" because some real questions can't be answered. A model reviewing its own draft draws on the same knowledge that produced the error, so the review tends to repeat it.
Self-review still has a use. A 2023 paper on Chain-of-Verification showed that a structured version helps: the model drafts an answer, writes verification questions, answers them independently of the draft, then revises. That cut hallucinations across several tasks. Treat it as a way to find suspects, never to clear them.
Grounded is not the same as correct
Handing the model a source to work from lowers the error rate. It doesn't zero it. Vectara's public leaderboard asks models to summarize more than 7,700 articles using only the facts in each one, then scores the summaries with Vectara's own detection model. Even in that easy setting, every model on the list adds unsupported content to some share of its summaries (Vectara).
In the latest update, GPT-6 Sol added unsupported material to 6.5% of summaries and Claude Opus 4.7 to 12%. A model grading another model is imperfect, Vectara says plainly that the test doesn't capture every kind of hallucination, and the table doesn't yet include every current release. The direction is still clear. A 2024 study of legal research tools that retrieve real case law, including Lexis+ AI and Westlaw's AI-Assisted Research, found they still hallucinated between 17% and 33% of the time. Grounding narrows the problem to one question: does the source say this? You still have to ask it.
A verification workflow sorted by risk
The fix is triage. Most of a draft is connective tissue, such as framing, transitions and opinions you'd defend anyway. A small set of claims carries the piece, and that's where an error does damage. Sort first, then spend your checking time where it pays.
- Extract the claims. Ask a model to list every checkable statement in the draft: names, numbers, dates, quotes, citations and cause-and-effect claims. You'll find claims you didn't notice you were making.
- Triage by stakes. Ask what happens if each claim is wrong. Anything legal, medical or financial is high stakes, as is every citation, every direct quote, any statistic in a headline or summary, and any claim about a named person or company. Supporting context a reader might repeat is medium. Framing you could defend from memory is low.
- Verify the high tier at the primary source: the court opinion, the filing, the dataset, the company's own page, the paper itself. A news story about a study is not the study.
- Spot-check the medium tier with lateral reading, covered below, and give the low tier an editor's read.
- Fix, soften or cut. If you can't confirm a claim in the time it deserves, remove it or attribute it plainly ("the vendor says"). Never keep a claim because it sounds right.
- Keep a log. One line per high-tier claim, with the claim, the source and where in it the support sits, takes minutes and answers the question an editor, client or judge will eventually ask.
Scale the effort to how far the output travels. A brainstorm needs a careful read. A client deliverable needs every high-tier claim opened at its source. If the checking bill looks too big, the draft probably has too many claims, and cutting is often faster than verifying.
Not sure which tier a piece of output falls into? Answer up to three questions below.
Notice what the tree asks about: where the output goes and what it contains, never which model wrote it. A stronger model shifts the odds. It doesn't change who answers for the mistake.
How to check citations, quotes and numbers
Each claim type has its own quick test. None needs special software, only the discipline to leave the chatbot and open the original.
Citations
Check existence first. Search for the title, DOI, case number or URL in a database or search engine you control, never in the chatbot that produced it. Then check support: open the source and find the passage that backs the sentence. If you can't find it in a few minutes, the citation fails. Watch the version too. The Tow Center found AI search tools often pointed to syndicated copies instead of the original publisher, and a preprint can differ from the published paper.
Quotes
Search a distinctive phrase of five or six words in quotation marks. If it turns up nowhere, or only on pages that look machine-written, treat the quote as invented. If it's real, read the paragraph around it and confirm the speaker, the date and the meaning. A genuine quote pulled out of context misleads as well as a fake one.
Numbers
Recompute anything you can. Confirm the unit (percent or percentage points, monthly or yearly, which currency), the year and the denominator. If the draft says a figure doubled, find both numbers. A statistic with no named source doesn't go in, however plausible it looks.
Recent events and viral claims
Check the date on every source and prefer reporting published after the event. For a claim already circulating online, Google's Fact Check Explorer searches published fact checks and shows which organization rated the claim and how. You can also upload an image to see whether it has been checked before.
Before you hit publish, run the draft through this checklist. The first four items count double.
The weighting follows the court data: fabricated citations, misrepresented sources and false quotes fill most of that database. If those four boxes stay empty, nothing else on the list makes the draft safe to send.
Lateral reading for claims without a paper trail
Some claims arrive with no citation to open. For those, borrow the habit of professional fact-checkers. Stanford's Civic Online Reasoning group, after watching fact-checkers work, concluded that "the best way to learn about a website is lateral reading": leave the page and see what other sources say about it, instead of studying the page itself.
Mike Caulfield turned that habit into a four-step routine called SIFT (Caulfield). It fits AI answers well, with one adjustment: the source you're investigating is the chatbot plus whatever link it chose.
The third move does the most work with chatbots. When the claim matters more than the article that carried it, Caulfield's advice is to "ignore the source that reached you, and look for trusted reporting or analysis on the claim." So search the claim fresh, in your own words. If something that should be widely covered has no independent coverage, you have your answer.
Lateral reading also settles the sources a model finds on its own. A real URL on a site you've never heard of is a weak source, however neatly the AI formatted the reference.
Prompts that make output easier to verify
You can't prompt hallucinations away, but you can shape output so errors are easier to catch. The vendors' own documentation is the best guide. Anthropic says its recommended techniques "significantly reduce hallucinations" but "don't eliminate them entirely" (Anthropic docs). For wider prompting habits, see our practical prompt engineering guide.
Give it permission to say it doesn't know
Both Anthropic and Perplexity tell developers to explicitly allow the model to admit uncertainty or report that it found nothing. OpenAI's research shows why it matters. In its comparison on the SimpleQA benchmark, a model that abstained on 52% of questions got 26% wrong, while one that abstained on 1% got 75% wrong, at almost the same accuracy. Use this as a standing instruction:
Answer using only facts you can attribute to a specific source.
For each factual claim, name the source and give a short direct quote from it.
If you are not sure of something, write "Unknown" instead of guessing.
Mark anything that is your own inference or opinion with [Inference].Ask for quotes before conclusions
Anthropic's docs recommend extracting word-for-word quotes from long documents before doing the task, then having the model find a supporting quote for each claim afterwards and retract any claim it can't support. In practice that looks like this:
Step 1: From the document below, extract the exact quotes relevant to [your question]. Number them. If none are relevant, write "No relevant quotes found."
Step 2: Answer using only the numbered quotes, citing them like [2].
Step 3: Review each sentence of your answer. If a sentence has no supporting quote, delete it and mark the spot with [].Make it flag near-misses, and don't force brevity
Perplexity's guide adds a rule worth stealing: "If you find related but non-matching results (for example, a different year, a parent company, or a subsidiary), state the mismatch explicitly before answering." That catches the right fact about the wrong entity, one of the hardest errors to spot by eye. Its sample rules also tell the model to cite sources by domain and "not write full URLs."
Length matters too. Giskard, a Paris-based AI testing company, found that asking GPT-4o, Mistral Large and Claude 3.7 Sonnet for concise answers made them less accurate, because "when forced to keep it short, models consistently choose brevity over accuracy." The same study found models were less likely to push back on a controversial claim the user stated confidently. Phrase questions neutrally, "Is it true that X?", rather than assuming the claim, as in "Explain why X."
Extract claims, then run it twice
This prompt feeds the first step of the workflow. Run it on any draft, including one you wrote yourself.
List every factual claim in the text below as a numbered table with three columns: the claim, its type (citation, quote, number, date, name or other), and the kind of source that would prove it.
Do not judge whether the claims are true. Do not add claims that are not in the text.Anthropic also suggests best-of-N checking: run the same prompt several times and compare, since inconsistent answers point to likely hallucinations. It's the same symptom OpenAI saw with the three dissertation titles. Disagreement between runs tells you where to look. Agreement proves nothing.
Source-grounded tools help, up to a point
A different class of tool narrows what the model can draw on. That turns "is this true?" into "does the source say this?", which is a faster check. It doesn't remove the check.
Google renamed NotebookLM to Gemini Notebook earlier this year and kept it a standalone research tool, which Google says more than 30 million people use (Google). In chat it answers from the sources you add and "uses direct quotes, text, and images from your sources as citations." Hover over a citation to see the quoted text, or click to jump to it in context (Gemini Notebook Help). That makes support checks quick. Know the gaps as well: Google's help pages say paywalled web pages aren't supported and footnotes and comments in Google files aren't imported. The newer agentic chat features, which can search the web and run code, are labelled experimental, with the instruction "Be sure to double check it."
Answer engines such as Perplexity attach links to each answer, a real step up from an unsourced chatbot. The links still need opening. Perplexity got 37% of queries wrong in the Tow Center's 2025 test (the tools have changed since), and Google's own Gemini help notes that "not all responses include related links or sources."
For research claims, academic search tools ground answers in papers rather than the open web. Consensus searches peer-reviewed literature and says that if it "can't find sufficient relevant evidence, it will tell you" instead of filling gaps. Elicit says it backs every AI-generated claim with sentence-level citations. Scite shows whether later research supported or contradicted a paper's findings, which catches a failure the others miss: a real study that has since been challenged.
If you build your own pipeline, Anthropic's Citations API returns the exact passage behind each claim, and Anthropic's docs say those citations are "guaranteed to contain valid pointers to the provided documents." That rules out invented sources. It doesn't confirm that the passage supports the sentence it's attached to. Reference checkers sit at the end of the line. GPTZero offers a hallucination detector that scans bibliographies, and its team flagged the KPMG report and, earlier this year, found 100 hallucinated citations across 51 papers accepted at NeurIPS, a leading AI conference, TechCrunch reported.
| Tool type | Grounds answers in | What you still check |
|---|---|---|
| Notebook tools (Gemini Notebook) | Sources you upload | That each cited passage supports the claim, and that your sources are right |
| Answer engines (Perplexity, ChatGPT with search) | Live web results | Open every link, check its date and find the original publisher |
| Research search (Consensus, Elicit, Scite) | Academic papers | The paper's actual findings, limits and any later contradicting work |
| Citation APIs (Claude Citations) | Documents you supply | Support for the claim, beyond a valid pointer |
| Reference checkers (GPTZero) | Bibliographic records | Whether a real reference says what the draft claims |
To compare options, browse the AI research assistants board, the AI search engines board and the board for searching academic literature. Our roundups of AI research assistants and AI search engines go deeper on features and pricing.
Frequently asked questions
Can I use one AI model to fact-check another?
As a first pass, yes. A second model, especially one with search, will flag some errors the first missed, and disagreements show you where to look. It can't confirm anything, since both may share the same gaps. Treat its flags as a to-do list and settle each one at the source. If you're choosing a second assistant, our ChatGPT alternatives guide compares the main options.
Do AI detectors check facts?
No. Tools such as the Originality.ai detector estimate whether text was machine-written. That says nothing about whether a claim is true, and a human-written draft can be just as wrong. Fact-checking needs the original source.
How long should fact-checking an AI draft take?
It depends on how many high-stakes claims the draft contains, not on its length. Count them after triage. A 2,000-word explainer with four statistics and two quotes is a short job, while a one-page memo full of citations is not.
Can I cite ChatGPT or another chatbot as a source?
Not for facts. OpenAI's own advice is to "use ChatGPT as a first draft, not a final source." Cite the underlying document you verified. If the AI's output is itself the subject, as in a test of a model, say which model you used and when.
Do deep research agents make checking unnecessary?
No. Agents such as OpenAI's deep research compile cited research reports from many web sources, which gives you more citations to open, not fewer claims to check. The misrepresented-source problem applies to every cited sentence, so the triage step matters more as reports get longer.
