Why AI Invents Citations — and What Actually Prevents It
In 2023 a New York lawyer filed a brief citing six cases that did not exist. He had asked ChatGPT for supporting precedent, and when he asked whether the cases were real, it said yes. The sanctions hearing that followed made the story famous, but the underlying behaviour was not a malfunction. It was the model working exactly as designed.
Understanding why matters, because the same failure mode shows up in a literature review with a fabricated DOI — and there, nobody gets a hearing. It just gets marked.
Why does an AI make up citations?
A language model predicts the next token given the preceding ones. Trained on a large corpus, it learns what an academic citation looks like: a surname, initials, a year in parentheses, a title in sentence case, an italicised journal name, a volume, an issue, a page range, a DOI beginning 10. followed by a registrant code.
Ask it for a citation and it produces a sequence with all of those properties. That is the whole operation. There is no step in which it consults an index of published work, because there is no index — the training data is compressed into weights, not stored as retrievable records.
This is why the failure is so convincing. A fabricated citation is not a corrupted real one. It is a fresh construction from the same statistical mould, so it has the right shape, a plausible author for the field, a real-sounding journal, and a DOI that is correctly formatted and resolves to nothing.
Why doesn't "only cite real sources" fix it?
The instinctive fix is to instruct the model: only cite sources you are certain exist. This fails for a specific reason — the model has no mechanism for being certain. It cannot distinguish a memorised citation from a generated one, because both arrive by the same process. Asking it to self-assess produces another prediction: what a confident answer sounds like.
Related approaches with the same limitation:
- "Cite only papers you know." The model has no representation of "papers I know" separate from "text I can produce".
- "Give me the DOI so I can verify." It will. The DOI will be well-formed and frequently fake.
- Asking twice. Regeneration produces a different plausible answer, not a verified one. Agreement between two guesses is not evidence.
None of these introduce a lookup, and lookup is the missing piece.
What actually prevents fabricated citations?
Retrieval-augmented generation inverts the order of operations. Instead of generating text and then producing citations for it, the system:
- Indexes a fixed corpus — documents you supplied, chunked into passages, each tagged with its document and page.
- Retrieves the passages relevant to the claim being written.
- Generates text conditioned on those retrieved passages.
- Cites the passage identifiers it was given.
The citation is not predicted. It is a reference to a record that was passed into the context. If a passage was not retrieved, there is nothing to cite — the failure mode becomes "I could not find support for this" rather than a fabricated reference.
This is a difference in kind, not degree. In an ungrounded system, a fake citation is always possible because generation is unconstrained. In a grounded system, a citation that does not correspond to a retrieved passage cannot be constructed, because the citation is a pointer rather than a prediction.
How can you tell if a tool is grounded?
Marketing copy for AI writing tools converged on the phrase "no hallucinations" some time ago, and it means nothing on its own. Useful questions:
Where does the corpus come from? If the tool cites "the literature" generally, it is either searching a live index (verifiable, but check what index) or predicting (not verifiable). If it cites documents you uploaded, the corpus is bounded and checkable.
Does every citation resolve to a location? Not just a document — a page. "Smith 2020" is a claim you still have to verify by reading Smith 2020. "Smith 2020, p. 47" with a link to page 47 is a claim you can check in five seconds.
What happens when there is no support? A grounded system should be able to say so. If a tool always produces a citation for any claim, it is not checking.
Can you see the retrieved passage? The strongest signal. If you can read the sentence the claim came from, verification is immediate.
What does retrieval grounding not solve?
Retrieval grounding fixes fabrication. It does not fix:
- Misreading. A model can cite a real passage and characterise it wrongly. Someone still has to read.
- Corpus quality. Grounding on twenty bad papers produces well-cited nonsense.
- Selection bias. Retrieval surfaces what matches the query. What you did not upload does not exist as far as the system is concerned — which is precisely why the literature search stays your job.
The author remains responsible for every sentence. Grounding narrows the category of error from "this source does not exist" to "this source does not say quite that", which is a normal scholarly problem with a normal scholarly remedy.
If you take one operational rule from this: check that a citation resolves before you trust it, every time, regardless of the tool. With an ungrounded model that check is essential and frequently fails. With a grounded one it is fast and usually passes — but the check is what makes the difference, not the marketing.