Artificial intelligence systems that generate text — large language models (LLMs) — are trained on vast datasets of text scraped from the internet and other sources. These models learn statistical associations between words, phrases, and ideas, enabling them to produce fluent and plausible-sounding text. A persistent concern about these systems is the phenomenon of 'hallucination' — the generation of confident, fluent statements that are factually incorrect or entirely fabricated. This is not a failure of language generation; the text is grammatically and stylistically coherent. It is a failure of correspondence between the generated text and reality.
The cause of hallucination is structural. LLMs do not store facts as a database of verified claims. They learn patterns of language use, and when prompted to produce information, they generate text that fits the pattern of what an answer to that question might look like, based on their training data. When the training data contained the correct answer, the model may produce it. When it did not, or when the model is operating near the edge of its training distribution, it generates plausible-seeming text that may bear no relation to fact.
This has significant implications for high-stakes uses of AI systems — in legal research, medical diagnosis, and educational settings. A fluent error is more dangerous than an obvious one because it is harder to detect. Responsible deployment of such systems in professional settings requires verification mechanisms, clear disclosure of limitations, and human oversight of AI-generated outputs.
