1. Introduction

September 2025. San Francisco. AI safety research OpenAI has published new findings, addressing one of the most enduring imperfections in large language models (LLMs): hallucination. Even after many years of rapid progress and billions spent on AI safety, systems like ChatGPT still produce confidently false answers. This marks OpenAI’s most direct acknowledgment of ChatGPT-5 hallucinations.

OpenAI’s research claims the issue arises less from arcane bugs and more from the training and testing methods of these models. The researchers suggest that current methods tend to incentivize guessing instead of incentivizing uncertainty. In other words, when the model has low confidence in the answer, it is more likely to make one up than admit it doesn’t know.

2. A problem hiding in plain sight

For years, users have joked about how the AI system was “making things up” (this is known as hallucination, in technical terms). Hallucination occurs when the AI provides text that is coherent and authoritative sounding, but is inaccurate or not factual.

The range of hallucinations can be as insignificant as creating a fictitious name of a book, to more significant events, such as creating a fictitious legal case or providing inaccurate medical instructions. As recently as May of 2023, a lawyer in New York used ChatGPT to cite six fictitious court cases in a litigation brief, which resulted in sanctions. In 2024, the chatbot accidentally provided loan terms that did not exist to a customer of a financial services client.

The most concerning part of hallucination is the confidence it presents with. To the untrained reader, there is often no difference between a correct answer and a hallucinated answer.

3. OpenAI’s experiment: confident, wrong, and repeated

In its September 2025 report, OpenAI described simple yet illustrative experiments. The OpenAI team asked a model to provide the title of Adam Tauman Kalai's PhD dissertation, who is a very well-known computer scientist. Instead of saying it did not know, the model offered three different, but inaccurate titles quite confidently.

When asked about Kalai’s birthday, it produced three incorrect dates - again quite confidently.

This is indicative of the core issue: the system would rather guess than say it does not know. The guesses, while wrong, all seemed linguistically fluent enough to sound plausible.

4. Why hallucinations persist

According to the study, these three drivers are:

→           Training objective

               LLMs are trained to produce the next word, not verify its correctness. Therefore, plausibility is scored, not accuracy or truth.

→           Evaluation metrics

               Benchmarks and human raters on average are looking for smooth, confidently expressed answers. “I don’t know” is invariably scored lower, even when it is the most accurate response.‍

→           Incentive structure

               Models are rewarded more for random guessing than for saying “I don’t know.” Over time, models learn confidently wrong is “better” than unsure.

Together, these collectively produce a structural bias leading to the AI overconfidence issue—the same phenomenon users see as hallucination.

5. The stakes for businesses and end users

ChatGPT hallucinations represent not only a nuisance for businesses but can also ultimately result in reputational harm, and risks of compliance failure, and monetary loss.

→ Healthcare: A chatbot regularly misunderstanding symptoms might provide bad advice.This highlights why AI makes things up can be dangerous in life-or-death contexts.

→ Legal services: Fabricated citations might open the provider up to malpractice.

→ Finance: False product information might invite regulatory scrutiny.

→ Customer support: Bots issuing. dishonest refunds or discounts erode trust and raise costs.

“The bottom line is that hallucinations are not just charming blunders,” said an analyst in the industry, “but systemic risks. The sooner firms realize it is a systematic risk, the quicker they can set up guardrails.”

6. AI Guardrails and the search for solutions

The sorrow researchers are working diligently on a pathological way to end hallucination, and OpenAI's research emphasizes that there is not a one pill solution, but a multi-pronged solution.

Retrieval Augmented Generation (RAG)

RAG allows the model to have "lookups" into relevant documents, instead of strictly. An example being medical chatbot could look up from pre-verified documents databases before "answering." Thereby, anchoring the answers to resources outside the model providing accountability.

RAG has its own difficulty that if retrievals system is incorrect, the model hallucinate. Strong guard rails are still necessary in verifying source relevance and possibly verifying citations.

Confidence calibration

Modeling needs training that does not strictly train provide the answer, but if it is confident in the answer. A calibrated system really should say, “I’m not sure, but here are my citations", paper over fabricates an answer. This could illustrate possibly some changes to the training data, or eval metric so that correct uncertainty (truth) is promoted over achieving evaluation metrics.

Verification Pipelines

Some organizations using some has various different post-processing pipelines of supervised ground truth to push outputs to users. Legal assistant could lookup citations against actual databases court case decisions.

Product design guardrails for AI

Technical problem outside their control could be foisted onto the use of Product Design (high stakes) outputs at financial or legal implications should it use input by human. Overwriting would not be embedded. Possibly at the automated disclaimers inform users there is potential problems with the automation outputs.

7. Industry reactions

OpenAI's publication of their research has ignited a discussion between AI creators, regulators, and business customers.

The developers see Opens a reason to re-evaluate evaluation techniques. "We can't just reward fluency anymore," one AI engineer said, "we have to reward humility."

Indeed, regulators are paying attention. The EU's AI Act prescribes a transparency requirement for high-risk uses -- and hallucinations could definitely be high-risk if high-risk e.g. health, finance, or legal services are implicated.

Enterprises want a safer system. Many seek product design guardrails for AI to mitigate regulatory risks. A recent Gartner survey indicated that adoption was struggle with hallucination with 63% of companies trialling generative AI tools, and governance issues generating risk concern.

8. Next steps: From plausibility to trust

Overall, OpenAI's efforts imply that the focus of the AI industry has to move again. For these 4 years, models were interpretable to plausibility; now the focus has to be truth, or established unfounded-ness.

That entails re-learning the models with new goals, re-defining benchmarks and establishing better guardrails in every product. It also means user log-in: anything that is the most plausible does not mean the conclusions of AI are correct.

As one researcher said of observations: “it’s not false - they’re just incomplete - I’m hypothesizing. Our job is not to reward potential blunder, but with accuracy.”

9. Conclusion

Hallucinations are on one of the most important threats of AI, and after OpenAI’s most recent publication they continued to be “damn Iowa,” for most part this makes them to falsify behavior purposefully. How to frame the dilemma an incentives not mathematical errors, hallucinations are not a wicked problem to be eradicated - but a problem worth systemic change.

In the perspective of businesses, “don’t use AI without safety goggles,” which is above all trust for social capital, regulation, and etc...

Looking ahead, we would be greatly improved training with safety goggles - retrieval system, confidence calibration, verification tracers and human in the loop world design. Short, hallucinations are no contagious rational, but businesses creating safety goggles less odd for today would be the best prepared for the net day.

‍

‍