# How to stop AI hallucinations in a production app

*By Roberto Lazar, founder of Dock30 · Published 2026-08-11 · Updated 2026-08-11 · 8 min read*

Practical ways to cut LLM hallucinations in a live app: grounding with RAG, forced citations, output validation, abstention, evals, and human review.

You cannot stop an LLM from hallucinating. What you can do is build the system around it so a wrong answer rarely reaches a user. In production that means five habits working together: ground the model in your own data, make it cite sources you can check, force structured output and validate it, give the model a way to say it does not know, and run evals plus human review on anything high stakes. The model is a probability engine, not a database, so the reliability lives in the plumbing you wrap around it.

We have shipped this layer into a lot of client products since we started adding AI features, and the pattern below is what actually holds up after launch. None of it is exotic. Most of it is the unglamorous work teams skip when a demo looks convincing, then rebuild in a hurry once the support tickets arrive.

## Why models hallucinate in the first place

A language model predicts the next token that best fits the text so far. When it knows the answer, that prediction is right. When it does not, it still predicts something, because producing plausible text is the only thing it does. There is no internal flag that quietly marks a sentence as a guess.

OpenAI's research team put a sharper point on it: models hallucinate partly because training and evaluation reward confident guessing over admitting uncertainty. Under grading that scores an answer as simply right or wrong, a guess that might be right beats an honest "I do not know," which scores as a miss, so the model learns to always answer, per [OpenAI's "Why language models hallucinate"](https://openai.com/index/why-language-models-hallucinate/). Errors get treated as no worse than abstentions, and the model optimizes accordingly.

The rates are lower than the scare headlines and still too high to ignore. On Vectara's updated grounded-summarization benchmark, where the model is handed a document and asked to summarize only what it contains, the best models land around **3 percent** and several reasoning models still clear 10 percent, per [Vectara's hallucination leaderboard](https://www.vectara.com/blog/introducing-the-next-generation-of-vectaras-hallucination-leaderboard). That is the easy version of the problem, with the facts sitting right there in the context. Ask a model to answer from memory and the floor drops out. It shows up in code too: a 2026 study re-testing frontier models found they still invent package names that do not exist, the kind of confident mistake that becomes a supply-chain risk the moment someone installs the hallucination, per [this arXiv analysis](https://arxiv.org/abs/2605.17062).

## Ground every answer in your own data

The single biggest reduction in hallucinations comes from not asking the model to remember. Retrieval-augmented generation (RAG) pulls the relevant passages from your own documents at query time and drops them into the prompt, so the model summarizes text in front of it instead of reconstructing facts from training. A model quoting a paragraph you handed it is a very different risk profile from a model recalling one.

We treat this as the default for any feature that answers questions about a customer's data. The mechanics, meaning chunking, embeddings, the vector store, and when retrieval is even the right tool versus fine-tuning, we covered in [RAG vs fine-tuning](/blog/rag-vs-fine-tuning). The reliability point here is narrower: retrieval turns an open-book memory test into a reading-comprehension task, and models are far better at the second one. Grounding, along with the evals further down, is most of the [reliability work we do when adding AI to a product](/services/ai-automation).

## Force citations, then check that they hold

Grounding helps, but a grounded model can still stitch together a claim its sources do not support. So make it cite. Ask for the specific chunk or document id behind each statement, and render answers with those citations attached. In a medical QA study across several current models, a strict citation-enforced prompt measurably suppressed unsupported content compared with free-form answering, per [this 2026 paper in Applied Sciences](https://www.mdpi.com/2076-3417/16/6/3013).

Here is the catch that trips teams up: a citation is not proof. Models cite the wrong passage, or invent a reference id that looks plausible. The citation only helps if your code checks it. After generation, confirm each cited id actually exists in what you retrieved, and where you can, that the quoted span appears in that source. A citation that does not resolve is a hallucination with a footnote, and it should be dropped or flagged, never shown as-is.

## Make the output structured, then validate it

Free text is hard to check. Structured output is not. When a feature feeds anything downstream, a database write, an API call, a UI that expects fields, have the model return JSON against a schema and parse it before anything trusts it. Current models like Claude Sonnet 5 and the GPT-5.6 tiers are good at holding a shape, and providers now offer structured-output modes that constrain generation to valid JSON. You still validate on your side, because "usually valid" is a production incident waiting for the one call that is not.

The pattern we ship on every feature is a schema check with a single retry and a fallback:

```ts
const Answer = z.object({
  status: z.enum(["answered", "insufficient_context"]),
  summary: z.string().max(600),
  sources: z.array(z.string()).min(1), // ids we can verify against retrieval
});

const parsed = Answer.safeParse(JSON.parse(raw));
if (!parsed.success) {
  // retry once with the validation error appended, then a static fallback
}
```

The `insufficient_context` status and the required `sources` array are doing reliability work, not decoration. We go deeper on this validation layer in [adding an AI feature to your existing app](/blog/add-ai-to-your-app).

## Let the model say it does not know

The fix OpenAI's paper points at, giving credit for abstention, is something you can build at the product level whether or not the model was trained for it. Give the model an explicit escape hatch, the `insufficient_context` branch above, and instructions to use it when the retrieved context does not answer the question. Then design the UI so that branch is a first-class response rather than an error. "I could not find that in your documents" beats a confident fabrication every time.

This is mostly a prompting and product-design decision, and it is the one teams resist most, because "I do not know" feels like the feature failing. It is the opposite. A support bot that abstains on 8 percent of questions and is right on the rest is far more useful than one that answers everything and is wrong one time in six. We build this behavior into every [chatbot grounded in a customer's own data](/blog/build-ai-chatbot-website), with a clean handoff to a human when the model steps back.

## Put guardrails around the call

Everything above assumes the model output passes through code you control before it reaches a user or another system. That layer is where the guardrails live:

- Constrain values. If a field can only be one of five statuses, enforce the enum. Do not let free text through.
- Check that citation ids resolve, as above, and strip any that do not.
- Filter obviously unsafe or off-topic output in both directions.
- Keep a kill switch. One flag that routes to a static fallback when the model misbehaves has saved more launches than any prompt tweak.

The rule we repeat in code reviews: **never let raw model text flow straight into a system that trusts it.** A model can return malformed JSON into a code path that assumed it could not, and it will, on the day nobody is watching.

## Run evals so regressions do not ship

You cannot improve what you do not measure, and with LLMs you often cannot even tell it got worse just by looking. A prompt tweak that helps one case can quietly break three others. The habit that prevents this is an eval set: 30 to 50 representative inputs with known-good outputs, run as a test suite every time you change a prompt, swap a model, or bump a dependency.

Load it with your hard cases. Questions whose answer is genuinely not in the data, where the model should abstain. Ambiguous phrasings. The exact failures a user already reported. On the projects we ship, this is the single habit that separates AI features that improve over time from ones that quietly rot. It costs an afternoon to build and pennies to run. Skipping it costs a confused thread every time someone "just tweaks the prompt." When a client asks us to make an AI feature production-ready, setting up grounding and this eval loop is usually the bulk of the job.

## Keep a human on high-stakes outputs

None of the above gets you to zero, so decide where a wrong answer is expensive enough that a person signs off. Money moving, medical or legal content, anything a user will act on without double-checking: those get human review, or at least a confidence gate that routes uncertain outputs to a queue. Low-stakes outputs like a draft, a suggestion, or an internal summary can ship straight through, because the reader already treats them as a starting point.

The trap is spending the review budget evenly across everything. Tier it instead. Auto-approve the safe outputs, sample the middle band, and gate the ones where being wrong actually costs something. That keeps a human where they add value and out of the way where they do not.

If you have backend capacity in house, none of this is exotic and you should build it yourselves, since it is a week or two of focused work. If you would rather have it built right the first time, this reliability layer is what our [AI and automation team](/services/ai-automation) sets up daily. We work fixed scope, with the exact price and delivery date agreed in writing before we start ([projects from EUR 350](/pricing/project)), plus 30 days of free support after launch. If you have an AI feature that is confidently wrong too often and want a second pair of eyes, [book a free 15-minute call](https://calendly.com/dock30/15min) or reach us through the [contact page](/contact). Bring the answers it got wrong. Those are the most useful thing you can show up with.

## Frequently asked questions

**Can you completely stop an LLM from hallucinating?**

No. A language model predicts likely text, so it will sometimes produce confident wrong answers no matter how good the model is. What you can do is build the system around it so those answers rarely reach a user: ground responses in your own data, validate the output, let the model abstain, and review high-stakes results. In production, reliability comes from the plumbing, not from a perfect model.

**Does RAG stop hallucinations?**

RAG reduces them a lot, because the model summarizes documents you hand it instead of recalling facts from memory, which it is much better at. It does not eliminate them, since a model can still make a claim its sources do not support. Pair retrieval with forced citations that your code actually verifies and the failure rate drops further.

**How do I make an AI feature admit it does not know?**

Give the model an explicit path to abstain, such as an insufficient_context status in its structured output, and instruct it to use that path when the retrieved context does not answer the question. Then treat that response as a normal successful outcome in your UI rather than an error. A feature that says it could not find something is more trustworthy than one that guesses.

**What hallucination rate should I expect from current models?**

On grounded tasks where the facts sit in the context, the best 2026 models hallucinate around 3 percent and some reasoning models still exceed 10 percent, per Vectara's leaderboard. Asking a model to answer from memory instead of provided context raises that rate a lot. Plan for a nonzero rate and design review around the cases where being wrong is expensive.

**How do I stop a prompt change from breaking things that worked?**

Build an eval set of 30 to 50 representative inputs with known-good answers and run it as a test suite on every prompt edit, model swap, or dependency bump. Include cases where the right answer is to abstain, plus any failure a user already reported. Without evals you cannot tell that a tweak which fixed one case quietly broke three others.

---

Written by Roberto Lazar, founder of Dock30. Book a call: https://dock30.com/contact