AI Agent Making Stuff Up? 5 Fixes That Stop the Hallucinations

The surprise is that the invented part is rarely the whole answer. It is one clause inside an otherwise perfect reply, so the transcript reads clean and the fake policy number sails straight past review.

Take a support agent answering from 40 help-center articles at about 200 questions a day. Last week it quoted a returns window of 45 days and a policy number that appear in none of the 40. Formatted like every correct answer beside it.

The model did not malfunction. Nothing sat between the answer and the customer, so the first check on the output was the person who received it.

The twelve fixes are ranked by the failure each one catches and by what must exist before it can catch anything. The first lands this week. Every one of them rests on the same 50-question golden set, built before any threshold gets tuned.

Fix	Layer	What it catches	Setup effort
1. Golden set	Foundation	Nothing itself: it makes every threshold tunable	Half a day with the support lead
2. Cleanlab TLM	Live score	Confident wrong answers with no clean reference	One developer, one afternoon
3. HHEM	Live score	Facts added beyond the retrieved articles	Half a day per content type bar
4. Patronus Lynx	Categorize	Says what kind of error, not just pass or fail	A day, RAG agents only
5. Chain-of-Verification	Heavy pass	One fabricated claim inside a faithful answer	Days, and many extra model calls
6. Grading agent	Quick check	Obvious fabrications shipping today	One afternoon
7. Ragas or DeepEval	CI	A future prompt change that raises fabrication	A day to wire into the pipeline
8. NeMo Guardrails	Enforce	Checks that only live in the prompt text	A week, Colang learning curve
9. Weave or Phoenix	Per step	The tool result that poisoned the final answer	A day on a multi-step agent
10. Opik or Phoenix alerts	Production	Detection that only runs while someone watches	One to two days
11. Schema validation	Tool args	Invented order numbers and dates	Half a day per tool
12. Retry cap	Fallback	The endless retry that ends in a guess	An hour

1. Build a 50-question golden set first

Fifty questions. That is the whole setup: 50 questions your support agent has actually been asked, each one paired with the answer your support lead confirms is correct.

Pull them from real chat logs, not from a brainstorm. The support lead writes the answers, because the model grading itself proves nothing.

Ten of those 50 are questions the agent has already gotten wrong, and that is deliberate. A set of easy questions passes whether or not your fix did anything.

With the old failures sitting in the set, the score drops when a fix breaks and climbs only when it works.

Write the score down each time you change a prompt, swap a model or touch retrieval, with the date beside it. One number tells you almost nothing. Ten numbers in a row show a trend, and that trend is what tells luck from progress.

When a tweak makes answers vaguer instead of wrong, a structured prompt formula often fixes the cause.

Why it is first: every cutoff later in this list gets its value from these 50 questions.

The 0.85 and 0.70 pass bars, the Cleanlab cutoff, the 0.7 consistency line and the 0.1 overlap line all rest on it, because detector scores are not true probabilities.

2. Score every answer with Cleanlab TLM

A score that shows your agent is wrong will not, on its own, stop the next wrong answer from reaching a customer. Scored answers need a gate.

The cheap version costs nothing you cannot undo: add a second call that rates the answer and routes it, and leave the agent's own prompt alone.

Cleanlab's Trustworthy Language Model rates each answer from wrong to right, and you decide what each band means.

Low scores get blocked, middling ones get regenerated, borderline ones go to a person. That is one extra model call per answer, so 200 daily questions become 400 calls a day.

The whole setup is a short afternoon if you have sample answers to calibrate against. Pick a starting cutoff, run all 50 golden questions through the scorer, then slide the cutoff until the 10 known-bad answers land below it and most good ones land above.

Do not use the cutoff that looks right to you. TLM entries have no calibrated meaning, so a 0.8 is not "80% correct." It is just a spot on the scale. Tune against the golden answers and that number means something specific to your agent.

The payoff: a working safety net one developer can put in place without writing a single prompt, and a cutoff built on real answers, so choosing to block one is a call you can defend.

3. Add a cheap faithfulness check with HHEM

That caught a company-cited answer on a question with no company name in the help articles.

The check compares the retrieved help articles against the draft reply and returns a match score from 0 to 1, so a sentence with details from nowhere sinks the whole score.

Catching an unsupported claim early mirrors the source-checking habits that keep answers grounded in research work too.

Vectara's HHEM does this in one pass; the open HHEM-2.1-Open sits on Hugging Face and Kaggle, while HHEM-2.3 is the paid version reachable through an API.

That's the programmatic API: a way for one piece of software to send a request to another over the internet.

Pass bars: legal and medical replies need 0.85 or higher. General help content needs 0.70 or higher. Use the paid model for this, because Vectara's own tests show the free one scoring lower.

The fix to apply: a fluent reply that looks right can score badly on an exact string match. Rebuild the score over the 50 golden questions, sorted by content type, before you trust any cutoff.

4. Name the failure type with Patronus Lynx

A made-up answer is a hallucination. That's the plain name for text the model produces that sounds correct and is not. A score tells you one arrived.

It never tells you which kind of mistake it is. When the same one returns week after week, the label is what moves the work forward.

Patronus Lynx is a detection model built for exactly that job. It sorts failures into eight hallucination types, including entity errors, predicate errors, circumstance errors and reasoning that does not match the conclusion.

It needs the retrieved context to check against, so it fits agents that answer from documents.

The cluster is the point. A run of entity errors usually starts in how documents were split and retrieved. A run of predicate errors usually points at the prompt. Those are different fixes, and the taxonomy tells the two apart before anyone edits anything.

Side by side: reading a week of flagged answers and tagging each one by hand takes about two hours; Lynx labels them in roughly 8 minutes, and checking the labels against the 50-question golden set takes about 25 more.

5. Run Chain-of-Verification as the heavy pass

Some answers survive every cheap check and still contain one invented clause. That failure is multi-claim: the shipping window and the restocking fee read straight from the 40 articles, and the return address in the same paragraph appears in none of them.

Chain-of-Verification exists for exactly this, and nowhere else.

For each claim it writes a verification question, looks up the answer in the retrieved passages alone, then rebuilds the reply from those looked-up answers.

Each claim triggers its own lookup, so a 200-question day can mean several calls per reply. Reserve it for refunds, policy exceptions and the 0.85 legal and medical bar.

Never let the model reread its own draft instead. Unaided self-review repeats the invented address back with more confidence.

Hand the model the passages, not just the answer, or the pass costs calls and catches nothing.

Spend it only where the cheap layers still miss, and a fabricated clause stops reaching a customer.

6. Put a grading agent on the output

Writing a separate check for every rule you care about is the tedious part, and it never keeps up. A single grader prompt covers most of it.

The grader reads the answer, returns pass or fail, and names the offending sentence. It cannot rewrite anything, so it stays a check rather than a second author.

The catch is the context. The grader must receive the same articles the answering agent had. A grader without them marks correct answers as fails, and someone switches it off within a week.

Feeding it those articles costs one extra model call per answer, so the 200-question agent goes to 400 calls a day. Each result is logged with its question, and the golden set grows from real failures rather than invented ones.

That is a check one developer can ship the same afternoon, and it catches the invented policy number while the heavier verification pass is still being built.

7. Add faithfulness tests with Ragas or DeepEval

That prompt produces a test file you commit. Ragas scores how well each answer sticks to the passages retrieval returned; DeepEval does the same job for hallucination testing, and both are open source.

That's the code is public and yours to edit. Either runs in your build pipeline before a release.

Run it on every pull request. Fifty fixed questions batch in well under a minute, so cost stays flat no matter how much live traffic the agent sees.

Fail on the drop against the recorded score, not on an absolute number at first. The first absolute number is always wrong.

What you should see: a red build naming the question that slipped and the sentence that broke faithfulness.

The failure this catches is quiet: a new prompt or model version raises fabrication while every old test still passes. It becomes a caught regression, not an incident debugged after.

8. Block bad output with NeMo Guardrails

A line in the system prompt saying never invent a policy number is not a guardrail.

Anyone can delete it in an edit nobody reviews, and the model can ignore it on the answer that matters.

Put the rule outside the prompt, in code, in front of the response instead.

NVIDIA NeMo Guardrails is free to install and lets a team write output rails that warn or block when an answer fails a hallucination or fact check.

The policy sits in files a reviewer can read and diff, so a rule change passes through review like any other code.

The price is Colang, the guardrail's own small language, which takes a day or two to get used to.

The self-check versions also add a model call per answer, taking the support agent from 200 calls a day to 400.

That fits a team already on NVIDIA tooling, or one that needs a blocking rule an auditor can inspect.

The threshold: set the block line from the golden set, never from a default. Tune it so the agent's 10 known-bad questions block and correct answers still reach customers. Then the stop holds even after someone rewrites the prompt at 5pm on a Friday.

9. Score every agent step, not just the end

The final answer reads fine. That's the problem. You have read the same correct-sounding paragraph four times and still cannot say which step introduced the wrong order number, because the mistake entered mid-conversation and the last message laundered it clean.

A rail on the output judges that last message and passes it. Nothing checks the tool result in turn two or the reasoning that followed. A multi-turn agent fails in the middle, and a final-response check cannot see the middle.

W&B Weave's HallucinationFree Scorer and Arize Phoenix's HallucinationEvaluator both apply at each agent step, not only the end.

Each dashboard shows the full execution trace, so the step that carried the bad fact is visible instead of guessed. Phoenix also prints a confidence score beside its label.

Watch for: scores across steps only compare if every step is judged against the same 50-question golden set. Change the reference between runs and the trace shows movement that means nothing.

The payoff is root cause. You fix the retrieval chunk or the tool call that broke, rather than rewriting the final prompt and hoping. The invented order number gets traced to the step that made it up.

10. Gate production traffic and alert on spikes

A production gate is the layer that stops a bad answer before a customer sees it and pages someone when the blocked rate climbs. Comet Opik sets quality gates that block responses above a hallucination threshold.

Arize Phoenix monitors live traffic and alerts on high-risk outputs for immediate review. W&B Weave scores responses inside the serving pipeline, so the check runs before the answer goes out.

Blocked, not errored: a caught answer routes to a human queue instead of returning a failure page. Those blocks are counted, and the blocked rate is the number to watch.

Alert on its week-over-week trend, not a fixed percentage, because a steady 4 percent and a jump from 1 to 4 mean different things.

The gate is where the tuned bars finally decide live traffic: 0.85 for legal and medical content, 0.70 for general knowledge, the TLM cutoff and the retry cap all in force at once.

Score the bad answer at each step, block it here, and the guardrail holds while the person who built it sleeps.

11. Validate tool arguments with a schema

A message reaches checkout reading "refund order 4471" when no such number was ever spoken by the customer. Every prose check on that turn passed clean. The invented digits were never prose. They were tool arguments, and nothing compared them to anything.

Define each tool with JSON Schema or OpenAPI and the arguments arrive as typed fields, so an order number must match a pattern and a date must parse.

Model Context Protocol: MCP is an open standard for handing a model a list of tools. Each tool ships a written description of its inputs, and the server is expected to reject anything that fails it.

Validate the arguments on their own. Amazon Bedrock Guardrails' content filters skip the tool input and the tool result, so a clean content verdict tells you nothing about the number inside it.

Reject the bad call and send the validation error back as the next turn, never a silent default that turns a wrong argument into a wrong action.

When the user names no order, the agent should be spending those calls asking for the number instead of filling one in. Schema validation is what forces that ask, and it runs before any model-based check is consulted.

12. Cap retries and hand off to a human

That prompt is the exhausted case. Validation sends a rejected call back to the model, and with no ceiling it keeps spending calls and seconds before guessing anyway. Set max_retries=2, which is the first call plus two validation retries, then stop.

Pydantic AI runs that pattern and raises UnexpectedModelBehavior once the budget is spent, so your code catches the exception by name instead of surfacing a 500.

The handoff carries the trace, so the person opens at the failing step. The budget of 2 comes from watching how often the golden set's 10 known-bad questions still pass.

Frequently Asked Questions

How much does the extra detection cost?

Each automated pass adds one model call per answer on top of the answering call. For the 200-question example, one pass takes you to 400 calls a day, and every further pass adds another 200. Budget them like any other per-task cost.

Do I need to be a programmer to set these up?

The grading agent and the retry cap need almost nothing beyond a prompt and a try/catch. Cleanlab TLM, HHEM and the CI tests need a developer who can wire an API and a pipeline. The golden set itself needs the support lead, not code.

Can I just write a strong prompt and skip all this?

A prompt helps, but it cannot be enforced and it cannot be measured. A check in the prompt is advice the model can ignore; a check in code blocks the response. Your golden set is what tells you whether the prompt actually worked.

What is the difference between grounding and faithfulness checks?

Grounding compares the answer's facts against the retrieved context, which is what HHEM does. Faithfulness checks whether each claim is supported by the retrieved passages, which is what Ragas and DeepEval test. Both need the retrieval step to supply the source.

Is my customer data safe in these tools?

On free and personal plans, chats may be used to improve the company's models unless you switch that off in settings. Never paste passwords, client files, or medical or financial records into a chatbot. Check each vendor's data terms before sending live traffic.

How do I know a fix is working?

The golden set score moves. Record it before the change and after it. If the score rises on the 10 known-bad questions without dropping on the rest, the fix worked. A change nobody measured is a hope.

The order that fits a small team's first sprint

Start with the golden set, because every threshold after it depends on those 50 questions. Then pick exactly one detection layer matched to how your agent works: HHEM if it answers from retrieved documents, Cleanlab TLM if it writes free-form with no fixed source.

Finish the week with the production gate and the retry cap, so a caught hallucination becomes a handoff rather than a wrong answer.

Skip Chain-of-Verification and the per-step dashboards for now, and add them only when the golden set shows the cheap checks missing something that matters.