You deployed your AI agent. The API calls are cheap. Token costs are logged. You're watching costs in a spreadsheet.
The number that doesn't show up in the spreadsheet: the downstream cost of a hallucinated output.
An LLM generating wrong text in a chatbot is annoying. An LLM fabricating a payment amount in a dunning email, inventing a ticket status in a dev briefing, or confidently filling in company funding data it doesn't have, those are different problems. The damage isn't the API call. It's what happens after the output leaves your system.
Here are three categories I've hit in production across real agents, with the patterns I now use to handle them.
Category 1: Fabrication in Structured Output
Structured output hallucination happens when a model fills in fields it has no data for. Instead of returning null, it invents something plausible.
Scout is a sales research agent that takes a company name, scrapes five sources (website, Google News, LinkedIn, Crunchbase, job listings), then calls Amazon Nova Lite via Bedrock to synthesize a structured briefing. The synthesis prompt produces a JSON object with fields like funding.total_raised, founded, headquarters, key_people.
Early versions had a subtle problem. When only two sources returned data, the model would still populate funding.total_raised with a number it made up from training data. A user would walk into a sales meeting, cite the funding figure, and be wrong. Not "I don't know" wrong. Wrong with confidence.
The first fix was explicit instruction in the system prompt:
SYNTHESIS_PROMPT = """...
## Rules
- Only include information from the source data. Never fabricate.
- If a field has no data, use null or empty array.
- Confidence: 0.9+ if 4+ sources succeeded, 0.7+ if 3, 0.5+ if 2, below 0.5 if only 1.
"""
The second fix was enforcing this at the schema level with Pydantic. All optional fields are typed Optional:
class FundingInfo(BaseModel):
total_raised: Optional[str] = None
last_round: Optional[str] = None
investors: List[str] = Field(default_factory=list)
class Briefing(BaseModel):
company_name: Optional[str] = None
summary: str
founded: Optional[str] = None
headquarters: Optional[str] = None
funding: Optional[FundingInfo] = None
confidence: float = 0.0
If the model returns a value where None was expected, Pydantic accepts it (it can't know the source was missing). But it does reject structurally invalid output. And the confidence field, set by the model based on source count, lets the UI warn the user when data coverage is thin.
The third fix is what surfaces the problem before a user acts on it: failing gracefully when the model returns non-JSON:
try:
text = raw_text.strip()
if text.startswith("```
"):
lines = text.splitlines()
text = "\n".join(lines[1:-1] if lines[-1].strip() == "
```" else lines[1:])
briefing_data = json.loads(text.strip())
return Briefing(**briefing_data)
except json.JSONDecodeError as e:
logger.error(f"Failed to parse Nova response as JSON: {e}")
return Briefing(
summary=f"Data extracted but synthesis JSON parse failed: {e}",
confidence=0.0,
)
The markdown fence stripping is not an edge case. Nova Lite added `json fences on roughly 15% of responses in testing despite "no markdown fences" in the prompt. If you don't strip them, you get a JSONDecodeError every time.
Category 2: Confident Wrong Answers
RAG (retrieval-augmented generation) is supposed to fix this. Give the model the relevant data, it answers from that data, not from training weights. In practice, RAG reduces this problem significantly but doesn't eliminate it.
DevContext is a developer briefing agent that aggregates GitHub pull requests, Google Calendar events, and Slack messages in real time via tool calls. The system prompt is explicit:
`plaintext
CRITICAL RULE: You MUST call tools before responding to ANY question about work, PRs,
meetings, messages, schedule, or developer context. Never generate a text response
about these topics without first calling the relevant tool(s). The tools handle their
own error states -- always invoke them.
`
The problem: a model with strong training data about GitHub, Jira, Linear, and common team workflows will sometimes answer a question about "what PRs do I have open" without calling the tool, because it has absorbed patterns that make a plausible answer easy to generate. The stronger the model, the more likely this is. GPT-4 was worse for this than Gemini Flash in our testing.
The mitigation is maxSteps combined with tool-first enforcement:
`typescript
const result = streamText({
model: getModel(),
system: SYSTEM_PROMPT,
messages,
tools: allTools,
maxSteps: 5,
stopWhen: stepCountIs(5),
});
`
And each tool returns explicit not_connected status when credentials are missing, rather than letting the model fill in:
`typescript
try {
token = getAccessTokenFromTokenVault();
} catch {
logAudit("github", "Token Vault Exchange", "No GitHub token -- service not connected", "error");
return {
status: "not_connected",
message: "GitHub is not connected. Visit /dashboard/permissions to connect it.",
};
}
`
The key design here: the tool returns a structured error. The model then relays that error to the user. It does not fabricate a list of PRs when the token call fails.
If you let tools fail silently (returning an empty response or throwing an exception the model doesn't see), the model fills the gap with plausible content from training data.
Category 3: Format Hallucination
Format hallucination is underrated. The model returns technically accurate information, but in the wrong format, with extra fields, or with fields renamed. Your parser fails. Downstream code reads stale data. The user sees nothing, or worse, sees a partial result that looks complete.
Rebill sends dunning emails using template substitution. The email templates use double-brace placeholders: {{customer_name}}, {{amount}}, {{product_name}}. These are filled at send time from Stripe webhook data:
`typescript
body: `Hi {{customer_name}},
We noticed that your recent payment of {{amount}} for {{product_name}} didn't go through.
Please update your payment method to keep your subscription active:
{{update_payment_link}}
Thanks,
{{company_name}},``
The templates are static here, which is the right call for a dunning system. But in an earlier version I experimented with AI-generated personalized variations. The model would sometimes return {customer_name} (single braces), {{ customer_name }} (spaces), or [CUSTOMER_NAME] (brackets). The substitution regex would miss these, and the email would go out with a raw placeholder in the subject line. Not a hallucination in the traditional sense. A format hallucination: wrong structural output that causes silent failure.
The fix for AI-generated content in templating contexts is validation before use:
`python
import re
REQUIRED_PLACEHOLDERS = ["{{customer_name}}", "{{amount}}", "{{update_payment_link}}"]
def validate_template(template: str) -> bool:
for placeholder in REQUIRED_PLACEHOLDERS:
if placeholder not in template:
return False
# Catch common format hallucinations: single braces, spaced braces, brackets
suspicious = re.findall(r'{[^{].?[^}]}|[.?]', template)
if suspicious:
return False
return True
`
If the model generates a template that fails this check, reject it and retry once. If it fails twice, fall back to the static default. Never send the broken version.
Production Patterns That Work
Temperature 0.1, not 0. Zero temperature causes repetition loops in some models, particularly when the context is long. 0.1 gives enough variation to avoid degenerate outputs while keeping factual tasks grounded.
`python
inferenceConfig={
"maxTokens": 2048,
"temperature": 0.1,
}
`
Explicit null instructions. "If a field has no data, use null or empty array" in the prompt does reduce fabrication. It doesn't eliminate it, but it shifts the distribution. Combine with typed schemas that accept null.
Pydantic validation on every output. When the model returns JSON, parse it through your schema immediately. Don't access raw dict keys downstream. Briefing(**briefing_data) will raise a ValidationError if required fields are missing. Catch it, log it, return a degraded result rather than crashing or silently passing bad data.
Confidence scoring. Scout's confidence field is set by the model based on source count: 0.9+ for 4+ successful sources, 0.5 for 2. This isn't perfect because the model is scoring its own output, but it surfaces a real signal. When confidence is below 0.5, the UI shows a warning. Users stop treating low-confidence results as authoritative.
3-tier fallback modes. Scout runs three modes depending on what's available:
`python
if app_settings.mock_mode:
# Dev mode -- no API keys needed
from backend.extractors.mock import MockWebsiteExtractor as WebsiteExtractor
from backend.synthesis.mock_briefing import mock_synthesize_briefing as synthesize_briefing
elif app_settings.nova_act_api_key:
# Full browser automation via Nova Act
from backend.extractors.website import WebsiteExtractor
from backend.synthesis.briefing import synthesize_briefing
else:
# HTTP fallback -- real data via requests + Bedrock synthesis
from backend.extractors.http_website import HttpWebsiteExtractor as WebsiteExtractor
from backend.synthesis.briefing import synthesize_briefing
`
The mock mode doesn't just skip synthesis. It returns a hardcoded Briefing object with known-good data. This lets you test the full rendering stack without touching a live model.
What Doesn't Work
"Just tell the model to be accurate." Prompt-only approaches have a ceiling. The model will comply until it doesn't have data, at which point the directive to be accurate competes with the directive to be helpful. Helpfulness often wins.
"Use a better model." Switching from Lite to Pro reduces fabrication rates. It doesn't eliminate them. A more capable model is also more capable of generating convincing fabrications.
"Retry on bad output." Retrying a malformed JSON response at temperature 0.1 will often produce the same malformed response. If the failure is structural (wrong format, missing fence stripping), retry won't fix it. Parse first, retry only when the error is stochastic (e.g., a failed tool call, not a format mismatch).
The Measurement Problem
The hard part: you often don't have ground truth. You can't compute a hallucination rate without knowing what the correct answer was.
The proxies I use:
-
json.JSONDecodeErrorandValidationErrorcatch rates. These are a lower bound on format hallucinations. - Fields left null vs. fields filled in, tracked over time. A sudden spike in filled-in
funding.total_raisedwhen source quality drops is a signal. - Confidence score distribution. If your agent's average confidence drops from 0.75 to 0.45 with no change in query mix, something changed in the model or your extraction pipeline.
- Manually auditing a 2% sample weekly. Slow, but catches things the other three miss.
None of these is a complete solution. The measurement problem is real and unsolved at the system level. The best you can do is instrument your failure modes and build audit habits early.
Structured output plus schema validation plus explicit null instructions plus confidence scoring gets you most of the way there. The combination is more robust than any single technique. And critically, it degrades gracefully: when things go wrong, the user sees low confidence or "data unavailable" rather than wrong data served with false certainty.
I build production AI systems with hallucination guardrails baked in. If your agents are generating outputs you can't fully trust, I'd like to hear about it. astraedus.dev
