Why Larger LLMs Give Incorrect Answers in Production

A team I worked with upgraded their support-ticket triage pipeline to a bigger, newer model expecting fewer mistakes.

A team I worked with upgraded their support-ticket triage pipeline to a bigger, newer model expecting fewer mistakes. Benchmarks said it was smarter across the board. Two weeks in, the error rate on a specific ticket category had gotten worse, not better — and worse in a particular way: the new model was wrong just as often as the old one, but its wrong answers now came with confident, detailed justifications instead of the old model’s hedgy “I’m not entirely sure, but…” The support team started trusting the wrong answers more, because they sounded more certain. Nobody had budgeted for that.

That’s the part the “AI hallucinates” conversation usually skips. Bigger models are not simply less wrong — in several measured respects they’re differently wrong, and the difference that matters most in production is confidence, not accuracy. There’s real, recent research behind this, and it points at three separate mechanisms: how models are trained to answer instead of abstain, how their reliability quietly degrades as the input grows even inside benchmarks-passing context windows, and how production conditions differ from the clean single-turn evals every leaderboard measures. None of that means larger models are worse. It means “larger” doesn’t buy you the thing production actually needs, which is knowing when to say “I don’t know.”

TL;DR

→ OpenAI’s 2025 research reframes hallucination as an incentive problem: next-token training and standard evals reward confident guessing over calibrated “I don’t know,” so models learn to bluff.
→ Pretrained models are reasonably well-calibrated; RLHF alignment measurably degrades that calibration, making models more confident without making them more correct — an effect researchers call the alignment tax.
→ Scaling model size doesn’t eliminate this; multiple studies describe larger models producing “confident nonsense” at a similar or greater rate, just more fluently.
→ Chroma’s 2025 “Context Rot” study tested 18 frontier models and found every one degrades as input length grows — well before the context window is full, even on simple tasks.
→ Distractors (content that’s topically close but doesn’t answer the question) hurt accuracy more than irrelevant filler, and the effect compounds as input grows.
→ In Chroma’s tests, Claude models abstained more under ambiguity while GPT models more often produced confident, incorrect answers — model families differ meaningfully in this failure mode.
→ Production adds failure modes benchmarks don’t test: retrieval quality in RAG, long accumulated agent context, and non-deterministic sampling — so a benchmark-topping model can still underperform in your actual pipeline.

The training incentive: models are rewarded for guessing

Start with the most fundamental mechanism, because it explains why this doesn’t go away as models get bigger. OpenAI’s 2025 paper, Why Language Models Hallucinate, argues that the standard training and evaluation setup structurally rewards confident guessing over calibrated uncertainty. A model is trained to predict the next token, and it’s evaluated on benchmarks that score right-or-wrong with no separate credit for a correct “I don’t know.” A model that always guesses will, on average, score higher than one that abstains when unsure — even if the abstaining model is more trustworthy. The behavior isn’t a bug that better data fixes; it’s the predictable output of an incentive structure, and scaling the model up scales the same incentive with it.

This gets compounded at the alignment stage. Research tracing model calibration through the training pipeline — from Kadavath et al.’s foundational 2022 work through more recent studies on the “alignment tax” — has found that base, pretrained models are reasonably well-calibrated: when a pretrained model says it’s 80% confident, it’s right roughly 80% of the time. RLHF, the fine-tuning stage that makes a model helpful and fluent, measurably degrades that calibration. The model comes out more confident and more polished, but not more accurate — confidence and correctness get pulled apart at exactly the stage designed to make the model pleasant to use.

Each stage of the standard training pipeline independently pushes toward confident answers. None of them individually looks like a bug — the compounding is the problem.

Scaling doesn’t fix this because scaling doesn’t change the incentive. A survey published in Frontiers in AI in 2025 makes the point directly: larger models remain capable of “confident nonsense,” and model scaling alone amplifies rather than eliminates hallucination in certain contexts. A bigger model has more capacity to construct a fluent, internally consistent, wrong answer — which is a worse failure mode for a downstream system to catch than an obviously garbled one.

Context rot: accuracy degrades long before the window fills

The second mechanism is specific to production because it’s about what happens once you start feeding a model real, long inputs — RAG context, chat history, tool outputs, agent state — rather than the short, clean prompts most benchmarks use. A 2025 Chroma technical report, deliberately titled Context Rot, tested 18 frontier models including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3, and found that every one of them degrades as input length increases — often well before the context window is close to full, and even on tasks as simple as finding one fact in a document or exactly repeating a string back.

Chroma’s 2025 study found this pattern held across all 18 models tested — accuracy declines with input length even when the task itself doesn’t get harder.

Three details from that report matter for anyone building production RAG or agent systems. First, ambiguity compounds the effect: when the needed fact doesn’t closely match the wording of the question — the realistic case, since users rarely phrase things the way a document does — performance degrades faster as input grows. Second, distractors hurt more than plain irrelevant filler; content that’s topically related but doesn’t actually answer the question pulls accuracy down non-uniformly, and the effect gets worse, not better, as more distractors are added. Third — and this is the one worth sitting with — the researchers found that structurally coherent context (well-organized, logically flowing text) degraded model performance more than shuffled, incoherent text did. That’s the opposite of the intuitive assumption that cleaner input is always easier for a model to use.

There’s also a genuinely useful, model-specific finding buried in that report: under ambiguity, Claude models more often abstained — explicitly stating that an answer couldn’t be determined from the given context — while GPT models more often produced a confident, incorrect answer instead. That’s not a claim that one vendor is unconditionally more accurate; it’s evidence that how a model handles uncertainty under long, messy context is a real, measurable, model-specific property worth testing before you pick what runs in production, not something you can assume from a benchmark leaderboard alone.

Why this specifically bites in production and not in your evaluation

Put those two mechanisms together and the production gap makes sense. Most public benchmarks are short, clean, single-turn, and score right-or-wrong with no credit for calibrated abstention — which is exactly the setup that rewards confident guessing and doesn’t test context rot at all. Production is the opposite on every count: it’s long (RAG context, chat history, tool outputs), ambiguous (real user phrasing rarely matches document wording), and cumulative (an agent’s context grows with every tool call it makes). A model can genuinely top the leaderboard and still be the wrong choice for a pipeline that hands it 40,000 tokens of retrieved documents and expects a single correct fact back.

This is also where architecture choices you already know matter start to compound the problem instead of solving it. Retrieving too many chunks “to be safe” in a RAG pipeline doesn’t just cost more — per the context rot findings, it actively degrades accuracy, especially when the extra chunks are distractors rather than clean irrelevant filler. An agent that accumulates tool results across a long-running task is accumulating exactly the kind of long, structurally coherent context Chroma found hurts performance most. And a model under uncertainty defaulting to a confident guess rather than an abstention is the last-mile version of the training-incentive problem — it’s not a separate bug, it’s the same one showing up downstream.

What actually helps

None of this is a reason to avoid larger models — it’s a reason to design around known failure modes instead of assuming scale solves them. A few things follow directly from the research above. Keep retrieved context tight rather than generous: fewer, more relevant chunks measurably outperform “retrieve broadly and let the model sort it out,” because more chunks means more distractors and distractors compound with length. Test for abstention behavior specifically, not just accuracy — a model that says “I can’t determine this from the given context” on an ambiguous case is behaving correctly even though a naive eval scores it as a miss; the more dangerous system is the one that never abstains. And treat long-running agent context as a liability to actively manage — summarizing or pruning accumulated state periodically rather than letting it grow unbounded, since the Chroma findings suggest that coherent, well-organized accumulated context degrades performance rather than protecting it.

It’s also worth remembering that non-determinism sits underneath all of this: the same prompt can produce a different answer on a different run, for reasons rooted in how these models sample tokens — I’ve written separately about why LLMs give different answers to the same question, and that variance means a single spot-check of a prompt tells you less than it feels like it does. If you’re using AI to generate or review structured output like SQL, the same discipline applies: never trust a green run as proof of correctness, a point I’ve made about why passing tests still ship bad data. And if agents are reading your pipeline’s metadata to answer questions, the context rot findings are a direct argument for curating what reaches them rather than dumping everything and hoping — the same principle behind giving agents metadata access deliberately rather than broadly.

The gotchas nobody warns you about

A benchmark win doesn’t transfer to your pipeline. Public leaderboards are short, clean, single-turn tests. If your production input is long, ambiguous, or accumulated over many turns, the benchmark isn’t measuring the failure mode you’ll actually hit.

More retrieved context is not a safety margin. The instinct to retrieve generously “in case the model needs it” directly works against the research: more chunks means more distractors, and distractors degrade accuracy faster as volume grows.

A model that never says “I don’t know” is a red flag, not a feature. If your eval only scores right/wrong, you can’t distinguish a model correctly abstaining from one confidently guessing wrong — and the second one is more dangerous precisely because it’s harder to catch downstream.

Well-organized context can hurt more than messy context. Counterintuitively, Chroma found coherent, logically flowing input degraded performance more than shuffled text with the same content. Don’t assume tidy prompt construction is automatically safer.

Model families differ on this specific behavior. Whether a model abstains or guesses under ambiguity is a measurable, model-specific property. If reliability under uncertainty matters for your use case, test for it directly rather than assuming it from general capability scores.

The one principle

A larger model gives you more capability, not more honesty about its limits — those are trained in separately, and mostly not trained in at all. The fix isn’t a bigger model or a longer context window; it’s designing the system around the specific, now well-documented ways models fail — tight retrieval instead of generous, explicit testing for abstention, and active management of anything that accumulates context over time. Production doesn’t need a model that’s never wrong. It needs one — and a system around it — that’s honest about when it doesn’t know.