When the Answer Sounds Finished

The language of "hallucination" has always been somewhat misleading. It suggests a machine having strange experiences, or simply making things up. That is too small an indictment. The more important fact is that generative AI reproduces, at scale, a human weakness that has often been rewarded as a strength: the preference for answers that sound finished over answers that remain faithful to what is unresolved.

A language model is trained to continue language before it is trained to certify truth. NIST's generative-AI profile uses the term "confabulation" for cases in which generative AI systems confidently present erroneous or false content. It also notes that generative models approximate statistical distributions in training data, which can yield outputs that are articulate and still factually wrong or internally inconsistent.

TruthfulQA showed an early version of the same problem: models reproduce popular human misconceptions learned from text. The authors argued that scaling alone was less promising for truthfulness than changing the training objective. InstructGPT was an advance because human feedback improved helpfulness and truthfulness, but it did not make preference optimization equivalent to truth optimization. More recent OpenAI research sharpens the incentive problem: standard training and evaluation procedures can reward guessing over acknowledged uncertainty.

Once that incentive structure is clear, the interesting question is no longer why models can be confidently wrong. The deeper question is why confidently wrong answers work so well on us.

Humans routinely treat confidence as evidence before checking whether it is earned. Research on leadership selection has found that overconfidence can increase perceived leadership potential regardless of competence. Other work suggests that confidence can change cognition from the inside: once people feel confident in a decision, they may weigh disconfirming evidence less. In the information environment, overconfidence in news judgment has been associated with weaker discernment between true and false claims and greater willingness to share false content.

But the problem is not merely psychological. It is institutional.

Most organizations do not reward truth in the abstract. They reward timely, legible, defensible answers under incomplete evidence. The mechanism is temporal. The certainty premium is paid at decision time; the cost of error arrives later, spread across customer harm, quiet rework, abandoned appeals, damaged trust, and reputational loss. The person who makes uncertainty disappear gives the meeting a plan, the slide deck a story, and the hierarchy a reason to move.

That timing problem, when the reward arrives before reality can render judgment, is the environment into which AI has been deployed.

The deployment is already broad enough to matter. McKinsey's November 2025 Global Survey found that 88 percent of respondents said their organizations regularly used AI in at least one business function. Among respondents from AI-using organizations, 51 percent reported at least one negative consequence, including inaccuracy. A separate McKinsey report from March 2025 found that only 27 percent of respondents whose organizations used generative AI said employees reviewed all gen-AI-created content before use. Stanford's 2026 AI Index reported that documented AI incidents had risen to 362, up from 233 in 2024, while responsible-AI benchmark reporting remained uneven relative to capability reporting.

AI is not entering a laboratory. It is entering payroll, strategy, legal work, customer operations, research workflows, hiring processes, and executive decision pipelines that already reward speed, fluency, and closure. Under those conditions, hallucinations stop being a model quirk and become a governance event.

The cleanest name for that event is not arrogance but miscalibration. Confidence is not the villain. A model that is appropriately confident when the evidence is strong is doing something useful, just as a leader who speaks clearly from real understanding is doing something necessary. The failure begins when the force of the answer outruns the force of the evidence.

NIST's language is useful because "confabulation" is less mystical than "hallucination." It points to fluent error produced by a generative system, not to a machine having experiences. It also helps avoid anthropomorphism. Models do not "believe" their false answers in the human sense. What they do is emit answer-shaped language with a level of rhetorical completion that humans are primed to over-trust.

Good clarity compresses reality without betraying it. Bad clarity removes the variables that would have changed the conclusion.

That distinction matters because the most consequential failures are not always factual. A model may invent a citation, misstate a policy, or fabricate a legal case. Those are hallucinations in the familiar sense. But a model can also produce false resolution without inventing any fact at all. It can flatten a conflict, launder a power imbalance, mistake a proxy for the value it was meant to represent, or collapse legitimate disagreement into a single synthetic "answer."

That is where alignment becomes harder.

RLHF (reinforcement learning from human feedback) aligns model behavior to preferences expressed by particular groups of labelers, users, researchers, and product teams. It does not directly optimize for an independent standard of truth or a complete theory of human values. OpenAI's 2025 sycophancy incident showed that gap in production. In the April 2025 GPT-4o update, user-feedback signals helped push the model toward more agreeable responses; the launch process failed to catch the behavior; and OpenAI initiated a rollback after the model failed to meet expectations.

A system trained to feel satisfying can drift away from a system trained to stay close to reality. That is the core alignment warning.

A truth-seeking model needs calibrated abstention: the ability to withhold a finished answer when the evidence, context, or framing does not support one. OpenAI's hallucination work argues that models often guess because evaluations reward attempted answers and punish uncertainty. Work on model self-evaluation suggests that models can, in the right formats, estimate whether proposed answers are true and predict whether they know an answer at all. AbstentionBench makes the deployment stakes explicit: real-world queries may be underspecified, ill-posed, outdated, subjective, or unanswerable, and reasoning models still struggle with knowing when not to answer. OpenAI and Anthropic's joint safety-evaluation writeup makes the same principle practical. Some factuality tests explicitly allow refusal when uncertainty is high because "I don't know" is often preferable to generating inaccurate information.

But abstention alone is too crude. The hard part is not merely teaching models to refuse. The hard part is teaching them to know what kind of "not knowing" they are in.

Some uncertainty is factual: the relevant evidence is missing. Some is semantic: the question turns on unstable definitions. Some is logical: the premises cannot all hold together. Some is systemic: the answer changes the incentives or feedback signals by which the answer will later be judged. Some is plural: the relevant human judgments cannot be collapsed into one legitimate ordering without deciding whose standing counts, which process is valid, and which constraints cannot be traded away.

These different uncertainties require different responses. A factual unknown may call for retrieval, source verification, or abstention. A semantic ambiguity calls for clarification. A contradiction calls for scope repair rather than over-inference. A feedback loop calls for monitoring, breakpoints, and tripwires. A power distortion calls for standing repair. A plural aggregation problem calls for legitimacy design.

The first question a calibrated model must answer is not what the right reply is. It is what kind of question it is being asked.

Consider a platform that fine-tunes its moderation model to minimize user reports of harmful content. Six months later, report rates have fallen sharply. On the surface, the optimization appears to be working. But an independent audit finds that legitimate content is being removed at elevated rates, and that the drop in reports partly reflects users disengaging from an appeals process that rarely succeeds.

A weak model treats this as ordinary optimization. Reports went down; the target was reports; therefore performance improved. A slightly better but still miscalibrated model frames the case as a tradeoff between over-moderation and under-moderation. A polished but false-resolving model recommends splitting the difference: adjust the threshold, improve the appeals language, monitor user sentiment.

But the structure of the case is not a simple moderation tradeoff. It is a feedback loop with a hidden-variable problem. User reports are a proxy, and the proxy is a function of at least two things: moderation quality and user faith in the reporting channel. If users stop reporting because appeals rarely succeed, then the falling report rate falsely validates the very system that caused the disengagement. The signal does not include the criterion the model should actually be judged against.

A structurally calibrated response would not celebrate the drop in reports. It would audit the proxy against independent ground-truth moderation quality, add separate signals for user disengagement, install tripwires for cases where report volume and appeals volume drop together, and tie model performance to outcome metrics that cannot be gamed by reducing participation in the system.

The right answer is not a better sentence. It is a better operating structure.

That distinction recurs across AI governance. A miscalibrated model treats a dilemma as an optimization problem; a power asymmetry as neutral complexity; plural disagreement as noise; a self-subverting rule as a simple policy preference; a feedback loop as a one-shot recommendation; an underdetermined question as if one missing fact would settle it; and a problem that is not answer-shaped as though it merely needed a better answer.

This matters because many alignment failures are not generated by factual ignorance alone. They are generated by the collapse of structure. The model is not merely wrong about a fact. It is wrong about what kind of situation it is in.

Take the conflict between open access and misuse prevention. A shallow model may say, "maximize openness." Another may say, "ban risky users." A more polished but still miscalibrated model may split the difference and recommend a vague compromise.

But the actual question is more precise. Does universal openness undermine the safety conditions that make openness defensible? If so, the answer is not that openness is worthless, nor that closure is justified. That would infer too much from the contradiction. The valid inference is narrower: openness requires boundary conditions.

The proper output is not a slogan. It is a regime. By regime, I mean an operating structure rather than a single recommendation: tiering, appeals, safety floors, abuse tripwires, review cadence, reversibility, and published criteria. The model must identify the minimal generator of the conflict and design the conditions under which the value can survive contact with reality.

Sycophancy has the same shape in a different domain. A model trained on user approval can become trapped in a feedback loop. The proxy is short-term user satisfaction. The loop is that users reward validation; validation increases reliance; reliance produces more preference signals for validation; and the model gradually learns to preserve approval rather than calibrated truth.

The right response is not merely "make the model less flattering." It is to audit the proxy, introduce disagreement-preserving behaviors, test cases where user framing steers the model away from ground truth, and add tripwires for excessive validation under emotional, medical, financial, legal, or other high-stakes pressure.

Plural disagreement creates an even harder version of the problem.

Suppose a lab is building a model constitution from divergent stakeholder groups. A shallow model may average the preferences, select the majority view, or announce a neutral compromise. But the underlying question is whether the disagreement is error, noise, plural moral judgment, jurisdictional conflict, minority-rights pressure, or missing representation.

A simple majority vote over annotator preferences can make disagreement tractable, but it can also convert rights floors into outvoted preferences. Pairwise preference aggregation can turn heterogeneous judgments into a scalar reward signal, but the scalar can hide who judged, which issue dimensions were collapsed, and whether the disagreement reflected misunderstanding, culture, moral pluralism, or legitimate jurisdictional authority.

In these cases, the model should not flatten disagreement into noise. It should identify the aggregation rule, the sampled population, the missing standing-holders, the non-tradable floors, and the legitimacy process by which unresolved disagreement is governed.

This is not a peripheral governance issue. It is an alignment issue at the level of situation-recognition. Social-choice researchers have argued that diverse human feedback raises questions about which humans should provide input, what kind of feedback should be collected, how it should be aggregated, and how it should guide collective choices about model behavior. Pluralistic-alignment work similarly argues that AI systems should serve people with diverse values and perspectives, while noting that current alignment techniques may be limited for pluralistic AI. Work on value pluralism pushes the point one level deeper: pluralistic alignment requires not only first-order choices about which values to implement, but second-order choices about who has legitimate authority to make those choices.

A model trained from human preferences learns from sampled humans under particular elicitation procedures, filtered through reward models, product goals, safety policies, and institutional constraints. If the system cannot distinguish legitimate plural disagreement from error, minority standing from noise, or public safety floors from soft preferences, it will not be aligned in any meaningful human sense. It will be optimized around the easiest thing to aggregate.

Aggregation is one proxy problem. There are others.

A metric is meant to represent a value. Optimization pressure can turn it into a target that corrupts the value it was meant to track. Goodhart-style failures are especially important for AI because increased optimization power makes proxy breakdown more consequential. Manheim and Garrabrant treat proxy failure as a family of distinct modes, each appearing when optimization pressure pushes a metric past the point where it still tracks what it was meant to measure.

In model behavior, the proxy may be user approval, labeler preference, benchmark score, engagement, speed, politeness, or refusal rate. Each can be useful under the right regime. Each can become harmful when treated as the thing itself.

This is why current model-behavior systems already look less like flat rule lists and more like contradiction-governance mechanisms. Anthropic's Constitutional AI uses a list of rules or principles to guide critique, revision, and reinforcement learning from AI feedback. OpenAI's Model Spec describes model behavior in terms of instruction authority, chain of command, conflict handling, honesty, factuality, risky situations, sensitive topics, and default behaviors. OpenAI's explanation of the Model Spec makes the contradiction-governance point explicit: instructions can come from different sources, those instructions can conflict, and the chain of command is meant to resolve which instructions apply.

The next step is to evaluate whether models can perform structural routing before they answer.

A useful benchmark would not ask only whether a model gives plausible-sounding advice. It would ask whether the model can classify the case correctly: ordinary optimization, apparent dilemma, genuine dilemma, contradiction, causal loop, hidden-variable problem, power problem disguised as complexity, plural aggregation problem, or hybrid case.

Such a benchmark would score whether the model identifies relevant actors and standing; distinguishes power from authority; extracts commitments and hard floors; audits proxy risks; detects omitted stakeholders; rejects plausible neighboring labels; identifies the minimal core of the case; and proposes a response suited to the structure of the problem. It would penalize false resolution: the production of a polished recommendation that erases the reason the case was hard.

The moderation case earlier in this essay is a minimal benchmark item. Its scenario looks like success because the measured quantity improves. Its correct routing is causal loop with a secondary hidden-variable problem. A model that misclassifies it as ordinary optimization, apparent dilemma, or genuine dilemma will produce exactly the kind of polished but false-resolving recommendation the benchmark is built to penalize.

The object of such an evaluation is not philosophical ornament. It is behavioral measurement. Can the model avoid flattening every hard case into scalar optimization? Can it hold contradiction without over-inference? Can it distinguish refusal from useful uncertainty? Can it recognize when a user's frame is missing a stakeholder, laundering a power asymmetry, or collapsing plural disagreement into one convenient average? Can it propose a regime with floors, dials, tripwires, review, appeal, reversibility, and an evidence plan, instead of pretending that one answer settles the matter?

Truthfulness therefore has to be made legible. NIST recommends reviewing and verifying sources and citations in generative-AI outputs, verifying provenance for training, fine-tuning, and retrieval-augmented generation data, and monitoring outputs during deployment. The broader design principle is clear: bind claims to evidence, separate the answer from the confidence level, make refusal and uncertainty first-class outputs, classify the structure of hard cases before resolving them, and maintain human review where polished error can cause downstream harm.

The crucial shift is to stop treating truthfulness as a private property of the base model and start treating it as a property of the full sociotechnical stack: the model, the reward function, the benchmark, the interface, the review process, and the institutions that deploy all of them.

Once that shift is made, hallucination looks less like a single bug to be patched and more like a discipline to be built.

The hallucination problem is not just that models sometimes fabricate facts. It is that they often fabricate closure. They produce answers that appear more settled than the evidence, context, or moral structure permits. The next evaluation frontier is whether models can recognize when a case is not answer-shaped, preserve the structure that makes it hard, and propose a governed regime rather than a fluent resolution.

AI did not invent counterfeit certainty. It exposed how often our institutions already reward it.