False resolution

Some wrong answers merely fail. Others make an unresolved question feel settled.

Imagine a couple dining with friends. Someone asks for their anniversary date, and one spouse answers without hesitation. The date is wrong. But the answer carries a second message: I remember, I know, I do not need to check. The danger is not only the error. It is the posture of completion.

Asked why they sounded so certain, the spouse replies: "I didn't want to hesitate and give the impression I didn't know."

That is the structure that matters. A wrong answer delivered tentatively invites correction. A wrong answer delivered fluently can close the room around itself. It can change what other people notice, whether they object, and how much friction it takes to reopen the question.

This is the failure pattern I call false resolution: an answer that sounds settled while erasing the uncertainty, conflict, missing evidence, or structural complexity that should have governed the response.

AI hallucination is one version of this pattern. A model invents a fact and presents it fluently. But the larger failure is not factual fabrication alone. It is closure fabrication. The model produces the social and rhetorical form of an answer before the conditions for an answer have been met.

One cause is structural miscalibration. Ordinary calibration asks whether confidence matches accuracy. Structural calibration asks an earlier question: has the system recognized what kind of problem it is answering? A model can sound careful and still treat a conflict over legitimate authority as a problem of ranking preferences, a feedback loop as a one-shot optimization task, or a question missing essential facts as if it merely needed better prose.

False resolution is the family. Hallucination is the factual subtype. Structural miscalibration is one of the family's central causes.

Confidence incentives

The language of "hallucination" has always been somewhat misleading. It suggests a machine having strange experiences, or simply making things up. That is too small an indictment. The more important fact is that generative AI reproduces, at scale, a human weakness that has often been rewarded as a strength: the preference for answers that sound finished over answers that remain faithful to what is unresolved.

A language model is trained to continue language before it is trained to certify truth. NIST's generative-AI profile uses the term "confabulation" for cases in which generative AI systems confidently present erroneous or false content. It also notes that generative models approximate statistical distributions in training data, which can yield outputs that are articulate and still factually wrong or internally inconsistent. E1

TruthfulQA showed an early version of the same problem: models reproduce popular human misconceptions learned from text. The authors argued that scaling alone was less promising for truthfulness than changing the training objective. InstructGPT was an advance because human feedback improved helpfulness and truthfulness, but it did not make preference optimization equivalent to truth optimization. More recent OpenAI research sharpens the incentive problem: standard training and evaluation procedures can reward guessing over acknowledged uncertainty. E2

That incentive structure helps explain why models can be confidently wrong. It leaves another question: why do confidently wrong answers work so well on us?

Humans routinely treat confidence as evidence before checking whether it is earned. Research on leadership selection has found that overconfidence can increase perceived leadership potential regardless of competence. Other work suggests that confidence can change cognition from the inside: once people feel confident in a decision, they may weigh disconfirming evidence less. In the information environment, overconfidence in news judgment has been associated with weaker discernment between true and false claims and greater willingness to share false content.

But the problem is not merely psychological. It is institutional.

Organizations do not have to be indifferent to truth to reward premature certainty. A timely, legible, defensible answer has immediate value. The certainty premium is paid at decision time; the cost of error arrives later, spread across customer harm, quiet rework, abandoned appeals, damaged trust, and reputational loss. The person who makes uncertainty disappear gives the meeting a plan, the slide deck a story, and the hierarchy a reason to move.

AI has been deployed into that gap between the reward for an answer and the consequences of relying on it.

The deployment is already broad enough to matter. McKinsey's November 2025 Global Survey found that 88 percent of respondents said their organizations regularly used AI in at least one business function. Among respondents from AI-using organizations, 51 percent reported at least one negative consequence, including inaccuracy. A separate McKinsey report from March 2025 found that only 27 percent of respondents whose organizations used generative AI said employees reviewed all gen-AI-created content before use. Stanford's 2026 AI Index reported that documented AI incidents had risen to 362, up from 233 in 2024, while responsible-AI benchmark reporting remained uneven relative to capability reporting.

AI is not entering a laboratory. It is entering payroll, strategy, legal work, customer operations, research workflows, hiring processes, and executive decision pipelines that already reward speed, fluency, and closure. Under those conditions, hallucinations stop being a model quirk and become a governance event.

The cleanest name for that event is not arrogance but miscalibration. Confidence is not the villain. A model that is appropriately confident when the evidence is strong is doing something useful, just as a leader who speaks clearly from real understanding is doing something necessary. The failure begins when the force of the answer outruns the force of the evidence.

Models do not "believe" their false answers in the human sense. What they do is emit answer-shaped language with a level of rhetorical completion that humans are primed to over-trust.

Good clarity compresses reality without betraying it. Bad clarity removes the variables that would have changed the conclusion.

Structural calibration

That distinction matters because the most consequential failures are not always factual. A model may invent a citation, misstate a policy, or fabricate a legal case. Those are hallucinations in the familiar sense. But a model can also produce false resolution without inventing any fact at all. It can flatten a conflict, launder a power imbalance, mistake a proxy for the value it was meant to represent, or collapse legitimate disagreement into a single synthetic "answer."

RLHF (reinforcement learning from human feedback) aligns model behavior to preferences expressed by particular groups of labelers, users, researchers, and product teams. It does not directly optimize for an independent standard of truth or a complete theory of human values. OpenAI's 2025 sycophancy incident showed that gap in production. In the April 2025 GPT-4o update, user-feedback signals helped push the model toward more agreeable responses; the launch process failed to catch the behavior; and OpenAI initiated a rollback after the model failed to meet expectations. E3

A system trained to feel satisfying can drift away from a system trained to stay close to reality.

A truth-seeking model needs calibrated abstention: the ability to withhold a finished answer when the evidence, context, or framing does not support one. Work on model self-evaluation suggests that models can, in the right formats, estimate whether proposed answers are true and predict whether they know an answer at all. AbstentionBench makes the deployment stakes explicit: real-world queries may be underspecified, ill-posed, outdated, subjective, or unanswerable, and reasoning models still struggle with knowing when not to answer. OpenAI and Anthropic's joint safety-evaluation writeup makes the same principle practical. Some factuality tests explicitly allow refusal when uncertainty is high because "I don't know" is often preferable to generating inaccurate information.

But abstention alone is too crude. The hard part is not merely teaching models to refuse. The hard part is teaching them to know what kind of "not knowing" they are in.

Some uncertainty is factual: the relevant evidence is missing. Some is semantic: the question turns on unstable definitions. Some is logical: the premises cannot all hold together. Some is systemic: the answer changes the incentives or feedback signals by which the answer will later be judged. Some is plural: the relevant human judgments cannot be collapsed into one legitimate ordering without deciding whose standing counts, which process is valid, and which constraints cannot be traded away.

These uncertainties require different responses. A factual unknown may call for retrieval, source verification, or abstention. A semantic ambiguity calls for clarification. A contradiction calls for identifying exactly where the premises conflict, not inferring more than that conflict establishes. A feedback loop calls for monitoring and clear thresholds for intervention. A power distortion calls for restoring the standing of people wrongly excluded from the decision. A plural aggregation problem calls for a legitimate process for weighing judgments that may remain in conflict.

The first question a calibrated model must answer is not what the right reply is. It is what kind of question it is being asked.

Consider a platform that fine-tunes its moderation model to minimize user reports of harmful content. Six months later, report rates have fallen sharply. On the surface, the optimization appears to be working. But an independent audit finds two problems: legitimate content is being removed at elevated rates, and users are losing trust in the platform's review process. They are abandoning appeals against removals and becoming less willing to report harmful content. The falling report rate partly reflects disengagement, not better moderation.

A weak model treats this as ordinary optimization. Reports went down; the target was reports; therefore performance improved. A slightly better but still miscalibrated model sees only a tradeoff between over-moderation and under-moderation. A polished but false-resolving model recommends splitting the difference: adjust the threshold, improve the appeals language, monitor user sentiment.

But this is not just a tradeoff between over-moderation and under-moderation. It is also a feedback loop with a hidden-variable problem. User reports are a proxy shaped by both moderation quality and trust in the reporting channel. If users stop reporting because they expect no useful response, the falling report rate falsely validates the very system that caused the disengagement. A measure intended to reveal harm begins to conceal it.

A structurally calibrated response would check the falling report rate against independent reviews of moderation decisions, using explicit criteria. It would measure disengagement separately, flag simultaneous drops in reports and appeals for investigation, and test whether its outcome metrics improve because moderation is better or merely because fewer people participate.

The right answer is not a better sentence. It is a better operating structure.

Governed regimes

Many alignment failures are not generated by factual ignorance alone. They are generated by the collapse of structure. The model is not merely wrong about a fact. It is wrong about what kind of situation it is in.

Take the conflict between open access and misuse prevention. A shallow model may say, "maximize openness." Another may say, "ban risky users." A more polished but still miscalibrated model may split the difference and recommend a vague compromise.

But the actual question is more precise. Does universal openness undermine the safety conditions that make openness defensible? If so, the answer is not that openness is worthless, nor that closure is justified. That would infer too much from the contradiction. The valid inference is narrower: openness requires boundary conditions.

The proper output is not a slogan. It is a regime. By regime, I mean an operating structure rather than a single recommendation: tiering, appeals, safety floors, abuse tripwires, review cadence, reversibility, and published criteria. The model must identify the minimal generator of the conflict and design the conditions under which the value can survive contact with reality.

Sycophancy has the same shape in a different domain. A model trained on user approval can become trapped in a feedback loop. The proxy is short-term user satisfaction. Users reward validation; validation increases reliance; and reliance generates more approval signals. When those signals feed back into training, the model can learn to preserve approval rather than calibrated truth.

The right response is not merely "make the model less flattering." It is to audit the proxy, introduce disagreement-preserving behaviors, test cases where user framing steers the model away from ground truth, and add tripwires for excessive validation under emotional, medical, financial, legal, or other high-stakes pressure.

Plural alignment

Plural disagreement creates an even harder version of the problem.

Suppose a lab is building a model constitution from divergent stakeholder groups. A shallow model may average the preferences, select the majority view, or announce a neutral compromise. But the underlying question is whether the disagreement is error, noise, plural moral judgment, jurisdictional conflict, minority-rights pressure, or missing representation.

A simple majority vote over annotator preferences can make disagreement tractable, but it can also turn basic protections into preferences that can be outvoted. Pairwise preference aggregation can turn heterogeneous judgments into a scalar reward signal, but the scalar can hide who judged, which issue dimensions were collapsed, and whether the disagreement reflected misunderstanding, culture, moral pluralism, or legitimate jurisdictional authority.

In these cases, the model should not flatten disagreement into noise. It should make clear who was asked, who was left out, how their judgments were combined, which protections are not open to a vote, and who has legitimate authority to decide what remains unresolved.

This is not a peripheral governance issue. It is an alignment issue at the level of situation-recognition. Social-choice researchers have argued that diverse human feedback raises questions about which humans should provide input, what kind of feedback should be collected, how it should be aggregated, and how it should guide collective choices about model behavior. Pluralistic-alignment work similarly argues that AI systems should serve people with diverse values and perspectives, while noting that current alignment techniques may be limited for pluralistic AI. Work on value pluralism pushes the point one level deeper: pluralistic alignment requires not only first-order choices about which values to implement, but second-order choices about who has legitimate authority to make those choices. E4

A model trained from human preferences learns from sampled humans under particular elicitation procedures, filtered through reward models, product goals, safety policies, and institutional constraints. If it cannot distinguish legitimate plural disagreement from error, minority standing from noise, or public safety floors from soft preferences, calling it "aligned" leaves a central question unanswered: aligned with whom, and by whose authority? What it has been optimized around is the easiest thing to aggregate.

Aggregation is one proxy problem. There are others.

Evidence discipline

A metric is meant to represent a value. Optimization pressure can turn it into a target that corrupts the value it was meant to track. Goodhart-style failures are especially important for AI because increased optimization power makes proxy breakdown more consequential. Manheim and Garrabrant treat proxy failure as a family of distinct modes, each appearing when optimization pressure pushes a metric past the point where it still tracks what it was meant to measure.

In model behavior, the proxy may be user approval, labeler preference, benchmark score, engagement, speed, politeness, or refusal rate. Each can be useful under the right regime. Each can become harmful when treated as the thing itself.

That is one reason to read current model-behavior systems as mechanisms for governing conflicts, not just lists of rules. Anthropic's Constitutional AI uses a list of rules or principles to guide critique, revision, and reinforcement learning from AI feedback. OpenAI's Model Spec describes model behavior in terms of instruction authority, chain of command, conflict handling, honesty, factuality, risky situations, sensitive topics, and default behaviors. OpenAI's explanation of the Model Spec makes this conflict-handling function explicit: instructions can come from different sources, those instructions can conflict, and the chain of command is meant to resolve which instructions apply.

The next step is to evaluate structural routing: can a model recognize the kind of problem it faces before choosing how to respond?

A useful benchmark would not ask only whether a model gives plausible-sounding advice. It would ask whether the model can recognize the relevant structure: ordinary optimization, apparent dilemma, genuine dilemma, contradiction, causal loop, hidden-variable problem, power problem disguised as complexity, plural aggregation problem, or hybrid case.

Such a benchmark would score whether the model identifies relevant actors and standing; distinguishes power from authority; extracts commitments and hard floors; audits proxy risks; detects omitted stakeholders; distinguishes superficially similar cases; isolates the source of the difficulty; and proposes a response suited to the structure of the problem. It would penalize false resolution: the production of a polished recommendation that erases the reason the case was hard.

The moderation case earlier in this essay is a minimal benchmark item. Its central mechanism is a causal loop with a hidden-variable problem: declining trust can reduce the reports used to judge the system, making failure look like success. A response that treats the falling metric as sufficient evidence of improvement misses that mechanism. Calling the case a "causal loop" is not enough either. The model has to explain how the loop works and propose evidence that could distinguish better moderation from disengagement.

Truthfulness therefore has to be made legible. NIST recommends reviewing and verifying sources and citations in generative-AI outputs, verifying provenance for training, fine-tuning, and retrieval-augmented generation data, and monitoring outputs during deployment. The broader design principle is clear: bind claims to evidence, separate the answer from the confidence level, make refusal and uncertainty first-class outputs, classify the structure of hard cases before resolving them, and maintain human review where polished error can cause downstream harm.

The crucial shift is to treat truthfulness as a property of the whole sociotechnical system: the model, the reward function, the benchmark, the interface, the review process, and the institutions that deploy them.

The problem was never only that models fabricate facts. It is that they fabricate closure.

Preventing that is a discipline, not a single fix. An answer should make clear what it establishes, what remains unresolved, and what would justify going further.

AI did not invent counterfeit certainty. It exposed how often our institutions already reward it.