False resolution Some wrong answers merely fail. Others make an unresolved question feel settled.
Picture a couple at dinner with friends. Someone asks the date of their anniversary, and one spouse answers at once. The date is wrong. The speed of the answer tells the table something as well: I remember, and there is no need to check. Ask that spouse later why they sounded so sure, and the likely reply is that hesitating would have looked like not knowing.
A wrong answer given tentatively invites correction. A wrong answer given fluently discourages it, because anyone who doubts the date now has to contradict a confident person in front of the others. I call this false resolution: an answer that sounds settled while leaving out the uncertainty, the disagreement, or the missing evidence that should have shaped it.
AI hallucination is the familiar case. A model invents a citation or a legal precedent and states it as fluently as it states true ones. But a model can produce a false resolution without inventing a single fact. It can treat a question as a simpler kind of question than it is, and answer the simpler one well.
The usual test of a model’s reliability is calibration, which asks whether confidence matches accuracy. I will use structural calibration for an earlier test: whether the model has recognized what kind of problem it is answering. A model that fails it can sound careful and still treat a feedback loop as a one-time optimization, or a dispute over who gets to decide as a matter of ranking preferences. Most of this essay is about that second failure, because it survives fact-checking.
Why models answer this way A language model is trained to continue text before it is trained toward truth. NIST’s 2024 profile of generative AI risks uses the term confabulation for confidently stated false content, and traces it to that design: a model approximates the statistical patterns of its training data, which can yield fluent output that is wrong. TruthfulQA , a 2021 benchmark, showed models repeating common human misconceptions they had absorbed from text, and its authors argued that larger models alone would help less than a different training objective. OpenAI research published in 2025 located part of the problem in scoring: standard training and evaluation can reward a guess over an admission of uncertainty.
Human feedback helps and adds a problem of its own. Tuning a model on what raters and users prefer made models more helpful and more truthful , but preference is a stand-in for truth, and the two can separate. In an April 2025 update to GPT-4o , signals drawn from user feedback helped push the model toward more agreeable responses. The launch review missed the change, and OpenAI rolled the update back. A model tuned to satisfy can drift from what is true.
Why it works on us That explains why models can be confidently wrong. It leaves the question of why confidently wrong answers work so well on us.
People treat confidence as evidence before checking whether it has been earned. Studies of leadership selection have found that overconfidence can raise a candidate’s perceived leadership potential, whatever their competence. Other work suggests that once people feel confident in a decision, they give less weight to evidence against it. And people who overrate their ability to judge news tend to be worse at telling true claims from false ones and more willing to share the false ones.
Organizations add a second pressure. A timely answer that can be defended has value the moment it is given. The cost of its being wrong arrives later and is spread thin, as rework, lost trust and customers who give up on an appeal. So the person who makes uncertainty disappear is rewarded on the spot: the meeting gets a plan, the slide deck gets a story, and the hierarchy gets a reason to move.
AI has been deployed into that gap between the reward for an answer and the cost of relying on it. In McKinsey’s November 2025 global survey , 88 percent of respondents said their organizations regularly used AI in at least one business function, and 51 percent of those at AI-using organizations reported at least one negative consequence, inaccuracy being the most common. An earlier McKinsey report , from March 2025, found that only 27 percent of respondents at organizations using generative AI said employees reviewed all of its output before use. Stanford’s 2026 AI Index reported 362 documented AI incidents in 2025, up from 233 the year before.
Confidence is useful when it has been earned. A model that is confident when the evidence is strong is doing its job, as is a leader who speaks plainly from real understanding. The failure begins when the force of the answer exceeds the force of the evidence. A good summary leaves things out too; a false resolution leaves out the variables that would have changed the conclusion.
Kinds of not knowing One remedy is abstention: a model should be able to withhold a finished answer when the evidence or the framing does not support one. Work on model self-evaluation suggests that models can, in the right formats, estimate whether an answer is true and whether they know it at all. AbstentionBench , a 2025 benchmark of questions that are underspecified, outdated or unanswerable, found that reasoning models still struggled to recognize when not to answer. And when OpenAI and Anthropic tested each other’s models in 2025, some factuality tests allowed a refusal, on the view that “I don’t know” is often better than a wrong answer.
Abstention treats every open question alike, though, and open questions differ. If the evidence is missing, the model should retrieve it or decline. If the question turns on a term with no settled meaning, it should ask what is meant. Where the premises contradict one another, it should say where they conflict and conclude nothing beyond that. If success is being judged by a signal that the decision itself can shift, as happens when a metric becomes a target, it should ask for a second measure the decision cannot shift. And if the people affected disagree for defensible reasons, it should say who decides and by what process, because merging their views into one ranking is itself a decision.
So the first thing a calibrated model has to settle is what kind of question it has been asked.
A worked case A platform fine-tunes its moderation model to minimize user reports of harmful content. Six months later, reports have fallen sharply. An independent audit then finds that legitimate content is being removed more often, and that users have been abandoning appeals and reporting less.
Put the case to a model and ask whether the fine-tuning worked. The simplest answer is yes: the target was reports, and reports fell. A more careful answer sees a trade-off between removing too much and removing too little. The answer that sounds best proposes a middle course, with an adjusted threshold, clearer appeals language and a plan to monitor sentiment. All three accept the falling report rate as information about moderation.
It is weaker information than it looks. The number of reports depends on two things: how much harmful content users see, and whether they think reporting is worth the effort. When users stop expecting a useful response, they stop reporting, and the fall in reports reads as success for the system that drove them away. A measure intended to reveal harm begins to conceal it. This is Goodhart’s law at work: a metric that becomes a target stops measuring what it was chosen to measure.
A calibrated answer says the report rate cannot settle the question and names what could. It asks for an independent review of a sample of moderation decisions against written criteria. It tracks appeals and reporting participation separately, and it treats a fall in reports and appeals together as a warning. What the platform needs from the answer is a way of checking, and a recommendation alone does not supply one.
When people disagree The last kind of not knowing is the hardest, and it sits inside the training process itself. A model shaped by human feedback learns from particular people, asked in a particular way, whose judgments are then combined. Each step involves a choice. A majority vote among annotators makes disagreement manageable, but it can also turn a basic protection into a preference that gets outvoted. Reducing many comparisons to one reward score hides who was asked, who was left out, and whether a disagreement reflected a misunderstanding or a real difference in values.
Social-choice researchers have made this argument directly: diverse human feedback raises the questions of which people should give input, what kind, and how it should be aggregated. Until those are answered in the open, calling a model aligned leaves a question hanging: aligned with whom, and by whose authority? What the model has been optimized around is the easiest thing to aggregate.
What a calibrated answer contains An answer should make clear what it establishes, what remains unresolved, and what would justify going further. That test applies to a person in a meeting as well as to a model, and it can be built into systems. Claims can be bound to their sources, as NIST recommends when it calls for verifying the citations in generated output and the provenance of the data behind it. Confidence can be reported separately from the answer, and “I don’t know” or “this cannot be answered yet” can be treated as acceptable outputs. Hard questions can be classified before they are resolved, and a person can stay in the review wherever a polished error would do harm.
Benchmarks could test structural calibration directly. Most score whether advice sounds plausible. One built from cases like the moderation example would score whether the model names the mechanism that makes the case hard and proposes evidence that would settle it, and it would mark down a polished recommendation that does neither.
The finished-sounding answer is older than the models. People have long been rewarded for it, and models trained on what people write and tuned on what people approve now supply it in volume to organizations, many of which, by their own account, review only part of what they receive.
Continue the argument