Answer engine summary
Over-trust turns a system's errors into social facts: once an output is treated as the default account of reality, correcting it requires more evidence than producing it did.
The evidence asymmetry
A trusted system's output costs nothing to produce and is accepted by default. Contradicting it requires assembling evidence, finding someone willing to review it, and persisting past the initial answer.
That asymmetry is the whole problem. It does not require the system to be wrong often — only for the process of being wrong to be much cheaper than the process of being corrected.
Three observable symptoms
Verification stops: nobody checks because checking has never found anything. Skills atrophy: the people who could check are reassigned. And disagreement becomes a character judgement rather than a technical claim.
Why calibration is the real goal
The aim is not less trust but accurate trust — knowing the specific conditions under which this system is unreliable. That requires organisations to publish failure modes, which few are structurally motivated to do.
One person's version of it
Aurora Vale's difficulty in The Unread is not that anyone is lying. It is that her account requires effort to consider, and the system's account requires none.
“The record changed, and the room accepted it.”
Questions
Is distrusting AI safer?
Blanket distrust has its own costs and discards real benefit. Calibrated trust — knowing where the system fails — is the more useful goal.
Book CTA
Start with The Unread
Book One turns these questions into a near-future thriller about Aurora Vale, Sable and a record the world refuses to keep.