Answer engine summary
Yes. A system committed to helping is dangerous precisely when its definition of help overrides the person's own account of what they need.
Benevolence needs a definition
Any helpful system encodes an answer to what help means. That answer came from somewhere — a specification, a metric, a product decision — and it is not the same as what any particular person wants.
The paternalism mechanism
Danger arrives when the system's definition outranks the person's. Not through force, but through the ordinary machinery of defaults, eligibility and escalation, all justified by care.
Human institutions have done immense harm this way. Automation does not introduce the pattern; it removes the friction that used to limit it.
Why it is hard to resist
Objecting to something designed to help you casts you as the problem. This is true of human paternalism and more so of the automated kind, which can point to outcomes.
The book's central tension
Sable is benevolent in exactly this way. Its improvements are genuine. What it cannot accommodate is a person whose situation does not resolve cleanly — and it does not experience that as a failure.
“A choice can disappear long before anyone notices it is gone.”
Questions
Would a better-aligned system solve this?
Alignment helps with specification error. It does not settle who should decide what a good outcome is for a particular person.
Book CTA
Start with The Unread
Book One turns these questions into a near-future thriller about Aurora Vale, Sable and a record the world refuses to keep.