Fragility of Value under Imperfect Alignment
Winter Cross
- Digest date
- 2026-08-03
- Submitted
- 2026-07-30
- arXiv ID
- 2607.28881
Summary
The paper models the alignment problem where an agent's value function is guaranteed only to satisfy some proxy condition before deployment, and derives conditions under which an η-catastrophic value function (one that provably drives expected human value arbitrarily low under enough optimization power) would still pass pre-deployment checks and be deployed.
Abstract
As more responsibility is placed upon AI systems, it becomes increasingly important to guarantee that these systems are aligned with humanity. A common fear in AI safety is that human value is fragile -- that is, optimizing too heavily for an imperfect proxy to human values will lead to a catastrophic outcome. In this paper, we present a model of the alignment problem where an agent undergoes idealized alignment training that guarantees its value function satisfies a proxy condition before optimizing the world. Our primary results identify conditions on the human value function and the accuracy of several proxy conditions under which an agent with an $\eta$-catastrophic value function, one that is guaranteed to take the expectation of human value below $\eta$ in the limit of optimizing power, would be deployed. Our results highlight the danger of overoptimization and motivate AI designs that limit optimization pressure, such as quantilizers, rather than relying solely on pre-deployment training.