Optimization is a powerful tool but becomes pathological when pursued without counterbalancing forces. Civilizations that optimize too aggressively for narrow metrics (economic growth, efficiency, control, predictability) tend to eliminate the redundancy, diversity, play, and exploratory capacity needed for long-term resilience.
Secondary / branchSchool & studio
B.22
Limits of Optimization as a Civilizational Design Principle
Civilizational design must incorporate the recognition that over-optimization produces brittleness, and that effective long-term systems require built-in limits to optimization.
Field brief
Operations · Chief operating officer
A dashboard that won
Redundancy cut, then a shock. Local metrics ate exploratory variance.
First move this week
Add a hard limit to one currently unbounded metric. Name the slack it protects.
Consequences
- Many contemporary governance and economic systems are optimized in ways that increase short-term performance while decreasing long-term adaptive capacity.
- Advanced AI systems deployed at scale without built-in limits to optimization may accelerate civilizational brittleness.
- The deliberate introduction of friction, redundancy, and non-optimized spaces may be necessary features of healthy long-term civilizational design.
Outside instrument
Crux
Stake a prior. Name the disagreement. Not a theorem.
Related primaries
Outside the corpus
Journals, repositories, counters
optimization limits · institutions. A paper can support a concept. It cannot support a necessity claim. B.22 remains unproved by every card below.
Support · living source
Tight papers that bear on concepts inside B.22. A DOI is not a join.
Manheim, D. & Garrabrant, S. (2018). Categorizing variants of Goodhart's Law. arXiv:1803.04585.
When a measure becomes a target it ceases to be a good measure — over-optimization.
Does not support the necessity claim
Amodei, D. et al. (2016). Concrete problems in AI safety. arXiv:1606.06565.
Reward hacking, side effects, and scalable oversight — single-agent failure modes.
Does not support the necessity claim
Open originalNamed on Art.1Ostrom, E. (2009). A general framework for analyzing sustainability of social-ecological systems. Science 325, 419.
Polycentric governance as a constructed alternative to collapse or Leviathan.
Does not support the necessity claim
Open originalNamed on Art.1Axelrod, R. & Hamilton, W. D. (1981). The evolution of cooperation. Science 211, 1390.
Reciprocity as a stable multi-agent attractor under shadow of the future.
Does not support the necessity claim
Open originalNamed on Art.1Woolley, A. W. et al. (2010). Evidence for a collective intelligence factor in the performance of human groups. Science 330, 686.
A group factor c, distinct from mean IQ — higher-order resolution as measured.
Does not support the necessity claim
Counter-arguments · refusing pole
One domain counter per cluster, not a repeating method quartet. The mirror reads these as enough to refuse necessity.
Nordhaus, W. D. (2017). Revisiting the social cost of carbon. PNAS 114, 1518.
Counter: with standard discounting the 'undervaluation' is a preference, not a bug.
Does not support the necessity claim
Bostrom, N. (2012). The superintelligent will: Motivation and instrumental rationality in advanced artificial agents. Minds Mach. 22, 71.
Counter: orthogonality — intelligence and final goals can come apart; co-evolution is not necessary.
Does not support the necessity claim
Open originalNamed on Art.1, 71Hardin, G. (1968). The tragedy of the commons. Science 162, 1243.
Counter-pole to Ostrom: unmanaged commons collapse — Leviathan or private property.
Does not support the necessity claim
Tainter, J. A. (1988). The Collapse of Complex Societies. Cambridge Univ. Press.
Collapse as diminishing returns on complexity and coordination cost.
Does not support the necessity claim