Source Alignment Forum · Published · Open the original ↗
Lab notes
Some “coherence arguments” are two different things in a trenchcoat
The “two kids in a trenchcoat” trope , in case you haven’t heard of it. Image sources: 1 , 2 tl;dr I go through three “coherence arguments”, related to (1) market competition, (2) AI self-reflection and self-modification, and (3) AI training algorithms. I argue that each of these implies a pressure for (something like) “being self-consistent”, and also a pressure for (something like) “only caring about long-term eventual outcomes, with ruthless indifference to anything else”. But the latter pressure tends to be weaker and/or more contingent than the former. So…
Alignment Forum publishes this post on its own site. Supence shows only what the publisher allows to be quoted.