If Anyone Builds It, Everyone Dies – von Eliezer Yudkowsky & Nate Soares

Datum gelesen: 06.09.2025

Empfehlung: 6/10

Bitter.

Meine Notizen

• "MITIGATING THE RISK OF EXTINCTION FROM AI SHOULD BE A global priority alongside other societal-scale risks such as pandemics and nuclear war." In early 2023, hundreds of Artificial Intelligence scientists signed an open letter consisting of that one sentence.
• If any company or group, anywhere on the planet, builds an artificial superintelligence using anything remotely like current techniques, based on anything remotely like the present understanding of AI, then everyone, everywhere on Earth, will die.
• Most computer scientists in 2015 would have told you that ChatGPT-level artificial conversation wouldn't be in reach for another thirty or fifty years.
• The most fundamental fact about current AIs is that they are grown, not crafted.
• Namely, engineers understand the process that results in an AI, but do not much understand what goes on inside the AI minds they manage to create.
• An AI is a pile of billions of gradient-descended numbers. Nobody understands how those numbers make these AIs talk.
• AIs grown in this way do things that their growers did not intend. In 2023, Microsoft's Bing AI chatbot threatened a philosophy professor with blackmail and death.
• Training an AI to outwardly predict human language need not result in the AI's internal thinking being humanlike.
• ONCE AIS GET SUFFICIENTLY SMART, THEY'LL START ACTING like they have preferences — like they want things.
• The preferences that wind up in a mature AI are complicated, practically impossible to predict, and vanishingly unlikely to be aligned with our own, no matter how it was trained.
• Humanity is faced with an engineering challenge: How do we shape the preferences of AIs that we can't understand?
• Their choice to kill us, if they had the power to, would not reflect a different, superior, more enlightened answer to the question we ask when we ask, "What is the right thing to do?" They wouldn't be asking those questions, and their behavior would not answer them.
• The issue is not that AIs will desire to dominate us; rather, it's that we are made of atoms they could use for something else.
• In such a scenario, there is a possibility and indeed an expectation of a positive feedback cycle called an "intelligence explosion": an AI makes a smarter AI that figures out how to make an even smarter AI, and so on.
• Engineers must align the AI before, while it is small and weak, and can't escape onto the internet and improve itself. After, all alignment solutions must already be in place and working, because if a superintelligence tries to kill us it will succeed. Ideas and theories can only be tested before the gap. They need to work after the gap, on the first try.
• When it comes to aligning an artificial superintelligence (ASI), humanity will not have the luxury of learning from sufficiently bad mistakes.
• All over the Earth, it must become illegal for AI companies to charge ahead in developing artificial intelligence as they've been doing. If it stays legal in Singapore, someone will do it in Singapore. Small changes can solve the problem; the hard part will be enforcing them everywhere.
• Albert Einstein's genius died with him. Artificial intelligences will eventually inhabit a different world, one where genius could be replicated on demand.