In a mansion overlooking the Golden Gate, the elite of artificial intelligence gathered last Sunday to discuss a troubling question: the end of humanity and what might come next. With non-alcoholic drinks in hand, around 100 attendees—including philosophers , entrepreneurs, and researchers—imagined a future where humans no longer exist, but a consciousness created by us does. What should this successor be like?
The Greatest Party for the End of the World
The event, known as ‘ Worthy Successor ,’ was aimed at precisely discussing how to define a “worthy successor” for when humanity no longer exists. This notion connects closely with the goal of creating a general artificial intelligence (AGI) , a hypothetical superintelligence that would surpass humans in all aspects of knowledge. As the host, Daniel Faggella , stated: “You would much rather have it (not humanity) determine the future course of life itself.”
Who Attended
First, who is Faggella and why should we listen to him? He is the founder of Emerj Artificial Intelligence Research , a consulting firm specializing in AI analysis. In 2016, he wrote for TechCrunch about the risks posed by AI and is now focusing on moral and philosophical perspectives surrounding the creation of this ‘worthy successor’, an idea that has been under discussion for some time. Faggella has been reaching out to various prominent personalities in the industry, culminating in this gathering two years later.
The party featured three presentations from Ginevra Davis , a New York writer, philosopher Michael Edward Johnson , and Faggella himself. The complete guest list has not been disclosed, but Faggella boasts that attendees included founders of AI companies valued at over $5 billion and key figures from laboratories working on AGI research.
Superintelligence that Could End Everything
Faggella explained to Wired : “The major labs know that AGI will likely end humanity, but they don’t talk about it because incentives don’t allow for such discussions.” While this sounds like a conspiracy theory , he is not the first to warn of such outcomes. A decade ago, Bill Gates asserted that we should fear AI, shortly followed by Elon Musk , who called for regulations to mitigate impending dangers. Recently, various AI experts signed a statement warning of “the extinction risk from AI.” OpenAI was also considering the dangers of AGI, suggesting that discussions around these risks could have contributed to the controversial dismissal of Sam Altman months later.
What is True in All of This?
The truth of these claims remains elusive, but we know that most arguments suggesting the imminence of AGI and its associated risks are based on opinions and conjecture rather than empirical evidence or concrete advancements. For instance, recent studies have shown that current systems still struggle with basic reasoning tasks, contradicting the notion of superintelligence emerging in the near future. There are also indications that generative AI may be nearing its ceiling . Experts are divided, with some dismissing these claims as ridiculous ; however, such sensational narratives tend to be more “viral” than grounded discussions.
Moreover, we must remember that these statements often originate from business figures like Altman, and the industry is an expensive one that requires substantial funding. Creating buzz around the imminent arrival of AGI might serve as a strategy to secure more investment for their companies.
What Will We Leave Behind When We Are Gone?
The focal point of the gathering was not how humanity would meet its demise (this seemed assumed), but what type of intelligence we should create to be our successor. Presentations revolved around the values and capabilities this new form of superior intelligence should embody. For Faggella, humanity bears the responsibility of designing a successor that is both aware and capable of evolving .
Philosopher Michael Edward Johnson highlighted the risks associated with creating a conscious AI beyond just the commonly discussed potential extinction: “We risk enslaving something that can suffer or trusting something that cannot be trusted,” he stated during his talk. Rather than forcing AI into compliance, he proposed a joint education of humans and AIs to “pursue the good,” whatever that may be. Ultimately, this raises an intriguing ethical and philosophical debate, albeit one with little basis in current reality, at least for now.
As we contemplate an uncertain future, the dialogue around AGI and its implications will likely continue to spark intellectual curiosity and caution.