In a \u00a0mansion\u00a0 overlooking the Golden Gate, the elite of \u00a0artificial intelligence\u00a0 gathered last Sunday to discuss a troubling question: the end of humanity and what might come next. With non-alcoholic drinks in hand, around 100 attendees\u2014including \u00a0philosophers\u00a0, entrepreneurs, and researchers\u2014imagined a future where humans no longer exist, but a \u00a0consciousness\u00a0 created by us does. What should this successor be like?<\/p>\n
<\/p>\n
The event, known as \u2018\u00a0Worthy Successor\u00a0,\u2019 was aimed at precisely discussing how to define a \u201cworthy successor\u201d for when humanity no longer exists. This notion connects closely with the goal of creating a \u00a0general artificial intelligence (AGI)\u00a0, a hypothetical superintelligence that would surpass humans in all aspects of knowledge. As the host, \u00a0Daniel Faggella\u00a0, stated: \u201cYou would much rather have it (not humanity) determine the future course of life itself.\u201d<\/p>\n
<\/p>\n
First, who is Faggella and why should we listen to him? He is the founder of \u00a0Emerj Artificial Intelligence Research\u00a0, a consulting firm specializing in AI analysis. In 2016, he wrote for \u00a0TechCrunch\u00a0 about the risks posed by AI and is now focusing on \u00a0moral and philosophical perspectives\u00a0 surrounding the creation of this ‘worthy successor’, an idea that has been under discussion for some time. Faggella has been reaching out to various prominent personalities in the industry, culminating in this gathering two years later. <\/p>\n
<\/p>\n
The party featured three presentations from \u00a0Ginevra Davis\u00a0, a New York writer, philosopher \u00a0Michael Edward Johnson\u00a0, and Faggella himself. The complete guest list has not been disclosed, but Faggella boasts that attendees included founders of \u00a0AI companies valued at over $5 billion\u00a0 and key figures from laboratories working on AGI research.<\/p>\n
<\/p>\n
Faggella explained to \u00a0Wired\u00a0: \u201cThe major labs know that AGI will likely end humanity, but they don\u2019t talk about it because \u00a0incentives\u00a0 don\u2019t allow for such discussions.\u201d While this sounds like a \u00a0conspiracy theory\u00a0, he is not the first to warn of such outcomes. A decade ago, \u00a0Bill Gates\u00a0 asserted that we should fear AI, shortly followed by \u00a0Elon Musk\u00a0, who called for regulations to mitigate impending dangers. Recently, various AI experts signed a statement warning of \u201cthe extinction risk from AI.\u201d OpenAI was also considering the \u00a0dangers\u00a0 of AGI, suggesting that discussions around these risks could have contributed to the \u00a0controversial dismissal of Sam Altman\u00a0 months later.<\/p>\n
<\/p>\n
The truth of these claims remains elusive, but we know that most arguments suggesting the \u00a0imminence of AGI\u00a0 and its associated risks are based on opinions and conjecture rather than empirical evidence or concrete advancements. For instance, recent studies have shown that current systems still struggle with basic reasoning tasks, contradicting the notion of \u00a0superintelligence\u00a0 emerging in the near future. There are also indications that generative AI may be nearing its \u00a0ceiling\u00a0. Experts are divided, with some dismissing these claims as \u00a0ridiculous\u00a0; however, such sensational narratives tend to be more “viral” than grounded discussions.<\/p>\n
<\/p>\n
Moreover, we must remember that these statements often originate from business figures like Altman, and the industry is an expensive one that requires substantial funding. Creating buzz around the \u00a0imminent arrival of AGI\u00a0 might serve as a strategy to secure more investment for their companies.<\/p>\n
<\/p>\n
The focal point of the gathering was not how humanity would meet its demise (this seemed assumed), but what type of \u00a0intelligence\u00a0 we should create to be our successor. Presentations revolved around the values and capabilities this new form of superior intelligence should embody. For Faggella, humanity bears the responsibility of designing a successor that is both \u00a0aware\u00a0 and capable of \u00a0evolving\u00a0.<\/p>\n
<\/p>\n
Philosopher \u00a0Michael Edward Johnson\u00a0 highlighted the risks associated with creating a \u00a0conscious AI\u00a0 beyond just the commonly discussed potential extinction: \u201cWe risk enslaving something that can suffer or trusting something that cannot be trusted,\u201d he stated during his talk. Rather than forcing AI into compliance, he proposed a joint \u00a0education\u00a0 of humans and AIs to \u201cpursue the good,\u201d whatever that may be. Ultimately, this raises an intriguing \u00a0ethical\u00a0 and philosophical debate, albeit one with little basis in current reality, at least for now.<\/p>\n
<\/p>\n
As we contemplate an uncertain future, the dialogue around AGI and its implications will likely continue to spark intellectual curiosity and caution.<\/p>\n