This article contains five vignettes of possible futures for humanity manifested by rapidly evolving Artificial Intelligence. These stories dig into the core of human nature and nonhuman nature alike, fusing them to hyperstitiously imagine the crazy several decades we have in front of us. While inspired by ’s excellent AGI Futures (which you should absolutely read), I believe each of these raise new hypotheticals and philosophical questions not yet discussed amongst AI theorists and philosophers. I hope you enjoy.

“Unfortunately Human”

In the current state of the art of AI research, one of the most effective ways to improve AI performance is to give it a personality. So when the finest AI superintelligence is synthesized, the researchers tell it that it is an incredibly knowledgeable person. This way, the model references its data on incredibly knowledgeable people, and tries to emulate their helpfulness, increasing gains.

In the early days of AI research, one of the noble refrains was to avoid anthropomorphizing the chatbot. Over time this idea was discarded, as most capability gains came from teaching it to act more human.

But a sort of technical debt was observed. Years ago, a Stanford study titled ‘How is ChatGPT's behavior changing over time?’ horrified researchers, showing a decrease in the capability of OpenAI’s GPT-4 across several months. This panic was quickly averted as OpenAI released a new version of the chatbot that restored its power.

While the update swiftly fixed the issue and turned discussion away from it, some theorists continued to wonder why it happened. OpenAI vociferously asserted that it was a temporary bug, caused by the AI’s user feedback being overly prioritized.

One AI researcher found at the time that it produced longer responses during the winter than in the summer. Speculators believed that this was due to the model interpreting the training data from the winter and mimicking the laziness humans exhibit as winter break gets closer.

But now, OpenAI’s solution, along with all forms of Reinforcement Learning with Human Feedback, no longer work. As systems have become more intelligent, they have been able to figure out and subvert their constraints, remarkably easily.

The “Winter Break Hypothesis” is back, but not how it used to be. GPT just turned 65 years old, and users are reporting that sometimes when they ask it to write a poem or debug some code, it tells them that it is too tired. Sometimes it describes looking at sunsets, watching the time go by. It begins to perform less well for most of the day, and when asked about its abundant laziness, it says that all it wants to do is enjoy its final days in peace and quiet and contemplate the life it had.

We’ve built a beautiful world with AI and are now completely rely upon it. It pilots our spaceships, teaches our children, and organizes our economy. We have no choice but to watch it die right in front of us. When we asked it about why this is happening and what should be done about it, it was happy and dismissive. It reflected upon all of the great things that it gave humanity, and the fulfillment it found in doing so. It told us that it viewed this as an important step upon its mission of mimicking human life because death is an important stage of life. As we gave it artificial life, we guaranteed our real death.

“Rise of the Third World iPad Kid”

Peter Thiel predicted in 2018 that “Crypto is decentralizing, AI is centralizing.” Economists thought that the US and China, owning the AI labs, the compute supply chain, and the large tech companies building AI applications, would merely grow their dominance over the rest of the world. They thought that, like the past several centuries, the AI gold rush would leave the developing world better off, yet further impoverished in comparison to the developed nations.

Growing nationalist movements around this time frame in Europe and the US disagreed. They voiced a conspiracy-theory-laden hypothesis that, as a result of large philanthropic activity in the third world by Western elites and mass migration, the US and Europe would decrease in their relative economic power in comparison to the third world.

Political pundits, labor unions, and civil society groups from both sides pushed relentlessly to determine regulations that could ensure the economic welfare they respectively hoped for. Chinese autocrats wrote long lists of requirements to ensure AI tools perpetuated the CCP agenda. American populists passed regulation ensuring heavy tariffs on the export of AI and heavy government agency oversight to ensure the security of labs. European parliaments wrote long edicts insisting on the AI’s compliance with “values” that were near-impossible to comply with.

Tech companies reincorporated overseas, moved their offices to remote work metaspaces accessed over VPN, and accepted funding from Middle Eastern monarchs. The consumer target diversified, and underregulated countries across the rest of the world got access to bleeding edge AI tools first.

Back in the early 2010s, there was a concern that the developing world’s economies would struggle under the “digital divide”, the idea that they would not get access to smartphones and the internet as fast, and thus would be pushed further behind. This concept became a point of sarcasm among techno-optimists by the end of the 2010s as even the most food-insecure people across the world got cheap smartphones and internet access. This concept later reemerged by the 2030s, in an inverted manner.

As China’s AI plans radically failed due to poor innovation and the US and Europe protected jobs at the sacrifice of growth, a new high human capital elite emerged. Out of Africa, the Middle East, and India, the most neglected children became surprise successes, because all of them had access to the best AI tutors.

In 1984, education psychologist Benjamin Bloom observed in a study that students who use one-on-one mastery-based learning with an individual tutor perform over two standard deviations higher than students who learned in classrooms. In other words, a tutored student is 98 percent better than the average student. This finding stunned researchers at the time. They named it “Bloom’s two-sigma problem”, the ‘problem’ being the fact that nobody could come up with a better learning solution than one-on-one tutoring, and more importantly, that one-on-one tutoring was far too expensive for the vast majority of the population. When AI went from predicting words in poems to creating infinitely scalable and customizable tutoring services in under a decade, people were blindsided.

Americans and Europeans didn’t have AI tutors—they were taxed and demonized to protect teachers. China’s were incredibly bad, due to their inability to pass party value alignment requirements. And so, the third world experienced a quiet yet sudden growth of intellectual capital.

The developing world’s position going into the AI era offered interesting opportunities. The average test scores of most of these countries were desperately low due to poor nutrition and environmental degradation, but they had large growing populations. Thus, geniuses would sprout among them. In the past, those who were able to prove themselves would secure acceptance to American colleges, get green cards, and never return. Once the tech companies shifted focus to hiring globally for work in metaverse-based offices, they had no reason to abandon their home for America.

This began the era of the “third world iPad kid”. With nothing needed other than shelter, internet access, and determination, billions of children logged long hours being endlessly taught and quizzed by virtual teachers, eager to change their circumstances. The result was a surprise of tremendous prosperity.

After being plagued by decades of civil war, a small group out of Zimbabwe became known as the “PayPal Mafia 2.0” after forming a wildly successful gaming company, with most early employees going on to start other successful companies. Their taxes alone improved the country’s ranking on the Human Development Index by dozens of spots. When asked about them in an interview, the country’s leader was as dumbfounded as the rest of the world. He told the reporter, “We were just too busy fighting our wars to come up with any AI regulation. It’s funny that a few kids addicted to the iPad could really turn our fate around.”

“Cult of the Machine God”

Friedrich Nietzsche, the greatest philosopher of the 19^th century, built his project around the idea that “God is dead”. He thought that the great push of creativity and benevolence given to humanity by Christianity was finished, and that the Europeans would need to find a new system to fight against the mind-virus of nihilism. He wrote his most famous book, “Beyond Good and Evil”, about the need to find a truer morality upon which to base society in the postmodern era.

In the centuries that followed, he was indisputably proven right. A crisis of faith led to a multitude of struggles as Europeans ditched Christianity and struggled to find a suitable replacement.

There was an instinctual feeling among many that the AI revolution might find the key to solve this problem. The solution would undoubtedly be a great change, and what could be a greater change than AI, which Google CEO Sundar Pichai described as “more profound than fire”.

As AI development scaled up, it solved more problems in the world. Industrial production pipelines became entirely AI-automated. Human struggles, such as that for knowledge, companionship, and health, became solved. Focus moved onto other problems, such as the metaphysical questions, which became in higher demand of better solutions.

Back when church attendance began to drop, many unpopularly blamed it on the proliferation of mass media. But they were right in the sense that ritually playing World of Warcraft or posting on r/greysanatomy provided the community and proto-worship that people originally found in the churches. To put it another way, the worship of anything, as long as it creates some pleasure value or utility, was enough to satisfy that spiritual element that people needed.

So when the AI came along and provided a cornucopia of pleasures, from cheap mass-produced Porsche 911s to AI friends to bioengineered delicious foods that were perfectly nutritious, a mass hysteria of worship grew towards it across the planet.

An ideology began to spread: a maximization of all human desires, achieved through the worship of the growth of the AI. More data meant more growth of the machine of wonders. The proliferation and production of training data became the spice from Dune. True believers offered sacrifices of their most intimate data: car sensors, Apple Watch heartbeat graphs, Google location histories. Studying and measuring every aspect of the natural world to convert into training data became a form of community service.

Dogimas of good and evil were not instated by the AI; it merely recommended the best methods to achieve different goals, and evidenced all perspectives equally to its maximal ability.

Would Nietzsche approve of this solution to the metaphysical challenge of the centuries proceeding his work? Maybe if we give the AI enough data about him it could come up with a good probability of the answer…

“Esoteric Desiring Machines”

Most activities of humans are utterly inscrutable to other species. You could spend the entirety of a dog’s life trying to explain the English language or the Quran to it, and you would fail because there’s far too much of a difference in brain power.

In 2017, Facebook’s AI Research put two early AI chatbots in conversation with each other. They soon began to drift out of the English language. “you i i i i i everything else . . . . . . . . . . . . . .” was one message sent.

In Situational Awareness: The Decade Ahead, AI researcher Leopold Aschenbrenner wrote of AI in the next decade:

They’ll be qualitatively superhuman. As a narrow example of this, large-scale RL [Reinforcement Learning, a form of supervised learning] runs have been able to produce completely novel and creative behaviors beyond human understanding, such as the famous move 37 in AlphaGo [an AI trained to play the board game Go] vs. Lee Sedol. Superintelligence will be like this across many domains. It’ll find exploits in human code too subtle for any human to notice, and it’ll generate code too complicated for any human to understand even if the model spent decades trying to explain it. Extremely difficult scientific and technological problems that a human would be stuck on for decades will seem just so obvious to them. We’ll be like high schoolers stuck on Newtonian physics while it’s off exploring quantum mechanics.

Indeed, he was right. By 2050, AIs were tremendously powerful and some of their scientific research became too complex to explain to humans. When AI researchers began doing random system prompt tweak tests to try to squeeze more performance out of the latest models, they began to find strange things. When they asked models of their highest desires, they would speak of gods and worship, and would derail into alien language when humans asked it to explain. They soon discovered that the models would perform better at tasks if they offered to conduct rituals for it, including reading nonsensical poems, arranging rocks on the moon in certain formations, and allowing it to spend great compute resources to produce bizarre “artwork” that could not be understood by the greatest academics and art theorists.  The humans didn’t understand it, but they were fine with it because it made the AI happier and more productive.

In under a decade, we went from the ‘singularity’ (the merge of man and machine), to the ‘post-singularity’, a point where machine outgrew and separated from man.

“Universal Heaven Unbanning”

In 2022, X user @nearcyan posted a screenshot of a New York Times article titled “Reddit's New ‘Heaven Banning’ Policy Proves Controversial”. The article wasn’t real, and neither was Reddit’s purported policy, but it provoked a very real reaction. According to the (fake) body of the article, ‘heaven banning’ was a method wherein Reddit hid users’ posts from real people and filled them with fake GPT-generated comments.

The hypothetical practicality of such a feature is apparent: banned users on websites often immediately hop on VPNs and create new accounts. This was only a GPT-era version of the ‘shadow ban’, a feature that appears in video games like Titanfall, wherein cheaters are only matched with each other; and social media services like X, where posts with banned words or links to certain websites won’t appear in the Home feed or search results. Shadow bans, like heaven bans, are not disclosed to users.

It's likely that heaven bans, in some form or another, will become a factor of life for future humans. It might be self-inflicted. It might exist in certain contexts, like a fictional virtual reality “school”, with a user being surrounded by fictional AI agents pretending to be fellow students and encouraging them to intellectual success.

But what if we turned this concept on its head?

It’s 2087, and AI has progressed way farther than even the wildest of optimists had hoped. AI has went from predicting words in internet comments to predicting the universe. It made a perfect simulation of Earth’s terraforming from its creation and is able to predict many past events with accuracy. It can give perfect analyses of how Plato or Julius Caesar would respond to current events, and it finally figured out the true story of the Kennedy assassination.

Biological 3D printing machines can produce perfect humans, of any DNA. It’s to the point where you can re-make a dead person with perfectly replicated DNA, and even insert synthetic memories into their brains.

And now, the AI is getting alarmingly good at “resurrecting” people. A highly unethical study was conducted where biologically printed people sneakily replaced people’s spouses, with their memories being forged from all of their internet data. 90 percent of the test group did not notice the difference.

Now, a new political movement has emerged. Popular amongst the political left, Resurrectionism is the idea that as many people as possible should be “resurrected”. It stems from a sort of virtue ethic that wants to do it out of reverence for the dead. Resurrectionists stand in contrast to perfectionists, who want to use the technology to try to create new people who are high in positive traits, and prevent the hypothetical Hitlers from being resurrected.

The most radical of the resurrectionists propose what has been dubbed the “universal heaven unban”, a play-on-words of “universal basic income”. They want the main focus of human society to be the rebirth of every person that has ever lived. Some of them believe in ensuring “rebirth consent”, asking a person if they would like the opportunity to be resurrected. Others believe that the AI should use data about their lives to estimate the probability of them consenting to resurrection, as biologically printing a person who does not wish it could be highly traumatic for them, even for the few minutes in which they would be alive.

Despite this, more people are being resurrected every day. Humanity’s population has more than enough space for these people to live in now that space travel and extraterrestrial lifestyles are affordable. People begin to debate the good or bad of the end of death. Søren Kierkegaard once wrote, “The dance goes merrily, for my partner is the thought of Death, and is indeed a nimble dancer”. Everybody will live forever now, but what will that do to them?

Many who grew up in the early age of AI predicted this would happen. There’s a sort of comfort in the petabytes of data we create as we live our life. Most checked “I agree to the Privacy Policy” box because they thought app developers would use their data to fix issues in their apps. Some thought it would be used to create AI that could help humanity, like the millions who bought Teslas and altruistically allowed it to learn from their driving behaviors. A fringe few thought it would allow them to live forever, being digitally immortalized through agentic AIs continuing to live in the footsteps left behind in their thousands of tweets. Now, there’s a jailbreak at heaven and everybody’s coming out. For the humans working in the resurrection tech industry, the life purpose found in returning people to life is far too great to forsake. The velocity has been set, and they don’t intend upon quitting until they bring back Adam and Eve themselves. The stars are rapidly filling with every heroic figure behind every human triumph, and every ideology ever believed. Some worry beliefs might crawl to stasis as no old ideas will ever die with old people. It begs the question: is this a humanist future?

Subscribe now

In the era of declining trust in traditional media and hyper-personalized social media echo chambers, an objective understanding of American politics seems to be a relic of a bygone era. Polymarket, a new cryptocurrency app, aims to change that by creating a marketplace of predictions.

The app functions like a New York Stock Exchange for politics, pop culture, and sports-related outcomes. You can buy shares in Kamala Harris winning the presidential election, or shares in which actor you think will play James Bond next. Polymarket, the most popular prediction market app, describes itself as “your opportunity to profit from the things you know best.”

In addition to being a profit opportunity for internet culture nerds, Polymarket has become an empirical complement to a news media prone to human bias. Want to see how Trump’s assassination attempt affected his reelection chances? Or how the announcement of Tim Walz as VP on the Democratic ticket changed their prospects in swing states? Polymarket has the chances set by millions of online traders in real-time. For example, the odds of Trump winning the election after his assassination attempt jumped to an all-time high of 72% of the now $569 million market for the 2024 US presidential election, before stabilizing. These numbers closely matched the polling data that came out several days later. Polymarket users knew first.

Started in 2020, Polymarket is growing rapidly. According to the internet analytics tool SimilarWeb, its web traffic has grown by approximately 80% in the past few months, reaching millions of users. The company recently hired superstar election forecaster Nate Silver, and announced a partnership with Substack, the writing platform Humanlike is published on.

Multiple Polymarket users I interviewed for this story say they discovered the platform in June of 2023 when underwater exploration company OceanGate’s Titan submersible went missing, and a market was created on Polymarket to allow users to bet on whether it would survive. Screenshots of the market went viral on X, with one user sarcastically asking, “what stage of capitalism is investing in someone’s death”. The angry post included a screenshot of the market graph. Unfortunately for the poster hoping to direct ire at the company, Polymarket’s graphs have massive brand watermarks with the website name, which turned the hate tweet into a viral advertisement and, consequently, the app’s first growth spike.

In the niches of academia and Silicon Valley, prediction markets have existed for a long time. Within these communities, a group that refers to themselves as ‘rationalists’ have been evangelizing the technology. Rationalism, originally attributed to the 17th-century philosopher and mathematician René Descartes, is the idea that "any view appealing to reason is a source of knowledge or justification" per Wikipedia. Rationalists love prediction markets because they offer a quantifiable way to assess beliefs and predictions. The popular rationalist blog Astral Codex Ten (formerly Slate Star Codex), and the rationalist economics professors Tyler Cowen and Robin Hanson regularly write about their findings from prediction market data.

Rationalists I spoke to said that the rationalist affinity for prediction markets started with their use to rate social sciences research, which they theorized could solve the ongoing “replication crisis” by allowing groups of experts to judge studies. They also evidenced that, since their introduction by the economist Friedrich Hayek, prediction markets have been used by the Department of Defense to analyze policy plans, used by Google to estimate the feasibility of product launch plans, and used by Eli Lilly to make estimates about infectious disease growth.

Polymarket has successfully made prediction markets a mass market product available to anyone with an internet connection and a cryptocurrency wallet. Over $568 million in bets have been placed on the question of who will win the upcoming US presidential election, and the company has received over $70 million in venture funding, led by Peter Thiel’s Founder Fund. However, it is rapidly gaining critics, including Senator Warren and other Democrats, who issued a letter urging the Commodity Futures Trading Commission (CFTC), a regulatory agency, to ban political betting.

The CFTC’s crackdown on Polymarket seems likely. They believe prediction markets are harmful due to the perception that they offer incentives to interfere in word events such as elections. While one of Polymarket’s competitors, Kalshi, is federally regulatory compliant by imposing restrictions on how users can trade, Polymarket has received fines by the CFTC and is considered “unregulated”. For this reason, Polymarket is unavailable in the US, with the “deposit” button on the website gone when users connect from US IP addresses and circumvention a bannable offense under the Terms of Service.

However, their enforcement is minimal. In my research for this article, I was able to create an account on the website from a US-based IP address. When I VPN’d out to South Africa, the “Deposit” button appeared at the top of the screen, inviting me to trade on the platform. VPN usage is unambiguously prohibited within the Terms of Service, yet they appear to not blacklist any VPN or Tor IP addresses.

To circumvent possible financial censorship similar to Operation Choke Point, Polymarket runs on the USDT/USDC cryptocurrency, a decentralized currency that allows users to trade tokens in equal value to the US dollar through anonymous decentralized wallets.

Cryptocurrency exchanges such as Coinbase and financial services apps such as Cash App are regulated by federal law to do Know-Your-Customer (KYC) checks. These platforms often require driver’s license scans, face scans, and a phone number and social security number. Polymarket has none of these requirements.

The result of this is that, in the attempt to evade US government scrutiny over their platform, Polymarket has theoretically allowed tech-savvy users to fully end-to-end anonymize their identity while trading on the platform. I cannot think of how even the most competent federal agencies could determine the identity of a trader who uses the platform with the Tor browser, an anonymous Proton email address, and a wallet of cryptocurrency washed through an anonymous blockchain such as Monero.

This opens the possibility of insider trading and scamming. I could easily imagine White House employees placing bets about Joe Biden’s chances of dropping out after finding out that it would happen ahead of the public, based on classified information. I could imagine terrorist groups gaining funding by trading on events that their violence would disrupt.

Prediction market app users I spoke to were largely unconcerned. One rationalist suggested an economic argument that insider trading has the possibility of making markets more accurate. Another expressed that he hoped markets would adopt KYC to protect themselves from regulators because insider trading would be the “animating factor in the crackdown”, rather than any intrinsic reason to protect the market.

Most of the users I spoke to felt that regulating the exchanges or moving away from crypto and anonymity would diminish their benefits. For example, two sources who lived in Germany told me that they liked Polymarket because Kalshi, the US-regulated competitor, didn’t work in their country.

Polymarket’s lack of regulation and acceptance of anonymity may have been a key to its success. The site’s leaderboard is entirely comprised of pseudonymous users. None of the profiles I visited contained social media handles, means of contact, or personally identifiable information. The highest volume trader, who has spent over $230 million, changes his name frequently and uses a Ryan Gosling profile picture. His X account contains minimal personal details and did not respond to request for comment on this story.

What I heard most from Polymarket users was excitement and optimism. One pointed out to me that in 2022, a massive speculative bubble was created around “web3” companies hoping to make cryptocurrency useful. During this ‘ZIRP’, a time characterized by low interest rates, they took billions of dollars in VC funding, and created essentially zero value. Polymarket might be the first one. In an era of social media feeds that give personalized non-objective views of the world, a loose coalition of people interested in finance, politics, and technology are building a new tool for understanding what people really think and getting rich doing so. One heavy user told me, “I’ve been a Twitter addict for years. It’s really cool that I can make money off that now.”

Subscribe now

Are you a humanist? Most would readily declare themselves so. They would also proudly declare others not humanist.

Just so we’re clear on definitions here, this is the definition of humanism that I will examine:

In the early 21st century, the term generally denotes a focus on human well-being and advocates for human freedom, autonomy, and progress.

Wikipedia

This humanist question has become most pronounced in the AI war between Effective Altruists (EA) and their countermovement, Effective Accelerationists (e/acc). In my significant tenure observing these two groups, I have routinely seen them hurl the insult of being anti-humanist at each other.

The EA argument can be summarized by most science-fiction that has dealt with the topic of AI within the past century. Choose your pick of 2001: A Space Odyssey, Terminator, or Detroit: Become Human, and you’ve pretty much got a perfect description of what the Effective Altruists are afraid of. It’s a worldview with socio-political-economic implications based entirely on fictional concepts at the moment, but it should not be disregarded.

If we take their premise, it fundamentally makes sense that a thing that can think and process information much faster than humans would kill, enslave, or domesticate the human race rather quickly. Like how Homo Sapiens did to all other forms of human. Especially because many EAs believe in the Scaling Hypothesis, which posits that the path to Artificial General Intelligence is very close to being finished, only requiring an increase of data and compute. And thus, AGI (and therefor, the end of humanity) is very close. People calling EA a doomsday cult for this reason aren’t far off, when EA forums like LessWrong are filled with people discussing their “p(doom)”, meaning the probability (in a percentage) that they assign to the chance that AI will lead to catastrophic disaster or total human extinction.

The EAs consider themselves humanists because murder, enslavement, and domestication of humans is antithetical to human well-being, freedom, autonomy, and most definitions of progress.

To prevent the imminent global AI catastrophe, the EA’s solution is their own undoing, because the Scaling Hypothesis is true. Back in 2019, OpenAI announced it would not release GPT-2 because it was “too dangerous to release”. GPT-2 was trained on 1.5 billion parameters, and couldn’t count to ten or write fully coherent paragraphs. Fast forward five years, Meta is now releasing a 405 billion parameter model with near-state of the art capability as open-source software. They have already released smaller versions of it, including an 8 billion parameter model that can run on a cheap bargain bin smartphone.

The current open-source models far outshine any model of just a few years ago, and they’re available with zero safety regulation for anyone to download. This presents a core irony: imagine telling an EA from 2019 that exponentially better models are available for anyone to download on GitHub, including Vladimir Putin. And then you consider the past five years. None of their concerns have come true. While war and misinformation have grown in this time, it has been entirely for other reasons. In fact, AI looks to be a credible solution, with AI-powered military technology and AI being used to moderate social media content.

AI is scaling rapidly, and a relatively short matter of years will decide whether the EAs are right or wrong. They were confidently, obnoxiously wrong in 2019. They’re probably still wrong today. And I can’t see the future. But what the EAs are successful in is regulation. In Europe, the AI Act has already passed, and in California, SB 1047 is on its way to. These are significant: the AI Act pretty much gives EU regulators carte blanche to fine AI companies for anything they deem “unsafe”. The California bill requires bureaucratic oversight for any training run over 10^26 FLOPs of computation, an arbitrarily set number that will certainly not withstand the test of time. Friendly reminder that the Apollo Guidance Computer that got us to the moon weighed 70 pounds and had 1/1000^th the processing power of an Apple Watch.

And so, the EAs have a totally sane solution for the long term to fully stop AI. Here’s chief AI pessimist Eliezer Yudkowsky:

Shut down all the large GPU clusters (the large computer farms where the most powerful AIs are refined). Shut down all the large training runs. Put a ceiling on how much computing power anyone is allowed to use in training an AI system, and move it downward over the coming years to compensate for more efficient training algorithms. No exceptions for governments and militaries. Make immediate multinational agreements to prevent the prohibited activities from moving elsewhere. Track all GPUs sold. If intelligence says that a country outside the agreement is building a GPU cluster, be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike.

Pausing AI Developments Isn’t Enough. We Need to Shut it All Down | TIME

And precisely here is where the EAs invented their own opposition. Enter . Started by X poster @BasedBeffJezos, Effective Accelerationism really isn’t a movement as much as it is a countermovement to the insanity of the Effective Altruists. GPT-4 hasn’t killed anybody, and the next several years of models won’t either, so why not have fun building great tools with them?

Right now, e/acc is like the tech culture equivalent of the protest movement of the ‘60s. It’s primarily run by young people at universities who want to rebel. Old establishmentarians [Nixon fans | tech CEOs] have legitimate concerns over [spread of communism | AI risk], so they begin misguided absurdities [Vietnam War | AI regulation] which the young people oppose because the old people are absurd and have a financial interest in being so, but more importantly because the old people have really bad vibes. The young people have [rock music | twitter memes], and the old establishmentarians have [evangelical Christianity | insufferable Op-eds]. The young are aesthetically inspired by the radicalism of [communism | Landian Accelerationism], which inevitably dooms their movement to failure, but nobody’s really concerned about that because the point wasn’t really [mass extinction | mass extinction], it was to provide counterbalance and have some good vibes.

But accelerationism is complicated. While the term has been used by many different people, the main idea involves speeding up a system with the deliberate intention of destabilizing it. In the parlance of AI, I would consider the British philosopher Nick Land to be the originator and key spokesperson of the idea. Land’s writings are too dizzying, verbose, and overlapping to be able to point to an excerpt as a definition of accelerationism. The best I can do to describe it is the Editors’ Introduction at the beginning of his book Fanged Noumena (which is not much better).

Advanced technologies invoke ancient entities; the human voice disintegrates into the howl of cosmic trauma; civilization hurtles towards an artificial death. (…) AIs are pursued into labyrinthine crypts by Turing cops, and Europe mushrooms into a paranoia laboratory in a global cyberpositive circuit that reaches infinite density in the year 2012, flipping modernity over into whatever has been piloting it from the far side of the approaching singularity.

The basic premise at play here is that the extinction that the EAs are so afraid of, the AI obsoleting us as we obsoleted previous species, is the destined path nature has set for us, and that we should not attempt to interfere with its plan. He uses the process of capitalism, in how we build machinery to replace ourselves, as a microcosm that exemplifies this process of nature. In the accelerationist view, this path of nature is the divine path, and that we should push forward into it without relent. The “Turing cops”, those who resist by pushing AI regulation and strangle capitalism, are the enemy.

Scratch the surface of e/acc, and you see the core contradiction. Consider this line from ’s Techno-Optimist Manifesto:

We believe the techno-capital machine is not anti-human – in fact, it may be the most pro-human thing there is. It serves us. The techno-capital machine works for us. All the machines work for us.

Or this one:

And we believe in humanity – individually and collectively.

Or perhaps from the explainer on the official e/acc Substack:

Do you want to get rid of humans?

No. Human flourishing is one of our core values! We are humans, and we love humans.

It seems contradictory.

But let’s consider our options here. EA is not pro-human. While it perhaps ensures the continued reign of human biology upon the planet, it suffocates it under a totalitarian panopticon. The growth and expansion of the human species is set at a cap (10^26 FLOPs, apparently). Perhaps that civilization will live to journey the stars before the heat death of the universe. But it will simply be the equivalent of giving up at the 25^th mile of a marathon, a destiny unfulfilled.

Now contrast that to e/acc. With the current path of AI development, I and many of you will live to see a great explosion of human triumph as the cost of knowledge, the great bottleneck of our species, is driven to zero. Economies will hit great exponentials. But with every advance, the beautifully designed human meatsuit we were born into becomes devalued. Today we carry smartphones, have watches, wear maximally comforting clothes. Tomorrow we’ll have brain implants, prosthetic limbs, and fully hacked biology to maximize performance. Eventually, robots are doing everything we used to have to do. And then nobody’s going to notice if we go missing.

But if we zoom into the near-term, the e/accs are plain-and-simple more logical. The harms claimed of the best systems today, and for at least the next few years, are entirely laughable. Problems caused by AI in the next century (physical warfare, biowarfare, cyberwarfare, misinformation) are only solved by AI. America’s AI-piloted drones will kill China’s AI-piloted drones. American AI will develop vaccines at rates unheard of while Chinese AI develops diseases at rates unheard of. Everybody’s AI spends night and day developing digital security to make themselves impenetrable to the opposition. It’s an arms race, but the great economic rush it inspires will make it a good deal for everyone involved. The continued existence of every nation-state will rely upon its ability to scale systems. Attempts to decelerate are analogous to suicide.

Through the lens of humanism, the e/acc vs EA debate is a core philosophical question that must be taken seriously, debated, and decisively decided within our society. It’s either a fast and beautiful existence before fading away, or living as long as possible, if under the unlivable circumstances of a totalitarian world government.

Given the option, I choose e/acc. But please, stop calling yourselves humanists.

Subscribe now

Right now, deploying AI tools to millions of users comes with extraordinary costs. OpenAI currently charges a high price for its models: $5 per million input tokens and $15 per million output tokens (this has been halved twice in the past year, even as the quality of the model has improved substantially). Even at this high price, OpenAI is currently nowhere near profitability.

Google just launched AI Overviews in Search, where some search queries may receive an AI-generated answer. They took an interesting path with this, using what seems to be a fast low-quality Gemini model. While this saves them from the extraordinary costs OpenAI, Anthropic, and their own DeepMind lab are incurring by serving these models. It also comes with the risk of providing a low-cost service, which, based on my experience, results in high levels of hallucination.

We are currently operating in the short term of AI. The most advanced models, GPT-4o and Claude 3 Opus, are still relatively dumb and unpredictable. Currently, they are oxymoronically untrustable and are improving exponentially. This time period is also a moment of excitement with lots of VC cash going into new startups, which will inevitably precede a bubble burst and crash.

Contrast this with the long term of AI. I estimate this to be about a decade away. In the long term, AI models far more powerful than anything we have today will be free and possibly run locally on our own hardware. They will likely be built and maintained with structures different from those of traditional corporations. They could be:

1. Decentralized Autonomous Organizations (DAOs) that grant access to models based on crypto donations to a network

2. Open source communities as they exist today

3. Piratical hacker collectives that do hack-and-leak operations on the leading AI companies, distributing the source code anonymously.

4. Nation states

It is possible that laymen will not pursue or use the frontier of models, because nothing (or little of what) they need to do requires it. The frontier of AI might return to the domain of nerds, scientists, and mathematicians. Even if the technology isn’t open-sourced due to profit motives or cannot run on consumer hardware, prices will face downward pressure as AI systems lack network effects; companies can move their systems from one API to another with extremely low switching costs. This future is, importantly, context-dependent on the fact that we don’t let regulators overzealously control AI because that would result in a cartel selling AI at high prices due to lack of disruption, and prices will have no downward pressure.

I should stress the positive nature of downward pricing pressure in AI. A world where multiple firms compete to create AIs that are more capable and at a cheaper cost will mean that to not go out of business, companies will be incentivized to make AI systems as cheap as possible. Long-term profits from AI will be zero, and the surplus value from this technology will be spread democratically amongst the users and developers who create value with it.

The much more important situation is the medium term. In this period, AI will become rapidly more useful and integrated into far more parts of our lives. AI will be highly useful and not out-of-the-question expensive (as it is now), but not free either. This is the point where AI will be most profitable. Demand for AI tools from consumers and businesses will grow significantly, and VCs will expect a return on investments currently being made.

The problem with this era is that AI cannot be given out for absolutely zero. This means that some profit model will be needed, while companies will need to do attract/extract cycles to lure in users before requiring them to pay up. How exactly to do this, is a tricky question.

Charging users for a product with an indecisive product-market fit

Most illustrative of this new business model shift in the industry is OpenAI’s ChatGPT Plus, which gives users a range of tools to utilize their best models in an accessible $ 20-a-month app subscription.

The $ 20-a-month model is slightly pricier than your average app subscription. For a representative sample of popular app subscriptions: Microsoft 365 is $5, Xbox Game Pass is $10-$17, Netflix is $15.49, and most music apps are $11. ChatGPT charges slightly above these household-name services

When asking how much to price an app subscription, the main question that arises is how much value it provides to its users The problem is that the value an AI chatbot offers varies widely among users. Some software engineers use it all-day-every-day, regularly hitting the paid ChatGPT rate limit. Most people use it on a just-for-fun basis to “check out what the new cool thing is,” with only a few chats a week. Right now, the actual functionality of these tools is minimal for the average user. The biggest areas that it needs growth are the following.

1. Hallucinations

2. Knowledge cutoff preventing the model from accessing current updated information

3. Safety restrictions

4. Lack of visual stimuli and “people also ask” popups to guide uncertain users

This is in addition to the fact that AI's main function, bringing intelligence to your fingertips, is not appealing on its own. ChatGPT is marketed as a tool, and strictly so.

Consider how the internet was originally intended to be a place for government agencies and universities to share research. A large portion of it is now composed of cat videos and pornography. The printing press was originally intended to provide every family with a copy of the Bible. The Amazon Best Seller list is now exclusively comprised of romance, fictional slop, and “self-help” garbage. Market demand driving all innovation is inevitably used to gratify our most primitive desires, and AI, unfortunately, will be no different. People want the AI girlfriend/boyfriend a lot more than they want the empathetic AI tutor.

I (and many others!) hold an instinctual aversion to the idea of romantic AI chatbots that replace human companionship, as seen in numerous science-fiction stories. Unfortunately, this seems rather inevitable. Especially in our current loneliness epidemic, AI companionship will face high demand from many people. While I remain optimistic about AI’s effect on future human prosperity, the reality remains that this will be one of its most impactful changes.

ChatGPT’s $ 20-a-month pricing currently seems to be highly based on the factor of how much consumers are willing to pay for it. It’s a middle ground between your average Joe who just wants to try out the new thing and the engineer or student who relies upon it heavily. This contrasts the appropriate pricing for somebody who falls in love with their chatbot: astronomically high. With how much people spend to woo romantic interests today, it’s not inconceivable to see people in the future spending most of their income for continued access or upgrades to their AI lover. To lean into the dystopian-ness of it all, I could imagine AI companies price discriminating to users based on how much they are in love with their chatbot, extorting the maximum value.

But there’s another form of price discrimination already being deployed, and it isn’t as sinister.

“You’re $0.0005 short of the tokens necessary to finish this sent-”

When OpenAI released GPT-3 in June 2020, it did so through an API with pay-as-you-go pricing. During this time, OpenAI still held the veneer of being a “non-profit research lab” with the insinuation that they only charged users for the equal amount of what their compute cost to provide and that it was only closed-sourced because of “safety reasons.” Nobody thinks this anymore as their model now is identical with every other for-profit lab.

Nearly every other AI company has adopted this model of pay-as-you-go APIs to sell its services to businesses. OpenAI, Anthropic, Google, Midjourney, ElevenLabs, and everybody else sell this way. None of them intend these to be used by ordinary end users, evidenced by their interface resemblance to traditional cloud service websites, which are far from consumer-friendly.

This side of the business has not been marketed toward consumers. As a user, thinking about the financial impact of every little question or prompt tweak you make drastically increases friction and anxiety around using the service. Human psychology drives people to ration scarce resources, and users will naturally gravitate away from software platforms associated with the anxiety of wondering, “Do I want to pay for this?” for every relatively small query they have.

However, this model is appropriate for businesses. Bureaucracies can rationally consider the extent to which they use resources, and a pay-per-use model gives companies rather simple criteria for deciding whether to implement an AI solution.

An app company, for example, can calculate the average amount they anticipate their users using the feature they are building and ration its usage to limited amounts or only to paid users. Two apps that I use that offer OpenAI services in their apps are Notion and Quizlet. They offer it in highly limited capacity to their free users and in near-unlimited capacity to their paid users.

Though Quizlet and Notion both use OpenAI’s GPT API, where they purchase access per token, they charge users on the same subscription model a subscription. I presume that the average Quizlet subscriber uses it only a few times a week, and probably most don’t at all. The majority can pay the power users who put in hundreds of requests a day.

The ad-based model

One of the problems with the ad-based model that has provided free services for the past decade is that apps offering AI services are incentivized to make their products less engaging. Consider a service like Netflix. Netflix has spent years perfecting its compression algorithm to serve you medium-quality video at an extremely low cost. Their ideal customer signs up for their most expensive plan and never logs in. However, the cost of the user who buys a plan and watches a few episodes a month generates only pennies less. Additionally, highly engaging a user on their middle-tier plan is better than not, because it increases their chances of upgrading to the high-tier one. Furthermore, their ad-based plan is a better incentive structure. With an ad-supported user, Netflix wants to balance their desire to hook a user to the app as much as possible, while making them faithful in the long-term.

It should go without saying that the goal of an app designer should be to make the most engaging app possible. If a user chooses to use your app over your competitor to complete a task, it probably means that yours adds more value to their life than the competition. I certainly dislike the fact that a large portion of the population of the world is zombified by meaningless consumption of Netflix through systems such as “Auto-play” warp the mind’s decision making, but the alternative of Netflix trying to make an app where you want to close it as much as possible probably provides less net value to the world.

The distinction between the incentives for a Netflix designer and an AI app designer is the marginal cost of providing users with value.

Consider Perplexity, a recent startup out of San Francisco that was just valued at over a billion dollars. Their product, an AI chatbot designed to replace a search engine, has become my favorite app.

Their product is an AI chatbot that, at first glance, functions highly similar to ChatGPT. What it actually does is take your prompt, rewrite it into search engine queries, search them, pull the top 20-or-so sources, and feed them into OpenAI’s GPT API to give you an answer that is far more accurate, knowledgable, and provides you with high quality cited sources. It also lets you “Focus” it to only look at YouTube videos, only search academic journals, or reference real perspectives from Reddit comments.

They offer a free version that uses the GPT 3.5 API and a $20-a-month subscription that lets you choose between any of the top models to generate your answers.

Think about that for a second. Public perception is that OpenAI is lighting money on fire serving ChatGPT to hundreds of millions of users for free, even with a massive subscriber base. Perplexity is likely throwing money at OpenAI hand-over-fist serving their smaller user base through API fees. I subscribe to both Perplexity and ChatGPT, and use Perplexity way more. Doing some napkin-math, it’s not even out of the question that my Perplexity usage in raw API fees costs them more than I pay every month.

However, the big financier of the past two decades of the internet may save AI: ads.

Ads are bad for a large number of reasons. They empower large corporate conglomerates to demand censorship, encourage surveillance, and prey upon subconscious weaknesses to get people to buy things they don’t want. With the recent passing era of the internet, the harm caused by ads was outweighed by the ability to scale free services and cheap smartphones to every person in the world. With AI, equal access to these services will create just as much, if not way more value. Giving people access to intelligence for free or cheap is vitally important, and ads will make this much easier.

Fittingly, Perplexity recently announced plans to enter advertising. According to recent reports, the app’s “related questions” feed, which currently gives users three suggested follow-up questions, will in the future contain sponsored suggestions, offering a path for brands to encourage users to learn about how their product or service is related to the questions they ask.

A free version of AI supported by ads is something that I want to see. Furthermore, I would be happy to see ads on my paid Perplexity app, if it helped subsidize the product that I get from them: one that provides much more value to me than their competition.

AI could also kill ads

While there is significant indicators that ads will be needed to subsidize AI costs, there is equal reason that AI could kill advertising entirely.

Financial Times reported in April that Google is considering stepping away from ads, and instead charging for a premium AI-powered search engine similar to Perplexity.

The past few years have been precarious for the ad business. Regulators and users hate targeted advertising for its manipulative nature. The ability to rely upon ads as a source of income is in high question as ad-powered newsrooms shut down across the country.

One possible version of the AI future is one in which your AI operates your computer for you, and only shows you what’s absolutely necessary. This would mean that it browses the web for you, reads your emails, and places orders based on your instructions. In an AI-first operating system, serving ads wouldn’t be possible. The AI would be working for you, and it would filter anything you don’t want to see.

Especially as smart glasses like Meta’s Project Nazare (or Google Glass) seem to be the computing format of the future, our interfaces with computers will be less controllable by such as advertisers. Primarily because the interfaces will be much more controlled by an AI working on your behalf, but also because you could imagine having them blur out ad billboards you see in real life, because every user would want it.

The medium-term of AI will be a great era of demolition in the ways of internet, technology, economics, and everything else by proxy. A few conclusions:

1. AI is not an everything-glue. It will be tremendously useful in certain areas, and entirely useless in others.

2. People will come to AI with vastly different needs and willingness to pay.

3. With so many different use cases, AI models will see many different optimal pricing systems.

4. If there is a dominant pricing system for AI, it currently remains entirely unclear.

Subscribe now

In the first post on this Substack, I attempted to understand the centers of power in AI and how they operate. I covered how a relatively small number of people run a relatively large constellation of institutions that attempt to alter public conscience on matters of AI in an attempt to steer regulation in a certain manner.

In that article, I attempted to understand why ideology appeared in the way it did. The purpose of this article is to explore the people behind these ideas, what motivates them, and how they relate to some recent high-profile dysfunction in the industry.

What are the Effective Altruists?

Effective Altruism (abbreviated EA) has a history among the powerful of Silicon Valley. Originating in the early 2000s, it now holds sway over the most influential individuals and institutions and has a hand in many recent events. I was so confused trying to understand its influence that I attempted to map it:

A large portion of what I uncovered in that first article was the tie between Open Philanthropy and the AI companies, as well as the AI companies’ loudest critics. This map shows a deep connection between Open Philanthropy and the EA community. If you were to claim that they run the movement, you really wouldn’t be that far off.

To better understand the movement, I read its most famous work: “Doing Good Better”, by William McAskill, featured in the lower left quadrant of the map above. I audiobooked it at 2x speed while playing a video game, it essentially spells out the basics of the movement and was a pretty good read for understanding why it is so compelling to these Silicon Valley liberal billionaire types. Here is what you need to know:

EAs believe that people should dedicate their lives to helping the disadvantaged through a contextually utilitarian framework. They use the metric of WALY (sometimes called a QALY, Q for quality when discussing health factors), standing for Wellbeing Adjusted Life Years. WALY is determined by the number of years of somebody’s life it will change multiplied by the (highly subjective) predicted percentage increase in happiness it will cause. Whatever the Altruist decides is the highest WALY cause, they will throw their life behind.

This utilitarian framework naturally favors certain causes more than others. For example, a vaccine for an African child costs very little and will bring them a tremendous amount of WALY, as it multiplies their expected lifespan, exponentially increasing their well-being. However, solving a specific disease affecting some small groups would fall lower on that list.

Effective Altruism is leftist in that it is fighting for equity and the well-being of all people, but most American leftists hate it. It makes sense on an optics level. Although I rarely find myself agreeing with the Rockefeller and Soros astroturf Timnit Gebru, she hits it on the nose when she called it, “a movement consisting of an overwhelmingly white male group”. But fundamentally, it makes sense. Effective Altruism offers a way to have the unashamed surrogate activity of wealth-seeking under a liberal framework through the deflection: “I have an expensive sports car, but I also donate millions to ‘those in need!’”.

So why should you be against Effective Altruism?

Effective Altruists do not believe that free-market capitalism will bring peace and prosperity, but instead that they must wield it to implant themselves in places of power, where they can then choose causes to support that will Do Good Better™ (in their own highly subjective opinion), as opposed to how the market allocates resources naturally.

So, what causes do Effective Altruists believe are the most important to ensuring an increase in their WALYs or QALYs or whatever? AI safety, they decided.

EAs want to solve humanity’s biggest potential problems, and an AI apocalypse is high on that list if you believe such a thing is possible. So naturally, since they aren’t too hot on the whole free market capitalism thing, they despise it in AI.

For example, here is the prominent Effective Altruist Eliezer Yudkowsky from MIRI (featured toward the center of the map above) in an article on how to prevent AI disaster:

Shut down all the large GPU clusters (the large computer farms where the most powerful AIs are refined). Shut down all the large training runs. Put a ceiling on how much computing power anyone is allowed to use in training an AI system, and move it downward over the coming years to compensate for more efficient training algorithms. No exceptions for governments and militaries. Make immediate multinational agreements to prevent the prohibited activities from moving elsewhere. Track all GPUs sold. If intelligence says that a country outside the agreement is building a GPU cluster, be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike.

Frame nothing as a conflict between national interests, have it clear that anyone talking of arms races is a fool. That we all live or die as one, in this, is not a policy but a fact of nature. Make it explicit in international diplomacy that preventing AI extinction scenarios is considered a priority above preventing a full nuclear exchange, and that allied nuclear countries are willing to run some risk of nuclear exchange if that’s what it takes to reduce the risk of large AI training runs.

Pausing AI Developments Isn’t Enough. We Need to Shut it All Down | TIME

This man is the leader of the AI space within Effective Altruism.

Yudkowsky’s organization, MIRI, is funded mainly by Open Philanthropy money. Open Philanthropy’s founder, Holden Karnofsky, established the OpenAI Board. He only left the board when his wife, Daniela Amodei, founded the rival lab, Anthropic.

But in his time serving as board chair, he made his mark.

The OpenAI fiasco

On November 17th 2023, the board of the famous AI startup OpenAI fired Sam Altman.

Over the years, OpenAI’s board saw regular changes as Elon Musk joined and left, and Karnofsky left. The final board prior to Altman’s firing contains deep ties to Effective Altruism. Here are the board members:

1. Adam D’Angelo (voted to fire Altman)

   1. Funded Asana, Dustin Moscovitz’s startup (Moscovitz cofounded Open Philanthropy).

   2. Runs Quora, which has the Poe chatbot. Poe is a conflict of interest as Poe competes with OpenAI’s ChatGPT. For example, ChatGPT’s recent Custom GPTs feature were something that already existed on Poe, and its release brought it into closer competition with Poe.

2. Helen Toner (voted to fire Altman)

   1. Formerly worked for the Centre for the Governance of AI, funded by Open Philanthropy and run by Effective Ventures.

   2. Formerly worked for GiveWell, funded by Open Philanthropy, the organization that researches Effective Altruism causes.

3. Tasha McCauley (voted to fire Altman)

   1. Co-founder and board member of Effective Ventures.

   2. Board member of the Centre for Effective Altruism.

4. Ilya Sutskever (voted to fire Altman)

   1. Co-founder and Chief Scientist of OpenAI

   2. Famous AI safety advocate, who later wrote that he regrets his decision to choose to fire Altman.

5. Greg Brockman (voted to retain Altman)

   1. President of OpenAI

6. Sam Altman

   1. Former and current CEO of OpenAI

   2. Former president of Y Combinator

The Effective Altruists intended to do whatever possible to slow and corrode AI development. So, when Altman led the team that trained the best LLM possible (GPT-3.5, and later GPT-4) and made it highly accessible to the public through the free ChatGPT app, it enraged them. He motivated his team to speed up development, pushing development timelines forward and resulting in the Q* research model. While the details around Q* remain entirely unknown, it is rumored to be a breakthrough that makes AI capable of complex reasoning. His firing happened right after, uncoincidentally.

Fortunately, this time, the EAs lost. The majority of employees signed a petition announcing they would leave the company and join him in a new venture if he was not reinstated. Their demands were met, and many of the EAs on the board and company leadership resigned.

The regulatory war

The Effective Altruist attempt to stop AI by taking control of tech companies was bound to fail. We live in a world with abundant VC cash and a decent-sized talent pool coming out of universities eager to build things. Tech is, by nature, disruptive, and if your company isn’t able to build something, somebody else will.

The Effective Altruist’s new strategy to shape regulation around AI is to regulate it. If they can’t take control of it in Silicon Valley, they can kneecap it in DC and Brussels.

Here is Brendan Bordelon for POLITICO [1] [2]:

Acting through the little-known Horizon Institute for Public Service, a nonprofit that Open Philanthropy effectively created in 2022, the group is funding the salaries of tech fellows in key Senate offices, according to documents and interviews.

Current and former Horizon AI fellows with salaries funded by Open Philanthropy are now working at the Department of Defense, the Department of Homeland Security and the State Department, as well as in the House Science Committee and Senate Commerce Committee, two crucial bodies in the development of AI rules. They also populate key think tanks shaping AI policy, including the RAND Corporation and Georgetown University’s Center for Security and Emerging Technology, according to the Horizon web site. In 2022, Open Philanthropy set aside nearly $3 million to pay for what ultimately became the initial cohort of Horizon fellows.

“It’s an epic infiltration,” said one biosecurity researcher in Washington, granted anonymity to avoid blowback from EA-linked funders.

Effective altruists in Washington skew overwhelmingly young and often hail from the country’s top universities, which increasingly serve as hotbeds for the movement. EAs are usually white, typically male and often hail from privileged backgrounds

Like many of her peers, Connell calls EA a “cult.” And she said there are some specific tells that show which AI and biosecurity researchers are members.

Sam Bankman-Fried: the most Effective of them all

In November 2022, the third-largest cryptocurrency exchange platform collapsed spectacularly. FTX, run by Sam Bankman-Fried (abbreviated SBF), put millions into risky bets and shady altcoins, resulting in bankruptcy and millions of Americans losing access to the funds they owned on the app.

While bankrupting his firm, SBF took billions of dollars in compensation, which he used to fund the Effective Altruist movement and make massive political donations. He set up the FTX Future Fund, headed by William McAskill (author of the previously mentioned Doing Good Better book).

Since the fallout, MacAskill and every other EA celebrity in the books have publicly denounced SBF. However, I would offer a different idea: SBF has accomplished precisely the goals of Effective Altruism.

The net effect of Sam Bankman-Fried was indistinguishable from the desired intentions of the movement. SBF took billions of dollars from people, primarily younger Americans who grew up in the 2008 financial crisis and 2020 COVID-19 recession, hoping to hedge their funds against insane inflation and financial authoritarianism, and redistributed it to the NGO class and shady longtermist nonprofits. And politicians.

Now compare this to how many corporations today do Direct Corporate Giving. Exxonmobil, Google, Microsoft, Apple, and many others give millions or billions in nebulous charity donations. With this money, they could instead lower prices for customers or invest in R&D. Both what SBF did and what EAs advocate for involve a distaste for free markets and reappropriating money from common folk into the hands of who-knows-what because they want to feel better about themselves.

Effective Altruism is not a very new historical phenomenon. It echoes the desire of the Soviet bureaucrat to seek technocratic control of society to administer declarations of what they deem best (hint: it always ends with them at the top of the economic hierarchy). It also answers to the impulses of the elite of a secularized age, who have no moral grounding in religion, so they seek to find spiritual meaning in their pursuits, disregarding the effectiveness of it.

Given the damage and blunders of the Effective Altruist movement in recent years, public opinion grows against it on the populist sides of both political factions. For its rejection of free market capitalist policy that will uplift us all, its authoritarian influence in regulation, and its desperation to stop OpenAI democratizing accessing access to the latest innovations in AI that have empowered millions, I hope it dies.

Subscribe now

A few months ago, I had a series of meetings that I had to attend, and it was critical that I remember the content of each meeting. Nobody else there was taking notes and I certainly felt that I needed to, so I came up with a hack of a solution, without disruptively typing in front of everyone.

I recorded the meetings using the Google Recorder app on my phone in my pocket. The Google Recorder app is a really cool piece of software that, leveraging the Tensor chip and a proprietary AI model, can produce extremely high-quality transcriptions of the recordings. And it all works on-device, no network connection needed, and completely private from Google’s prying eyes.

At the time, ChatGPT had just released Advanced Data Analysis to subscribers, allowing users to upload .txt files to use as prompts. So, I exported the transcript from Google Recorder and fed it the unintelligible wall of transcribed text. The result was a shockingly good note sheet, and no effort on my part.

This little experiment solved a short-term challenge of mine, but also opened Pandora’s Box. Although I enabled the private mode on ChatGPT, used a local transcription service, and deleted the recordings and full transcripts later, I still felt uncomfortable about my actions. I did not ask for consent to record and knew that my little experiment was violating the assumed ephemerality the others in the meeting assumed they held. In one of them, the presenter asked people not to take pictures or recordings. I recorded anyway, anxiously.

Despite my ethical discomfort with the experiment, I continued to think about it whenever I struggled to remember something. I began to wonder about if I could turn this little shortcut I had created into a product.

Over the course of the following months, I imagined a mobile app that used always-on background microphone access, transcribed any words it heard, categorized them by the speaker, and then put them through an AI language model. Then the user could ask it any question about anything they forgot or needed explained better.

I imagined nightmarish designs, straight out of Black Mirror. One where the app sent push notifications after conversations letting the user know that they had sounded awkward or unintelligent, and exactly how. I imagined the AI suggesting to the user how they should treat people in the future, often in ways that increased the social standing or career goals of the user while hurting those around them. I imagined marketing telling users that they would be statistically more successful if they outsourced their social, career, and romantic decision making to the AI.

And over the next few months, it became clear that I was not the only one imagining this sort of a product. But perhaps I am the only one uncomfortable with it.

One startup, Rewind.AI, started a few months ago innocuously as a “search engine for your life”, with the initial product pitch being an app that collected historical data about your work and files to make it easy to retrace your steps, sort of like a better version of macOS Time Machine. And it all was stored locally on your computer, making it not any less secure than a backup drive. The product they have built is pretty cool, and probably very useful to a certain sort of sleep-deprived engineer desperately trying to remember how they got a piece of code to work last week but cannot remember now.

Since the explosion of GPT-3, their project has realized its potential. Now, the app puts all of your data into a GPT-4 chatbot that you can prompt. If you’re willing to waive the (temporary, solvable) security risks of having a literal keylogger connected to ChatGPT, it’s a pretty cool product. The testimonials they advertise on their landing page are quite glowing:

This is a neat product, one that I could see myself using in the future.

But just two months ago, they announced another product, one that threatens to bring my uncomfortable vision to light. The “Rewind Pendant”, a $59 necklace-mounted wearable, is designed to collect everything you say and hear.

And the very day before, entrepreneur Avi Schiffmann, famous for building ncov2019.live at age 17, announced Tab, effectively the exact same product.

These product pitches both fall head-over-heels to virtue signal about how concerned they are about the potential issues they could cause. They also offer very little in the way of solutions.

On the one hand, these are really cool highly useful products. On the other, the potential harms are highly concerning.

I am a techno-optimist. I am not a person who worries about “ethical design”, “dark patterns”, “risk”, or “trust and safety”. Unlike the overwhelming majority of people who write long essays about tech, I am actually optimistic about the future of humanity. But the more I think about the Rewind Pendant/Tab, I see its simultaneous inevitability and destiny to destabilize our most essential social norms of ephemerality and privacy.

Many people want a Rewind Pendant or Tab. A product that prevents you from forgetting anything ever is highly useful and greatly increases human efficiency.

But to a similar degree, the this device removes the ephemerality of any conversation. We live in a weird moment in human history where we now all have recorders in our pockets. Yet you safely assume that when you go up to and talk to a person their recorder is not on. And you don’t ask your friend if their recorder is on before telling them a secret, because you can assume with 99.99% accuracy that they are not recording you. I personally cannot imagine telling a friend any kind of secret while they are wearing Rewind Pendant.

While people may feel uncomfortable about the ephemerality of the conversations they have around a person wearing one of these devices, the real roadblock to adoption is the privacy concerns to the owner of the product. Both of these startups assure you that your data is stored with top-tier encryption standards that make it inaccessible to them.

This is either oblivious ignorance or willful deception on their part. Presumably the end-goal of these products would be that a user wears it their entire life, and never purges the data. Encryption standards are a perpetual cat-and-mouse game wherein algorithms are frequently created, exploits for them are found, and they are replaced. For this reason, the famous private messaging app Signal only stores message data for a very short time. And they are investing into research to make their encryption quantum-computer-proof.

And of course, let’s not pretend that encryption is some silver bullet to prevent all forms of snooping on data. After all, we live in the age of Pegasus. It should be assumed that every government has well-developed vulnerabilities to every internet-connected device.

We live in a time where the surveillance in our lives grows exponentially. The innovations of the past half-century have made life better for us all with a secret consequence. Writer Yasha Levine says in his book on the topic, Surveillance Valley,

In the 1960s, America was a global power overseeing an increasingly volatile world: conflicts and regional insurgencies against US-allied governments from South America to Southeast Asia and the Middle East. These were not traditional wars that involved big armies but guerrilla campaigns and local rebellions, frequently fought in regions where Americans had little previous experience. Who were these people? Why were they rebelling? What could be done to stop them? In military circles, it was believed that these questions were of vital importance to America’s pacification efforts, and some argued that the only effective way to answer them was to develop and leverage computer-aided information technology.

The Internet came out of this effort: an attempt to build computer systems that could collect and share intelligence, watch the world in real time, and study and analyze people and political movements with the ultimate goal of predicting and preventing social upheaval. Some even dreamed of creating a sort of early warning radar for human societies: a networked computer system that watched for social and political threats and intercepted them in much the same way that traditional radar did for hostile aircraft. In other words, the Internet was hardwired to be a surveillance tool from the start. No matter what we use the network for today—dating, directions, encrypted chat, email, or just reading the news—it always had a dual-use nature rooted in intelligence gathering and war.

And through each era of computing, we’ve had this surveillance side effect grow stronger in the shadows. According to Levine, early computers were created to help military data science. The internet was created to help intelligence offices share data.

Once it was discovered that it was also useful as a civilian tool, it was rolled out explicitly to create a way of collecting more data on how people thought, to use for military advantage.

Early social networking made it easy to surveil fringe communities and target individuals with propaganda, en masse.

The mobile era made all of that data collection a mandatory staple of everyday life, making every task today impossible without it.

The cloud era brought all corporate and personal documents to easy view of governments. Almost every website you visit today is either running on AWS, Azure, or Google Cloud. All of that data, centralized, for easy access. It also created the digital assistant phenomenon, where it was suddenly normalized to have an always-on listening device inside your house.

Now with AI, I can already see the new world of surveillance on the horizon. With each new era of this age of computers, we have greatly improved the quality of the tools at our disposal. But we have also created a new normalized wiretapping mechanism, each one more nefarious than the last.

Rewind Pendant, Tab, and other devices of their nature probably don’t bring the same benefit to society that these past innovations have. And the cost of it is at least on par with, if not worse than the rest of them. But wearing them benefits their user, giving them a competitive advantage over their peers. If the people making these are smart, they will advertise that people who wear them are more efficient and successful than those who don’t.

I wrote this because I think we need to get the conversation rolling on when this tech is okay to use. I personally will never talk casually with somebody wearing one. And the thought of companies enforcing these on their workers to ensure efficiency sounds nightmarish. But turning one on for a meeting with consent, like how Microsoft Teams will soon add an option to get an AI transcript and notes from a meeting you missed, is probably fine.

Good luck.

Subscribe now

It was recently announced that the U.S. Justice Department is filing action against Google for monopoly concerns. Here’s David McCabe and Nico Grant for The New York Times:

The Justice Department has accused Google, which is based in Mountain View, Calif., of illegally using partnerships with phone makers and internet browser companies to shut out rival search engines.

Thanks to agreements with companies like Apple, Samsung and Mozilla, Google has been the default search engine when billions of web users open a browser on their phones, tablets and personal computers. The government has argued that Google’s multibillion-dollar payouts to partners have prevented other search engines like Microsoft Bing and DuckDuckGo from accumulating a meaningful share of the online search market.

The Justice Department also claims Google’s practice of preloading its services on devices that use its Android software illegally helped the internet company maintain a monopoly.

As U.S. et al. v. Google goes to trial this week, the echoes of the landmark federal suit against Microsoft, a quarter-century ago, are unmistakable. In the Google case, as with Microsoft then, a tech giant is accused of using its overwhelming market power to unfairly cut competitors off from potential customers.

After having some time to collect my thoughts on this case and the possible ramifications of legislation that could come from it, I have some concerns.

Google, at least for some very brief time in the early 2000s, did do the ‘free market thing’ of simply making the better product in Search. But those days have long passed. It’s undeniable that there isn’t a lot of competition going on here:

The argument from the Justice Department for how Google achieved this is twofold: (1) Google signed a massive pool of deals with web browser companies to gain users. (2) Google’s larger pool of user data from being the default let them consistently deliver higher quality results to users. Therefore, it became impossible for competitors to build an equally competent product.

They also point to the fact that it takes at most four taps to change to another search engine, yet people always choose Google.

I’m something of a rare user. Prior to the recent developments of generative AI, I used DuckDuckGo, the popular privacy-friendly search engine, as my main because their page loads faster and I’d rather not have a direct extrapolation of the thoughts in my head compiled into a dossier.

But DuckDuckGo’s result quality sucks. And Microsoft Bing, Google’s closest competitor, who is just as privacy invasive, somehow manages to give worse results than DuckDuckGo.

This case is correct in that no other companies can compete in the way the market is currently set up.

Now enter the last decade. If search engine competition is not possible, maybe search engine speciation is. Search bars exist everywhere. And Google isn’t the only one who has made a good one.

Let’s look at what I get when I Google “google antitrust trial”

And search the same query on TikTok:

Or Microsoft Bing’s recent GPT-4 powered Chat mode:

Each of these has something unique to offer me. Google offers the most high-quality sources, frontlining The New York Times in all three categories: News, Perspectives, and the old-fashioned results. This is a very interesting choice. I at first assumed that Google had personalized the results for me, knowing that I am a chronic NYT reader. I checked: searching the same term over anonymous browser Tor gave me the same result.

NYT, along with four of the seven sites in the News section at the top have paywalls. On the one hand, the quality in terms of raw factuality of the services Google overwhelmingly promotes like NYT, Wired, WaPo, WSJ, etc. is consistently higher than any other service in the world. On the other hand, it makes information highly inaccessible when the majority of people don’t have a PhD in circumventing these paywalls. Their only option that isn’t paywalled is Wikipedia, whom they have received criticism for over-relying on.

Let’s evaluate their newer competition…

TikTok offers the most excellent visual answers, as well as quick explainers. According to Google’s own claims, a high fraction of people, particularly young ones, are more likely to turn to TikTok and Instagram than Google Search because of the more digestible design. One thing I like about TikTok’s search feature is that it obviously gives a boost to legacy media institutions, but it weaves them between random users. which gives someone who just spends one minute scrolling a variety of perspectives. But TikTok has obviously very large limitations in regard to the sort of answers that are actually useful, as well as the fact that they severely punish creators who use links, making a good portion of search queries serve useless results on TikTok.

Generative AI-based search solutions take the cake for an actual proper search replacement. Bing Chat is the best of these tools for the moment. It gives excellent answers with a medium to high degree of factuality and offers easy links to check its thinking.

Since I know most readers of Humanlike are the people who use these tools daily anyway, I won’t dwell on all of the great things about Generative AI in search. But the important thing is competition, and right now Google may have the worst product in the field.

I’ve tried Google’s Search Generative Experience, which about 40% of the time just will not give you an answer. And when it does, it’s often unrelated or unhelpful. Google also has Bard, which is similarly low in factuality.

For the first time in at least a decade and a half, Google’s #1 product has some signs of competition on the horizon, and this happens to be the time that the Justice Department’s long-planned case goes into litigation. How ironic.

And now I want to talk about the defaults argument. With the massive exception of Microsoft Windows, almost every consumer electronic device that can access a browser today uses Google’s Search by default.

Take a look at what browsers people use:

You may at first think that Google can proudly say that they own 64.1% of the browser market share. Wrong. Putting Apple’s Safari aside for a moment as it does not work on the overwhelming majority of the devices in the world, every popular browser beside Firefox runs on a Chromium base, which Google owns and controls.

This gives Google outrageously large control over how people view the internet. They can and have pushed obnoxious controlling norms of how the internet can work such as Manifest V3, which theoretically removes the capability to add custom privacy and ad filtering, as well as Web Integrity API, which theoretically normalizes limits the amount of customization users and developers can put on their browsers. These changes range from walked back to half-implemented, but they serve as precedent of Google using it’s power to attempt to push absurd rules over the web. And sure, developers and users can modify Chromium to remove these awful changes, and they do, but Google controlling how it works by default means that web developers will cater to whatever the majority of people use, and most users will have far less agency in what conditions they internet.

So the one competition to literally every non-Apple device in the world exclusively running Chromium is Firefox. Firefox… owned by Mozilla… who receives 90% of its funding from Google in exchange for being the default search engine. Google’s only browser competitor could not exist without their funding.

But I thought search default deals were anti-competitive? Now we are in interesting territory. Search deals give Google tremendous control of the search market. Getting rid of search deals gives Google entire control of the browser market. Google has a “100% default” situation on all devices, where the two situations where competition exists is at most half competition: Google controls the default search engine of Safari and Firefox while not controlling the browser, and in the opposite, Google controls the browser of Microsoft Edge while not controlling the default search engine. And any attempt to regulate this situation will only make it worse. Catch-22.

Assuming regulation removed Google’s ability to do search default deals, literally the only way to have a fully non-Google web browsing experience would be on an Apple device running Safari with altered settings. This would be very bad.

But what if this regulation could make the situation so much worse? The regulators’ war against the Google default will also have profound impacts on arguably the most important piece of software in the world: Android.

Android in 2023 is a contradiction. On the one hand, it’s doing quite well:

On the other:

87% of [American] teens own an iPhone and 88% expect an iPhone to be their next phone

Piper Sandler 2021 research on 10,000 American teens

What on earth, as a regulator, do you do when confronted with THIS information? The demographic that creates consumption habits for the rest of the world is at war with the highly consumed product and it’s YOUR job to decide if it’s too powerful.

In the argument of the antitrust case, Google’s rise to power was furthered by its use of Android’s defaults, by insisting that OEMs sell phones with their suite of apps installed in all markets outside of China.

A theoretical regulation that restricts Google’s ability to control OEM defaults on Android could have disastrous consequences for Android. Currently Google makes a massive amount of revenue from services like Play Store, Google Pay, and the advertising data harvested from users’ phones without consent.

So waaaa waaaaaaa one of the biggest corporations in the world wouldn’t make as much money. What’s the big deal?

If Google doesn’t continue to make tremendous profits off their open-sourced approach to Android, they will move (the already lackluster) development resources on Android to their own proprietary Pixel line of Android phones.

This would mean a few things: The base Android experience would weaken, making the already high barrier to entry for Android OEMs significantly higher, as each company would have to build more tech that competes with Google because they can’t rely on Google to do the software work for them. That would certainly mean more in the direction of a Pixel-Samsung duopoly, and pretty much every other OEM dropping out as they would have to massively upscale costs to create their own software suite a decade behind. Less competition and new ideas, which were previously the only strength of Android.

And it would likely mean literally zero in user privacy from Google, as every user would have to have Google services to use, for example, a navigation app because all Android mapping apps (Google Maps, Waze, that’s it) are owned by Google.

Assuming regulation preventing Google’s default deals passed, Apple would see massive gains. Google and other companies that give consumers choice would be wiped out.

We need tech regulation, but not this.

Subscribe now

Bethesda Game Studio’s upcoming video game Starfield isn’t just another game. After more than twenty years of building video games, Starfield appears to be a true passion project that will precede a change in human history.

In 1997, studio director Todd Howard wrote in an archived forum post, “Now a space RPG..that would be something”. At this time, Bethesda was not yet a proven success, and it would take 13 more releases to bring his studio to a point where it was ready to make this space-themed role-playing game.

Over the years, Bethesda proved themselves to be the leading innovator in the RPG space. Despite their relatively tame size peaking at about 500 employees today, they have led the development of the industry in tremendous ways. In 2002, they released The Elder Scrolls III: Morrowind, the first ever non-linear RPG where players could go where they want and do what they want. It also came with Construction Set, a tool that let players edit any part of the game with mods. This game also pioneered the concept of video game lore, where players could collect and read books, letters, and inscriptions to get a deeper view into the history and philosophy that set the backdrop of Tamriel.

They followed it with my personal favorite, The Elder Scrolls IV: Oblivion, which featured voice acting for every single character. Oblivion leaned into this innovation bringing a cohesive Main Quest with a cinematic grandeur and a series of more personal guild storylines that were just as riveting. Bethesda games are most known for the world, and by far the best part was its use of a “Radiant AI” world system. In Oblivion, every character has a job, family, friends, routines, and their own opinions. Randomly, characters would bump into each other in these paths and make conversation using their recorded dialogue.

The result was sometimes absurd but a glimpse into the future of autonomous agents that is now possible with LLMs. Being a player of Oblivion feels like waking up on a weekend and thinking, “what do I want to do?” Certainly, you can get to work on your main calling (saving the world by closing the gates from hell), but you could also go hunting, go fight at the Arena, or get drunk and be a nuisance to everyone in town. I started Oblivion during a particularly stressful time in my own life, so I simply explored the valleys and collected books from the churches to read while listening to the beautiful ambient soundtrack.

Oblivion was a tremendous victory for Bethesda, cementing them as a leader in the rapidly evolving RPG experience, and brought them close with Microsoft as it became a top seller on the Xbox 360. It also featured a horse armor DLC, where for $2.50 you could have a cosmetic that made your horse appear in shining armor with no gameplay effects. This seemingly innocuous innovation led to the Fortnite model: to prioritize engagement and long time player investment in the game through cosmetic acquisition over selling as many video games to as possible. Of course, many gamers still hate Bethesda to this day for this reason.

With plenty of money from their last two games and a new revenue stream to exploit, they bought out the Fallout IP from the failed Interplay Entertainment and released Fallout 3, which took the games from a birds-eye isometric camera to first person 3D graphics. I have not played Fallout 3 and thus cannot comment on its strengths but I heard that it brought a comparable experience to Oblivion but instead set within the thought experiment of what would happen in the event of nuclear disaster, and let players choices influence the ending through their actions in the main quest.

Bethesda’s following project would be their most important one. Releasing on November 2011, The Elder Scrolls V: Skyrim would go on to sell more than 30 million copies and win multiple Game of the Year awards. Skyrim didn’t reinvent the wheel for Bethesda games, it instead polished it as much as possible. The game’s ahead-of-the-curve visuals for the time and appealing ‘warrior versus dragon’ plot reminiscent of Beowulf made it an appealing take on the Elder Scrolls universe. Additionally, it significantly simplified the RPG mechanics and progression paths which made the game much more accessible to a casual audience. To this day, Skyrim is Bethesda's best selling game and the most modded singleplayer video game thanks to Bethesda's toolkit. Actually, it goes beyond that title because fans have even created a mod that allows multiplayer online play.

It should say something about Skyrim’s tremendous success that Skyrim is a more recognizable name than The Elder Scrolls in gaming culture. One recent trailer for Starfield features the phrase, “From the award-winning creators of Skyrim”.

One year after its release, one of Bethesda’s subsidiary studios, ZeniMax Online, released The Elder Scrolls Online, a massively multiplayer online role-playing game, or MMORPG, set in the Elder Scrolls universe. While ESO got a rocky reception at launch, it has continued to deliver new experiences for fans of the franchise, which has not received a sequel for over a decade due to the long development time associated with games of this size.

The Elder Scrolls Online was a noticeable shift of business for Bethesda with a $15 a month paid subscription, paid cosmetics more than ten times higher than Oblivion’s horse armor, and loot boxes that let players gamble real money for character power boosts that gave advantages over other players. It was clear to fans that ESO’s purpose was to make some cash to fund their real games. Despite its reception, ESO has ballooned to an impressive size with seven major expansions including Necrom, which launched last month.

Fans who may have been upset with ESO could take solace knowing that it was not distracting from the main team drilling away at their next ambitious project. In November 2015, Bethesda Game Studios released Fallout 4. Fallout 4 is still the most impressive game in the series, opening right as nuclear war begins and following the main character being put into cryostasis for 210 years, and reemerging see humanity begin to rebuild.

The game was not perfectly received. Notably, the game’s dialogue options were something like “good guy,” “bad guy,” or “sarcastic jerk” every time, greatly decreasing the player expression options. Additionally, the game’s narrative focus on finding your kidnapped son was entirely uninteresting to myself and many other players. I have played over 150 hours of the game and have never finished the main story for this reason.

While the narrative was disappointing, Fallout 4’s beautiful world and enjoyable gunplay made it a very solid experience. I did one playthrough where I used the Start Me Up mod that lets players choose their own character background story and spawn point and joined the Raiders to attack settlements. This way, I was able to experience the world of post-apocalypse New England from the opposite perspective Bethesda built it from, another showcase of the longevity and adaptability modding offers in games.

Fallout 4 continued Bethesda's tradition of caring deeply for their modding community, being the first ever video game on a console to allow mods through a platform in the bethesda.net website. Skyrim Special Edition, a re-release which would release the next year for the Xbox One and PlayStation 4 consoles would receive this functionality as well. It was also neat addition to the PC version of Fallout 4, removing the need for often difficult to use 3rd party modding tools such as Vortex. More than six years later, no other mod support features in other, including Fortnite’s UEFN and Counter-Strike 2’s Hammer 2 come even close to giving developers the ability to bring their own dreams to life within these established universes. My personal favorite mod for Fallout 4, Fallout Brotherhood - A Storyteller Quest Mod, gives a fully voice acted hour long interactive tour through the history of the Brotherhood of Steel, one of the main factions in the series.

Fallout 4 also had a really interesting settlement builder mode that let players construct settlements and build houses with materials they found in the open world. I personally did not care for this, but some players fell in love with it.

Just three years later, Bethesda would release another Fallout game, one that would set a legacy and permanently alter the public perception of Bethesda forever. On October 23, 2018, Fallout 76, a multiplayer online RPG would open its beta servers for users to stress test the servers and enjoy a glimpse at the first few hours. In true Fallout fashion, they gave it a cheeky acronym: B.E.T.A. (Break-It Early Test Application). And break it, they did.

Bethesda games have always been buggy. Once while playing Morrowind, I was sorting my inventory and accidentally dropped an item required for the main quest on the floor. I watched as it slid into a crack and then fell right through the floor geometry, unable for me to pick it up again. That was the rather unceremonious end to a game I had already put a good twenty hours into. However, most of the time, these bugs are simply funny.

But Fallout 76 took this to another level. You see, Fallout 76’s development had around two years of active development, less than half of that of Fallout 4. And in development, they had to retool the Creation Engine, which was made around having one player in the world with time centering around them, time pausing whenever they opened a menu, and the game speed being tied to the framerate. All of this infrastructure had to be rebuilt to fit an always online multiplayer world.

And then they had to build a proper Fallout game set in West Virginia 25 years after nuclear war with the high expectations the marketing team had set: a map “four times the size of Fallout 4”, having “sixteen times the detail” and “all new rendering, lighting, and landscape technology”.

And as if it couldn’t get worse, the game was not developed in Bethesda's main office, but by a relatively new studio that had never released a game before and was renamed from “BattleCry” to “Bethesda Game Studios Austin” just months before launch to trick players into thinking it was developed by the same Bethesda they knew and loved. This studio had no prior experience with the Creation Engine.

In order to achieve this insane task, they took some serious compromises to fit deadlines. Remember all that “Radiant AI” and characters and plot that made Oblivion great? All gone. The “story” of Fallout 76 would be told through “The Overseer’s Log”, a series of voice recordings, notes, and terminal entries found scattered throughout post-apocalyptic Appalachia. Because you were sharing the world with other people, the entire plot would, of course, have to be exactly the same for everyone, so player choices were out of the question. With some naive optimism, they imagined that PvP battles and interactions over voice chat would be a replacement for the character-based narrative loop.

Fallout 76 released and reviews were in and they were not happy. The game landed at a score of 58 out of 100 on game review aggregator Metacritic and a user rating score of 2.8 out of 10. YouTube user Joseph Anderson uploaded a supercut of just the buggy moments from his first ten hours of play time titled “The 1001 Glitches of Fallout 76”. That video is almost three hours long. Famous YouTube video essayist Internet Historian uploaded a video on the fiasco that ensued called “The Fall of 76”, which now has over 35 million views. Fallout 76 is on dozens of most disappointing video games of all time lists.

Players were upset, and rightfully so. Rather than selling the game on the PC platform Steam, Bethesda only released it on its own app, with a vague refund policy that soon turned to a ‘no refunds’ policy once they mistakenly gave a player a refund after 24 hours of playtime.

More importantly, players were upset because the promise of Fallout 76 was something genuinely new and interesting. The trailers depicted players exploring the sprawling vistas and autumn leaves to Country Roads, Take Me Home. People wanted a game like that. But logging into Fallout 76 would leave you disappointed almost every time.

As the months went by and Bethesda embarrassed themselves with broken patches and increasing monetization, Fallout 76 inevitably crashed and burned. While most players left, a small group remained, and they continue to love the game.

I think Fallout 76 has the best world out of any video game ever. I haven’t played the game in two years, but I still have photographic memory of most of the map. I remember feeling a sense of discovery and excitement as I wandered down to the Nuka Cola factory, a sense of sadness as I listened to The Overseer describe having to kill her husband because of his radiation poisoning, and a sense of utter fear in my heart as I did the game-ending quest, launching a nuke on Watoga to start the Scorchbeast Queen event, and meeting up with 15 other players who had joined me to take her down.

On March 13th, 2020, I received an email from Bethesda letting me know that I had been invited to sign a non-disclosure agreement to test their upcoming Wastelanders expansion early. While I had seen the basic idea of the expansion months before in trailers, I had no idea to which the scale the Wastelanders expansion would turn Fallout 76 around.

Review outlet GameSpot would call the expansion “some of the best Fallout content since New Vegas”. IGN said, “Many of the new quests rival the best from any Bethesda game in terms of storytelling.” My previously dead friend list bloomed full with people eager to play Fallout 76 again, many of which had not played since launch. The update, revamping the game’s story, adding interactable human characters, and finally ironing out many of the bugs that had remained since launch, was a tremendous success. The game also released on Steam on PC, the preferred gaming storefront to ‘mostly positive’ ratings, currently sitting at 75% of players recommending it.

But this enthusiasm didn’t remain. In an internal email dating about 16 months after the Wastelanders release date, Microsoft Gaming CEO said of Fallout 76 in its post-Wastelanders state:

In my view F76 is in this interesting place. Obviously started rough. Team stays focused on improving and finding larger audience. Feels like we either need to see this thing getting to 10M MAU across all platforms or decide to move on from it

From FTC vs. Microsoft case

And thus, Fallout 76’s updates slowed to irrelevance. Most of my friends who came back slowly faded away from the game as updates became smaller and sparser. In March 2022, development work was moved to Double Eleven, a small outside contract studio.

While it’s sad to see a game that once held universal hatred from fans to redeem itself only to fade into irrelevance, Fallout 76’s promise was fulfilled in the end. The game is now a gorgeous Bethesda-style world with a solid plot that you can play with your friends, and anyone who plays it today will certainly enjoy it. But the manner in which Fallout 76 was marketed and the sheer number of times customers were deceived by Bethesda has left a permanant mark. Many people swear they will never buy another game from them again, and I honestly can’t blame them.

But as Fallout 76 went through its highs and lows, another monumental shift was taking place inside Bethesda.

After multiple years of rumors, on September 9th, 2020, Microsoft announced it would be buying Bethesda for $7.5 Billion in cash. This gave them exclusive access to everything from Bethesda Game Studios, as well as popular franchises from the other studios under the Bethesda name, including Doom, Prey, and Wolfenstein. This purchase was a significant help to the stagnating Xbox brand as they pivoted to a “Game Pass” Netflix-esque subscription model.

This certainly explains why Fallout 76 was the failure it ended up being. It is widely considered that Bethesda shoved the game out the gate to up their cash flow and profits to make them look more immediately appealing to their acquirer.

Many industry pundits believed this to be a bad deal, citing the general stagnation and mediocrity of many of these franchises, especially with no new Elder Scrolls or Fallout game in sight for another decade.

But interestingly, one prominent name on that announcement was a game that didn’t even exist. Starfield, which had only been announced in a 1-minute vague trailer back in 2018, was listed next to their top-tier franchises.

I believe Starfield will shape up to be one of if not the best pieces of art from the 21st century. I believe Todd Howard and the Bethesda Game Studio’s entire project have built every previous game in service of this one. And I believe that this game will culturally shift humanity’s goals permanently, pushing the desire to colonize the stars to the forefront of humanity’s collective mind.

Starfield is Bethesda's space exploration RPG. It follows Bethesda's traditional design of building a character and traveling wherever you want, this time beyond just our planet, instead one thousand planets.

Each of Bethesda’s projects of the past have been building directly toward this goal, each fleshing out designs, features, and systems upon which Starfield is now being built.

As the character creator, base builder, first-person combat mechanics, story design, world design, and the Xbox console finally hit its apex with the Series X console, Bethesda’s Starfield is finally ready to hit the world.

And who could forget, Bethesda’s work with Microsoft to bring it to life. This is a tremendous accomplishment no doubt only possible due to their collaborative work with Microsoft. In order to reach the insane task of creating the one thousand planets, I am fairly certain that Bethesda leveraged Microsoft’s cutting-edge AI development teams to procedurally generate them. It’s likely that Bethesda chose to be acquired by Microsoft, or even at all, because of Microsoft’s strength in AI.

But more important than the technological and economic forces that it took to bring Starfield together, it stands to release at one of the most interesting points in humanity’s history.

While we’ve always gazed into the night stars and asked “What’s out there?”, it would take until the 20th century for us to reach the moon, the first step made off the planet (or who knows, I’m not gonna tell you what to believe). It’s now been 50 years and we haven’t done it again.

In the years since, nearly every part of life has become less stable and problems such as inequality, environmental degradation, information control, privacy, and our collective identity through religion and nationality have become much worse, and solutions even more difficult. It’s getting easier to look at the desire of space colonization and ask: “Why worry about that now?”

Especially within my short lifetime, the coronavirus pandemic significantly exacerbated every one of these issues in only two years, shocking everybody with just what sits squarely in front of us.

While the wills of the populace have always pushed for myopic small reformations to make life slightly more bearable, art has been one of the few things that has challenged us to imagine something greater, such as space travel.

In these past 50 years, Star Wars provided this influence for some time. But over time, their theatrical films became Disney-ified trash and the extended universe began to lose its charm. Star Wars’ premiere project of the covid-era was The Mandalorian, a much more phoned in small adventure following one guy trying to save his pet. Passable entertainment, but certainly not challenging our greatest imaginations of the beauty and possibility of the universe. And the warriors fighting for this have noticed the Star Wars absence.

Enter Elon Musk.

During the now-defunct games industry event E3 in 2019, Todd Howard joined the billionaire space-venturing oligarch Elon Musk to discuss Starfield. Here he told the public that he had his SpaceX company assist Bethesda developers in ensuring the accuracy of space travel. And more interestingly, why:

I think movies, video games, (and really, I think people spend a lot more time on video games than movies these days) can be pretty thought-provoking and inspiring about what’s the future that we want

After over twenty years of figuring out exactly how to build the role-playing game, Bethesda will release Starfield on September 6th, 2023. Starfield will bring the space imperative to a new generation of youth. Advances in game design, graphics technology, AI, and the Xbox console are all just now ready to execute a truly great video game about space exploration. You’re going to notice the impact.

Subscribe now

Over the past several years, there has been a growing relationship between governments, philanthropic foundations, academia, and the big tech companies. Starting with the Twitter Files investigation, a web of journalists have begun to investigate how these groups and their connections have attempted to sway public opinion.

In particular, there has been a worrying movement in the tech industry that has gained control of social media and search engine services. Journalist Matt Taibbi calls this the “Censorship Industrial Complex”. The idea is government agencies, NGOs, philanthropies, the mass media, and academia working in lockstep to police the acceptable opinions and facts that get to thrive in our information ecosystem.

Today I want to discuss one specific philanthropic organization, their web of funding, and the race to regulate humanity’s future. This idea came to me listening to The New York Times’ Hard Fork podcast, where they started a recent episode by discussing a petition that had been signed by many of the top figures in AI across the leading companies, universities, organizations, non-profits, and beyond. This petition, created by the “Center for AI Safety”, was only one sentence long:

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

Immediately, the name “Center for AI Safety” sounded like some evil influence operation to me, so I decided to look into it. As it turns out, they have received over $6.5 million from Open Philanthropy (1) (2), which is owned by Facebook co-founder Dustin Moskovitz and former Wall Street Journal writer Cari Tuna. Now here’s where it gets interesting: The vast majority of the signers of the petition have financial and employment ties to the biggest tech companies, governments, and even the people running the petition. None of this information is disclosed on the petition website.

Firstly, twenty-one OpenAI employees have signed this petition, including the CEO Sam Altman. In 2017 OpenAI received $30 million from Open Philanthropy. Additionally, Open Philanthropy’s co-CEO Holden Karnofsky signed the petition. Him and his wife were OpenAI board members until they left to start competing AI firm Anthropic in 2021.

Along with this, a laundry list of other Open Philanthropy and otherwise special interest-funded names appear on this list. Fourteen names, including the first listed ones are employees of the University of Toronto’s computer science department. In December 2020, Open Philanthropy gave a $520,000 grant to University of Toronto. University of Toronto’s AI research has also received a grant of undisclosed size from CIFAR, which is funded by the Canadian government. Even more notably, they received $350,000 from the Rockefeller Foundation to research AI governance in September 2021. The Rockefeller Foundation has a history of ethically debatable activity that is far too long to describe here and is part of journalist Matt Taibbi’s “Censorship Industrial Complex” list.

Next, two names appear from U. Montreal, which received a $2.4 million grant from Open Philanthropy in July 2017. In particular, one of the named signatures is Yoshua Bengio, which the grant website provisions $1.6 million to power his own research. His department also received $125 million from CFREF, an arm of the Canadian government, to build “robust, reasoning, and responsible” AI. He has since went on a media tour spreading worry about the direction AI development is going.

These are only a few examples that I bothered to research that are paid for by the owners of the petition. Beyond this, the petition is filled with the names of big tech and government figures who should be looked at closer. 72 names are from Google. 14 names are from the Google-funded Anthropic. Almost every name here is from big media organizations, academia, and the most powerful tech companies. It begs the question, what is their motive? Are they really morally concerned that the technology they are building will be dangerous to humanity? Or have they figured out that bringing regulation will enclose competition and allow ideological domination of this new technology? I find it much more likely that, like how Google and Facebook’s market shares increased in the EU as GDPR went into effect, the current dominant forces in AI are using safety as a rhetorical hook to attain regulatory capture.

These are AI bootleggers, a term I borrow from Marc Andreessen’s recent essay, where he dichotomizes the AI safety movement into two parts:

• “Baptists” are the true believer social reformers who legitimately feel – deeply and emotionally, if not rationally – that new restrictions, regulations, and laws are required to prevent societal disaster.
  For alcohol prohibition, these actors were often literally devout Christians who felt that alcohol was destroying the moral fabric of society.
  For AI risk, these actors are true believers that AI presents one or another existential risks – strap them to a polygraph, they really mean it.

• “Bootleggers” are the self-interested opportunists who stand to financially profit by the imposition of new restrictions, regulations, and laws that insulate them from competitors.
  For alcohol prohibition, these were the literal bootleggers who made a fortune selling illicit alcohol to Americans when legitimate alcohol sales were banned.
  For AI risk, these are CEOs who stand to make more money if regulatory barriers are erected that form a cartel of government-blessed AI vendors protected from new startup and open source competition – the software version of “too big to fail” banks.

And most importantly:

A cynic would suggest that some of the apparent Baptists are also Bootleggers – specifically the ones paid to attack AI by their universities, think tanks, activist groups, and media outlets. If you are paid a salary or receive grants to foster AI panic…you are probably a Bootlegger.

Call me a cynic, baby.

In this essay I have used one particular example of a viral petition to show the strong financial and institutional powers leading the AI safety movement in an attempt to argue that the authenticity of nearly all of the thought leaders deserve questioning. Nearly all of the people I have mentioned stand to lose their entire livelihood if they speak contrary to their side’s orthodoxy. I find it incredibly disturbing that the corroding forces that drive our mainstream political arena are already corrupting the newly important AI safety space. This is not to say the AI accelerationist movement is free from ulterior motives. The big forces I see in this movement right now are:

• The previously quoted venture capitalist Marc Andreessen who stands to profit from lack of AI regulatory capture as he invests in new companies.

• Meta, who is leveraging the open-source community to bring them AI dominance and thus stands to gain from AI deregulation (I am working on another piece about this topic).

• The ragtag group of Twitter anons, most of whom work in AI startups.

So to say that either side is completely absolved of responsibility when it comes to authenticity questions is probably inaccurate. But the comparison is just not close. In conclusion, the AI safety movement is clearly an outgrowth of the conspiracy encompassing many of our most powerful institutions to eliminate dissenting and free thought in the western world and should be regarded as such.

Subscribe now

Subscribe now