What Happens When Software Eats The World?

Previously we discussed how software and markets are eating the world. We discussed how the internet ate media, and how it needs to eat education and healthcare. 

How much more will it eat? It certainly won’t stop at newspapers, that’s for sure. 

The truth is there’s no limit: Software will eat every part of the world until it wears down the fabric of reality to the quantum level. 

What does this mean?

Zooming out: At some point in their history, all prior civilizations have collapsed, allowing new ones to take their place. From the Mayans to the Greeks to the Romans, we’ve seen history repeat itself time and time again — civilizations rise and fall, except now things are different because we have exponential technology. Thanks to increasing non-zero-sumness, we’re all increasingly interconnected, and one civilization’s collapse could mean the collapse of all civilizations (think MAD). And because our global supply chain is so integrated with our biosphere, one civilization’s collapse also threatens our planet at large.

Think about it. We have exponentially more people making exponentially more impact per person, and the resulting exponential growth is reaching escape velocity. Eventually you just can't externalize anymore.

Our intuition isn’t scoped for the challenge we now face. We increasingly have the power of the gods without the wisdom of the gods, and any society that has an exponential increase in risk without an exponential increase in choice-making capacity faces the risk of self-termination

So for the first time one civilization’s collapse is truly existential. We can’t put the technological curve caps back in the bag — we must innovate our way through.

Balaji says that the internet increases variance, and in this context we could say that though things are currently getting exponentially better, they could also just as easily become exponentially worse, leading to destabilization. 

How does this relate to software eating the world? Software and markets are paperclip maximizers. They’re systems optimized to reduce everything complex into something simple — like making a tree into wood or money, for example. 

Solving existential risks requires root cause analysis — if you try to solve one without finding its root cause, you’re just playing whack-a-mole. One of the root causes of societal collapse is the presence of and our participation in rivalrous or “zero-sum” games. Rivalrous games are inherently zero-sum — in order for you to win, I have to lose, or vice versa. It’s a fragile system: I can only advance my own well being at your expense, or at the expense of the commons.

Anti-rivalrous games are the opposite; they’re “positive-sum” — either we both win together, or we lose together. These games are anti-fragile since everyone’s incentives are aligned; I can advance my own well being to your benefit and to the benefit of the commons. 

A rivalrous good is valuable because it’s scarce, like gold or Bitcoin. An anti-rivalrous good is valuable because it’s abundant, like language. If only two people use a language, it’s not as valuable as if a million people use it, which is a bit backwards: If air isn’t worth anything, because there’s enough of it for everybody, and I can’t increase my competitive advantage over you by hoarding more air, then air is literally worth nothing. Even though we all die without it, it has no economic value to any of us. As a result, we pollute the air, fill it full of CO2, and pull out the O2 as we oxidize hydrocarbons simply because it’s not a scarce asset. Protecting air doesn’t provide any individual person a competitive advantage, so we don’t give it much weight. Gold, on the other hand, we fight wars over and destroy entire environments just to mine it. We don’t care about the same environments that create the oxygen in the air because we’ve found we can put other, more scarce, assets like gold in a safe deposit box, look at it, and profit as a result.

Put differently: Abundant things become economically worthless, even if they’re foundational for life. Scarce things make people rich, even if they’re meaningless. These ideas incentivize people to artificially manufacture scarcity and avoid abundance everywhere, and this perverse incentive model results in what we call “multipolar traps” — scenarios where the things that work well for individuals locally are directly against global well-being.

For example, if I don’t develop AI or cut down trees, I’ll lose, because I’ll be worse off today without new technology or the products trees could provide. But if I do do those things, I won’t lose now, but we’ll all lose later. This is a scenario where the short-term incentive of the agent locally is directly against the long-term global wellbeing. And if I’m internalizing externalities, but you’re not, then I’m losing. So no one can internalize externalities and still survive.

These multipolar traps occur all over society, both in the private and public sectors: Take two employees at the same company, for example — both want the same promotion and try to sabotage each other, even though that harms the company. They act this way because their individual incentives are at odds with the company’s incentives.

Our government provides another example: Different government agencies competing for the same chunk of budget will sometimes seek to undermine each other to get a bigger slice of the pie.

When things that should be positive-sum become zero-sum, bad things happen.

Patents are a good example of taking something that should be positive-sum (knowledge) and making it zero-sum. Look at this this way: the best computer the world could build doesn’t exist because Apple has some of the IP, Google has some of the IP, and 10 other companies have the rest.

If someone made software they could give to the whole world for free (after covering costs), but instead they patent it and sue you if you pirate it, even though they have no marginal cost, that blocks innovation and enriches the individual at the expense of society.

These multipolar traps and rivalrous games have always caused the collapse of civilizations, but this time is different because eliminating our civilization means eliminating the rest of the planet along with it.

Regulation falls short of fixing these problems because you can’t legislate against long-term incentives. If you make a law that says it's illegal to do X, but it's still profitable to do X, companies and governments will still do X.

Even our sacred democracies can’t solve these problems. When Winston Churchill said democracy is the worst form of governance (except for all the others), he was conceding that getting lots of humans to agree on anything is just a hard thing to do.

We need to take rivalry out of our systems, making them positive-sum in the process for all participants and the commons. The incentive of every agent in the system and that every person or group has to be better aligned with the well being of every other agent *and* of the commons writ large. This  involves a real-time balance sheet of the commons that internalizes more externalities, and we now have new technologies that can help us do exactly that (Sensor grid, IoT, and satellites to name a few).

We know there has to be a structure whereby actual innovation is rewarded in proportion to its value in a way that rent seeking is eliminated entirely and the reward takes into account externalities produced by innovation.

We also need new governance systems. How can we avoid both top down organizational models of consensus making and bottom up models which make decision making increasingly time consuming to the point where we’ve missed the boat on solving the problem entirely?

As Robert Wright wrote in Nonzero, any time there’s an increase in social complexity, we’ve needed an increase in moral progress to complement it. 

Today is no different. We need a change in consciousness that acknowledges the interconnectedness of our fates. When we identify as interconnected parts of an interconnected universe (rather than separate discrete things) we inherently become more positive-sum. In effect, we stop thinking there is any definition of success for oneself that isn't also success for the whole.

Native American tribes did a good job of this. When kids asked their elders "What's that?”, rather than saying “It's a crow” or “It’s a tree, the elders would say, "that's the great spirit expressed as a crow" — their initial semiotics/ontology had the interconnectivity of everything first, and then the distinction second.

Consider for a second what or who you are without the context of the world around you: What are you without the atmosphere? What are you without the plants that generate the atmosphere? You don't even exist.

So understanding the interconnectedness of our society helps solve the rivalry problem, because zero-sum games require a separate sense of self, leading to rivalrous competition for some scarce asset. If instead, however, we see ourselves as emergent properties of everything, then we can't get ahead at the expense of that which we're connected to. When we internalize interconnectedness, we become positive-sum by default.

For inspiration for what these new anti-rivalrous and totally positive-sum systems could look like, consider the human body.

The heart and lungs are not rivalrous. They’re not competing to extract scarce resources and hoard them. Instead, they operate in this radically necessary symbiotic relationship. And if you tried to model these organs in a relationship where the heart and lungs were competing for scarce resources, the body would die very quickly.

So in conclusion, software will eat the world down to the quantum level, and that will encourage us to fundamentally re-imagine all elements of society, introducing exponential benefit but also exponential risk.

This piece was inspired by conversations with Daniel Schmactenberger and the broader Game B community.

---

Until next week,

Erik