Incentives, Not Destinies

I’m not a huge fan of Yancey Strickler’s ideas, but they are in the same realm as mine¹, so I continue to pay attention. His “The breaking of the world” continues in this it feels close but…off? vein, but has a couple of things that resonated, starting with the phrase I lifted for this post’s title.

But starting with his post’s title: I feel like it would have served his ideas better to call it The breaking of my world, as after reading it and looking back at the title I had to wonder, whose world?

One of the best parts is actually a quote from Hacker News of all places.

In a Hacker News discussion about a recent batch of proposed mathematical proofs published by OpenAI, a commenter who had worked on one of the problems for 24 years writes:

I don’t know what to think exactly. I spent thousands of hours on that problem. I really enjoyed it. Hearing that it is solved somehow makes me sad in a far-off way, like hearing an ex-girlfriend died suddenly in a car crash. I don’t know, there’s probably a lot of people feeling odd emotions tonight.

I recognize this odd emotional state. For me, it shows up as loss of trust. Because digital media is now so easy to produce and so hard to verify, all forms of digital expression feel less valuable to me than they did two years ago. I’m less open, more suspicious. When I encounter something interesting online, my first instinct is no longer to share it. Now I doubt whether it’s real.

The state of things has changed my output too. I’ve delayed sending this piece for more than a month, and there’s another piece that’s been sitting in my drafts folder even longer. Why add to the noise when so little feels honest and true?

The Hacker News commenter’s description evokes a familiar emotion, that of some identity-forming element or aspect of one’s self from one’s past terminating–a long-held assumption proved inaccurate, the realization you haven’t thought about a life-altering loss on the day of its anniversary, hearing a phrase uttered that used to quicken the pulse but now is barely perceived. Earlier today I smelled something, some combination of cigarettes and a specific cologne, that transported me to my high school best friend’s room. I haven’t even spoken to him since soon after graduation. I certainly haven’t experienced that particular aroma.

But Yancey’s Why add to the noise as a novel feeling? One of the most frequently deployed² quotes on this very blog is I frequently experience moments of nausea in bookstores at the thought of adding one more page to all that fucking print, from 1993, the year Tim Berners-Lee made his World Wide Web project public, long before it became the attention annihilating³ medium it is now.

Ok back to the Yancey post:

People describe AI as inevitable. Like a storm approaching on the horizon. But this isn’t the weather. These are decisions being made by a small number of people. What to build, how quickly to release it, what risks to take. Beneath it all, the promise of AGI calls: an apple offering total and complete knowledge.

But once we take that first bite, can we ever return to the garden?

I fear what gets destroyed on the way there. Livelihoods, skills, relationships, a reason to call someone. Our agency, our abilities, and our independence could very well diminish as we become more dependent on always-on machines and less on each other and ourselves.

Let’s get the 🥱 Garden of Eden metaphor out of the way and move on to …livelihoods, skills, relationships, a reason to call someone…we become more dependent on always-on machines and less on each other and ourselves. The first two on his list are in an entirely different category than the rest. The original Luddites didn’t win, we all wear machine-made garments from machine-made textiles. And yet we can care about other humans like the Luddites did. And LLMs are the thing that’s finally going to push us past some Rubicon of mediation? Because it convincingly simulates language?

People have been trying to conflate human language with the numinous for a long time. Millions of people believe that a bunch of ancient texts compiled by politically-appointed religious leaders in the 300s represent the literal and complete teachings of God.⁴ And we’re surprised when a machine learning model that talks like a person is confused with a higher power?

Communicating over time and space is a powerful technology, be it books, radios, web pages, or “social media” posts. The problem isn’t the technology. The problem is older than the technology. It’s enclosure.

When outcast nerds started writing each other across the world with email, and then started writing and reading each other’s ideas over the World Wide Web, we felt a new wave of connectedness not owned by economic powers. When we used our unique skills of writing code and gluing together things that ran on computers that could send information over telephone wires to create blogs and feeds and animations and messages, we thought those people who created and controlled these new modes of mediation were special boys like us, that they would never become like the old bullies we hated.

We were so very wrong.

On to my favorite phrase from the Yancey’s piece. It actually landed above the above-quoted portion.

I understand why someone with an enormous investment in these machines wants them to keep going. I get the attraction of being first as a founder. Race conditions are hard to escape. But those are incentives, not destinies.

These new LLM-based tools are extremely powerful, there is no doubt. But they take a lot of compute. In terms of the capital required to build and run them vs. the value they create, it seems pretty clear that all these companies are out over their skiis. Software, as a technology with economic value, has proven a speculative model enough times for this moment to even be a thing. (I’m old enough to remember a time when companies had to be profitable to go public.) But the math in this instance should cause anyone to stop and squint.

The entire narrative around “AI” is one of incentives, not technology. And our collective agreements about our destinies is what creates those incentives. We could revise those agreements. But if history is any indication, we won’t.

Last night I went to see my friend Chris Holt play at the Kessler Theater and he covered a song by the late Carter Albrecht.

I promise you, you can change the world
And they’ll just change it back again

We think LLMs will help us speak with God, but it’s the musicians we should be listening to. Always has been.