email #51 — on taking alarming things seriously, while trying to be a person
4 views
Skip to first unread message
Leon Kraiem
Nov 14, 2025, 4:32:58 PM11/14/25
to Leon’s List
Hi friends,
I write to you from Cambridge, Mass., where I'm staying with my friends Heila and Yoni, both of whom are researchers in the field of AI safety. I'm here primarily to visit another friend of mine, and to meet his fiancée, of whom I've heard much but seen none; but I've also had a standing appointment with my hosts for some time, to talk through with me the very convincing and very scary arguments, which they maintain are mistaken, that humanity is racing toward self-inflicted extinction through the reckless creation of machine superintelligence.
On the train here, I read the little book that's taken these arguments mainstream, and won them attention on the world's largest podcasts and news programs and front pages: it's called "If Anyone Builds It, Everyone Dies." The argument basically boils down to: We've figured out how to make smart, tenacious machines, but we haven't figured out how to control them, or even understand them. If we keep making them smarter, and more tenacious, but not more controllable or understandable, one of them will eventually set its mind on a goal to which our own flourishing is inconvenient, and it’ll sweep us out of its way.
The authors — Eliezer Yudkowsky and Nate Soares, two serious people who have been in this field for a long time — don’t claim that AIs are sentient or malevolent, only that machine-learning, like biological evolution, is complex and chaotic and takes twists and turns that you can’t predict, even if you specify the outcome you’re looking for.
The two examples that stuck with me: Human evolution maximizes for reproductive success. So humans evolved a sex drive. But once we got smart enough to separate sex from reproduction, we made birth control, and now the birth rate is falling, because the means-to-an-end took on a life of its own. Similarly, evolution wired us to seek out nutritious, energy-rich foods. So we developed a taste for sweet, and greasy, and fatty foods, which tended to be nutritious and energy-rich. But once we got smart enough to separate the tastes from the nutrition, we made ice cream and Doritos, because the means-to-an-end took on a life of its own.
When an AI person says, “The machine won’t hurt us — it’s not designed to! We told it to seek out our approval,” what they’re missing — according to these guys — is that we “tell it” to cooperate with us the same way evolution “tells us” to reproduce, and eat healthy — by specifying an end-goal, but not the means-to-an-end to arrive there. And over the course of all those iterations (biological generations for us, rounds of feedback and automatic, under-the-hood parameter-tinkering for the AI), that evolutionary process creates weird, novel epiphenomena that were never intended, like libidos and taste buds, that end up overriding the original end-goal. AIs are designed to want things, and to pursue those things creatively and tenaciously. But we can’t predict what sorts of things they’ll end up wanting, which means it’s only a matter of time until we lose control of them.
The authors explain these ideas better than I do, and I recommend you all read the book. The upshot is: Unless humanity starts treating AI like nuclear power — a valuable, useful tool, but one that can go dramatically wrong if we aren’t really careful with it — and especially capable AIs like nuclear weapons — capable not just of apocalyptic destruction, but of setting off a cascade of that type of destruction that eventually gets all of us killed — we won’t last much longer as a species.
—
Heila and Yoni take a more circumspect view of the “superintelligence” thing. They are both very concerned about other sorts of AI-induced disasters — the development and use of novel, targeted bioweapons, floods of misinformation that drive people literally insane, unprecedented unemployment that upends the social order and gives rise to strongmen and high-tech warlords, that sort of thing. But the Terminator-type kills-us-all-to-make-paperclips scenario isn’t one that keeps them up at night.
I write to you from Cambridge, Mass., where I'm staying with my friends Heila and Yoni, both of whom are researchers in the field of AI safety. I'm here primarily to visit another friend of mine, and to meet his fiancée, of whom I've heard much but seen none; but I've also had a standing appointment with my hosts for some time, to talk through with me the very convincing and very scary arguments, which they maintain are mistaken, that humanity is racing toward self-inflicted extinction through the reckless creation of machine superintelligence.
On the train here, I read the little book that's taken these arguments mainstream, and won them attention on the world's largest podcasts and news programs and front pages: it's called "If Anyone Builds It, Everyone Dies." The argument basically boils down to: We've figured out how to make smart, tenacious machines, but we haven't figured out how to control them, or even understand them. If we keep making them smarter, and more tenacious, but not more controllable or understandable, one of them will eventually set its mind on a goal to which our own flourishing is inconvenient, and it’ll sweep us out of its way.
The authors — Eliezer Yudkowsky and Nate Soares, two serious people who have been in this field for a long time — don’t claim that AIs are sentient or malevolent, only that machine-learning, like biological evolution, is complex and chaotic and takes twists and turns that you can’t predict, even if you specify the outcome you’re looking for.
The two examples that stuck with me: Human evolution maximizes for reproductive success. So humans evolved a sex drive. But once we got smart enough to separate sex from reproduction, we made birth control, and now the birth rate is falling, because the means-to-an-end took on a life of its own. Similarly, evolution wired us to seek out nutritious, energy-rich foods. So we developed a taste for sweet, and greasy, and fatty foods, which tended to be nutritious and energy-rich. But once we got smart enough to separate the tastes from the nutrition, we made ice cream and Doritos, because the means-to-an-end took on a life of its own.
When an AI person says, “The machine won’t hurt us — it’s not designed to! We told it to seek out our approval,” what they’re missing — according to these guys — is that we “tell it” to cooperate with us the same way evolution “tells us” to reproduce, and eat healthy — by specifying an end-goal, but not the means-to-an-end to arrive there. And over the course of all those iterations (biological generations for us, rounds of feedback and automatic, under-the-hood parameter-tinkering for the AI), that evolutionary process creates weird, novel epiphenomena that were never intended, like libidos and taste buds, that end up overriding the original end-goal. AIs are designed to want things, and to pursue those things creatively and tenaciously. But we can’t predict what sorts of things they’ll end up wanting, which means it’s only a matter of time until we lose control of them.
The authors explain these ideas better than I do, and I recommend you all read the book. The upshot is: Unless humanity starts treating AI like nuclear power — a valuable, useful tool, but one that can go dramatically wrong if we aren’t really careful with it — and especially capable AIs like nuclear weapons — capable not just of apocalyptic destruction, but of setting off a cascade of that type of destruction that eventually gets all of us killed — we won’t last much longer as a species.
—
Heila and Yoni take a more circumspect view of the “superintelligence” thing. They are both very concerned about other sorts of AI-induced disasters — the development and use of novel, targeted bioweapons, floods of misinformation that drive people literally insane, unprecedented unemployment that upends the social order and gives rise to strongmen and high-tech warlords, that sort of thing. But the Terminator-type kills-us-all-to-make-paperclips scenario isn’t one that keeps them up at night.
I hope my they can convince me. The more important thing, of course, is that they be right — I hope the distance between the world as it currently exists and the end of all life on earth to be a few reckless decisions by Elon Musk and Sam Altman. But I also hope my friends can convince me that they’re right, because if the distance between the world as it currently exists and the end of all life on earth is a few reckless decisions by Elon Musk and Sam Altman, then preventing those few reckless decisions would sort of have to be the only thing I care about, until the threat is successfully taken care of.
(What the authors of the book advocate is some sort of international treaty, along the lines of the nuclear NPT, that treats mega-data-centers and super-duper-computer-chips as akin to uranium enrichment facilities and ballistic missile factories. There may not be much time, and it’s a very serious undertaking — but compared to the nuclear thing, there are a lot fewer moving parts, and the arms-race dynamics pose less of a problem, because human-human conflicts are probably irrelevant to an out-of-control AI; so this is a very possible thing — and even if it had only a 1% chance of success, you’d sort of have to pursue it if the alternative is imminent extinction.)
I got off the train in a daze yesterday, unhappy with how convinced I was, and asking the question: Is this my whole life now? When I was seventeen, and I studied at washed-up-hippie farm-school in Maine, one of the fellows — fresh out of college, and even fresher off a cross-country sustainable-ethanol-powered bus tour, which I guess was supposed to "raise awareness," as they say — told us that working on “climate justice” (in the lingo of the day) is hard, and dispiriting, and asks a lot of you — but that at the end of the day, you can take comfort in knowing that it’s the most important thing you could possibly be doing. That always stuck with me.
I, of course, didn’t devote my life to fighting climate change — I was and remain an environmentalist, but there were a lot of other problems I also found worthy of my attention, like animal rights, and refugee resettlement, and freedom of speech, and the rule of law. Also, you know — making friends, having new experiences, eating, praying, loving, etc. I admired that farm-school fellow’s earnestness, but I dismissed her monomania, as unnecessary and probably unhelpful in a world as complicated as this one.
The point is, though: If she were right descriptively, then she’d also be right prescriptively. If there’s one problem that threatens everyone and a limited amount of time to solve it, then you should probably leave everything and everyone behind in order to play your part. That attitue makes you boring, and unavailable, and it can easily make you ineffective, if you don’t take a wide-enough view, but so long as you’re strategic, it’s sort of the only justifiable way to live, if that scenario maps onto reality. Think about the Jehova’s Witness lady in the subway, who’s trying to save good, decent people from never-ending Hellfire. You can roll your eyes, or make a bless-your-heart expression, but really, given the universe she inhabits, what’s more appropriate to judge her for: proselytizing, or doing anything else, ever?
—
And so, having been introduced to enough doubt last night that I could get some sleep, make it through a work day, and then sit down to hammer out this email without selling all my possessions and becoming a full-time AI nut, it feels appropriate to make some record of this dramatic yo-yo of mental states, which I suspect will recur in the near future, because ultimately my faith in Heila and Yoni’s deflationism is more a function of how smart and informed I know them both to be, and less a function of what they said last night having actually neutralized the book’s arguments for me.
In July 2023, I had lunch with my friend Ben Ross in Jerusalem — keep reading if you’ve heard this one before, I tell it a lot. At lunch, Ben predicted that Israel would be at war within the next six months. I asked him how I might make myself useful in such a scenario, and he recommended I return to journalism, and make it my full-time gig. That is, incidentally, how I wound up working for JPost, and then the Times of Israel.
The more relevant point, though, is: I went home in a daze then, too, and spent quite a while chewing on two basic truths: 1) that believing Israel to be on the brink of a major armed conflict seriously interfered with my ability to be and do in the world day-to-day, and 2) that this didn’t mean it wasn’t the case. And for the next few months, the thought of imminent war on the homefront became something of an obsession — I thought about how the country’s internal strife made us look weak, and I sent anxious WhatsApp voice notes to activist friends, asking whether the Jewish-calendrically-eery timing of the judicial overhaul fight frightened them, like it frightened me; I thought about troop movements and bomb shelters; on the cab to the airport on my way to the US, I looked through the window with the distinct thought: This might all be different when I come back. And when my flight back to Israel got cancelled, I was frantic to get the next one out, consumed by a vague — and, ultimately, correct — concern that something might happen that would strand me in the States.
I don’t want to stay in my ToI job much longer, and I’ve already applied for another ten-day Vipassana course in January, which would basically require I quit my job by New Years’. (It was brazen enough to tell the New Israel Fund I’d be taking a few extra days off Sukkot vacation to sit still for ten hours a day without talking, reading, or writing — try telling that to a team of sleep-deprived journalists who haven’t stopped liveblogging in fifteen years.) I think there’s a significant-enough career pivot in my future, whether that’s to the world of AI policy or something else entirely. But that’s normal caring-about-the-world, not crazy-street-preacher-caring-about-it. It doesn’t pose a threat to my personality, or my social life, or these emails.
If, on the other hand, those alarmists keep making this much sense…
—
Shabbat shalom. The world is broken, fundamentally and ubiquitously, and it is our job to fix it, via AI safety or otherwise; nevertheless, we’re in the world, not of it, and it’s a sacred obligation to set aside some time for being, rather than doing, an obligation that I intend momentarily to fulfill.
If we haven’t made plans yet, you know how to reach me.
Leon
I got off the train in a daze yesterday, unhappy with how convinced I was, and asking the question: Is this my whole life now? When I was seventeen, and I studied at washed-up-hippie farm-school in Maine, one of the fellows — fresh out of college, and even fresher off a cross-country sustainable-ethanol-powered bus tour, which I guess was supposed to "raise awareness," as they say — told us that working on “climate justice” (in the lingo of the day) is hard, and dispiriting, and asks a lot of you — but that at the end of the day, you can take comfort in knowing that it’s the most important thing you could possibly be doing. That always stuck with me.
I, of course, didn’t devote my life to fighting climate change — I was and remain an environmentalist, but there were a lot of other problems I also found worthy of my attention, like animal rights, and refugee resettlement, and freedom of speech, and the rule of law. Also, you know — making friends, having new experiences, eating, praying, loving, etc. I admired that farm-school fellow’s earnestness, but I dismissed her monomania, as unnecessary and probably unhelpful in a world as complicated as this one.
The point is, though: If she were right descriptively, then she’d also be right prescriptively. If there’s one problem that threatens everyone and a limited amount of time to solve it, then you should probably leave everything and everyone behind in order to play your part. That attitue makes you boring, and unavailable, and it can easily make you ineffective, if you don’t take a wide-enough view, but so long as you’re strategic, it’s sort of the only justifiable way to live, if that scenario maps onto reality. Think about the Jehova’s Witness lady in the subway, who’s trying to save good, decent people from never-ending Hellfire. You can roll your eyes, or make a bless-your-heart expression, but really, given the universe she inhabits, what’s more appropriate to judge her for: proselytizing, or doing anything else, ever?
—
And so, having been introduced to enough doubt last night that I could get some sleep, make it through a work day, and then sit down to hammer out this email without selling all my possessions and becoming a full-time AI nut, it feels appropriate to make some record of this dramatic yo-yo of mental states, which I suspect will recur in the near future, because ultimately my faith in Heila and Yoni’s deflationism is more a function of how smart and informed I know them both to be, and less a function of what they said last night having actually neutralized the book’s arguments for me.
In July 2023, I had lunch with my friend Ben Ross in Jerusalem — keep reading if you’ve heard this one before, I tell it a lot. At lunch, Ben predicted that Israel would be at war within the next six months. I asked him how I might make myself useful in such a scenario, and he recommended I return to journalism, and make it my full-time gig. That is, incidentally, how I wound up working for JPost, and then the Times of Israel.
The more relevant point, though, is: I went home in a daze then, too, and spent quite a while chewing on two basic truths: 1) that believing Israel to be on the brink of a major armed conflict seriously interfered with my ability to be and do in the world day-to-day, and 2) that this didn’t mean it wasn’t the case. And for the next few months, the thought of imminent war on the homefront became something of an obsession — I thought about how the country’s internal strife made us look weak, and I sent anxious WhatsApp voice notes to activist friends, asking whether the Jewish-calendrically-eery timing of the judicial overhaul fight frightened them, like it frightened me; I thought about troop movements and bomb shelters; on the cab to the airport on my way to the US, I looked through the window with the distinct thought: This might all be different when I come back. And when my flight back to Israel got cancelled, I was frantic to get the next one out, consumed by a vague — and, ultimately, correct — concern that something might happen that would strand me in the States.
I don’t want to stay in my ToI job much longer, and I’ve already applied for another ten-day Vipassana course in January, which would basically require I quit my job by New Years’. (It was brazen enough to tell the New Israel Fund I’d be taking a few extra days off Sukkot vacation to sit still for ten hours a day without talking, reading, or writing — try telling that to a team of sleep-deprived journalists who haven’t stopped liveblogging in fifteen years.) I think there’s a significant-enough career pivot in my future, whether that’s to the world of AI policy or something else entirely. But that’s normal caring-about-the-world, not crazy-street-preacher-caring-about-it. It doesn’t pose a threat to my personality, or my social life, or these emails.
If, on the other hand, those alarmists keep making this much sense…
—
Shabbat shalom. The world is broken, fundamentally and ubiquitously, and it is our job to fix it, via AI safety or otherwise; nevertheless, we’re in the world, not of it, and it’s a sacred obligation to set aside some time for being, rather than doing, an obligation that I intend momentarily to fulfill.
If we haven’t made plans yet, you know how to reach me.
Leon
Reply all
Reply to author
Forward
0 new messages