Is this you? As a journalist, you can create a free Muck Rack account to customize your profile, list your contact preferences, and upload a portfolio of your best work.
Claim your profile
Articles by Rob Bensinger
MIRI Newsletter #127
Meeting the moment The global conversation on AI has begun. In the wake of Jacob Coxon’s resignation from Anthropic, calls by AI CEOs and over a thousand employees for a potential industry slowdown, and a series of rogue AI incidents that sparked worldwide alarm, we’ve seen a tidal shift in the mainstream discourse. We’ve also seen the first legislative proposal that stands a real chance at dramatically reducing the threat of extinction from AI.
If Anyone Builds It, Everyone Dies: One Year Closer
In celebration of still being alive and fighting, we are giving away 1,000 Amazon e-books of “If Anyone Builds It, Everyone Dies”. Feel free to send a copy to yourself, a loved one, or a friend—we need all hands on deck. Today marks exactly one year sinceIf Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All, by Eliezer Yudkowsky and Nate Soares, hit bookshelves as an instant bestseller.
The AI Doc : Your Questions Answered
So you’ve just seen The AI Doc: Or How I Became an Apocaloptimist, and you suddenly have questions, lots of them. The 104-minute documentary (currently in theaters) takes viewers on a fast-paced tour through the many dimensions of the AI problem, featuring interviews from a wide range of experts. The documentary is a great place to start, in our opinion. But if you were left wanting more context and explanations, this guide is here to help.
MIRI Newsletter #125
On Thursday, March 26th, a major new AI documentary is coming out: The AI Doc: Or How I Became an Apocaloptimist. Tickets are on sale now. The movie is excellent, and we generally believe it belongs in the same tier as If Anyone Builds It, Everyone Dies—an incredibly valuable resource for alerting policymakers and the general public about AI risk, especially if it smashes the box office.
The Problem - LessWrong
This is a new introduction to AI as an extinction threat, previously posted to the MIRI website in February alongside a summary. It was written independently of Eliezer and Nate's forthcoming book, If Anyone Builds It, Everyone Dies, and isn't a sneak peak of the book. Since the book is long and costs money, we expect this to be a valuable resource in its own right even after the book comes out next month.
MIRI Newsletter #123 — LessWrong
This is a linkpost for https://intelligence.org/2025/07/03/miri-newsletter-123/
Yudkowsky and Soares Announce Major New Book: "If Anyone Builds It, Everyone Dies"
A huge announcement today: Eliezer Yudkowsky and Nate Soares have written a book attempting to raise the alarm about superintelligent AI for the widest possible audience — If Anyone Builds It, Everyone Dies. The book comes out this September 16, and preorders start today. This is Nate and Eliezer’s first mass-market book, and advance copies have been getting an extremely positive reception.
MIRI’s 2024 End-of-Year Update — LessWrong
MIRI is a nonprofit research organization with a mission of addressing the most serious hazards posed by smarter-than-human artificial intelligence. In our general strategy update and communications strategy update earlier this year, we announced a new strategy that we’re executing on at MIRI, and several new teams we’re spinning up as a result. This post serves as a status update on where things stand at MIRI today.
Sequences - LessWrong
A sequence is a series of multiple posts on Less Wrong on the same topic, to coherently and fully explore a particular thesis. See the Library page for a list of LessWrong sequences in their modern form. The original sequences were written by Eliezer Yudkowsky with the goal of creating a book on rationality. MIRI has since collated and editedthe sequences intoRationality: From AI to Zombies. If you are new to Less Wrong, this book is the best place to start.
Response to Aschenbrenner's "Situational Awareness" — LessWrong
(Cross-posted from Twitter.) My take on Leopold Aschenbrenner's new report: I think Leopold gets it right on a bunch of important counts. Three that I especially care about: Full AGI and ASI soon. (I think his arguments for this have a lot of holes, but he gets the basic point that superintelligence looks 5 or 15 years off rather than 50+.) This technology is an overwhelmingly huge deal, and if we play our cards wrong we're all dead.
Ilya Sutskever and Jan Leike resign from OpenAI [updated] — LessWrong
Ilya Sutskever and Jan Leike have resigned. They led OpenAI's alignment work. Superalignment will now be led by John Schulman, it seems. Jakub Pachocki replaced Sutskever as Chief Scientist. Reasons are unclear (as usual when safety people leave OpenAI). The NYT piece (archive) and others I've seen don't really have details. OpenAI announced Sutskever's departure in a blogpost. Sutskever and Leike confirmed their departures in tweets. Updates: Friday May 17: Superalignment dissolves.
When is a mind me? — LessWrong
xlr8harder writes: In general I don’t think an uploaded mind is you, but rather a copy. But one thought experiment makes me question this. A Ship of Theseus concept where individual neurons are replaced one at a time with a nanotechnological functional equivalent. Are you still you? Presumably the question xlr8harder cares about here isn't semantic question of how linguistic communities use the word "you", or predictions about how whole-brain emulation tech might change the way we use pronouns.
MIRI 2024 Mission and Strategy Update — LessWrong
As we announced back in October, I have taken on the senior leadership role at MIRI as its CEO. It’s a big pair of shoes to fill, and an awesome responsibility that I’m honored to take on. There have been several changes at MIRI since our 2020 strategic update, so let’s get into it.
"AI Alignment" is a Dangerously Overloaded Term — LessWrong
Alignment as Aimability or as Goalcraft? The Less Wrong and AI risk communities have obviously had a huge role in mainstreaming the concept of risks from artificial intelligence, but we have a serious terminology problem. The term "AI Alignment" has become popular, but people cannot agree whether it means something like making "Good" AI or whether it means something like making "Aimable" AI.
AI Views Snapshots — LessWrong
is pretty much defined by how the question is interpreted. It could swing pretty wildly, but the obvious interpretation seems ~tautologically bad. So there's an argument here, one I don't subscribe to, but I have seen prominent AI experts make it implicitly. If you think about it, if you have children, and they have children, and so in a series of mortal generations, with each n+1 generation more and more of your genetic distinctiveness is being lost. Language and culture will evolve as well.
AI as a science, and three obstacles to alignment strategies — LessWrong
AI used to be a science. In the old days (back when AI didn't work very well), people were attempting to develop a working theory of cognition. Those scientists didn’t succeed, and those days are behind us. For most people working in AI today and dividing up their work hours between tasks, gone is the ambition to understand minds.
Evaluating the historical value misspecification argument — LessWrong
ETA: I'm not saying that MIRI thought AIs wouldn't understand human values. If there's only one thing you take away from this post, please don't take away that. Here is Linch's attempted summary of this post, which I largely agree with.
EA Vegan Advocacy is not truthseeking, and it’s everyone’s problem — LessWrong
Introduction Effective altruism prides itself on truthseeking. That pride is justified in the sense that EA is better at truthseeking than most members of its reference category, and unjustified in that it is far from meeting its own standards.
Ideological Turing Tests
The Ideological Turing Test is an exercise where you try to pretend to hold an opposing ideology convincingly enough that outside observers can't reliably distinguish you from a true believer. It was first described by economist Bryan Caplan: Put me and five random liberal social science Ph.D.s in a chat room. Let liberal readers ask questions for an hour, then vote on who isn't really a liberal. Then put [economist Paul] Krugman and five random libertarian social science Ph.D.s in a chat room.
The basic reasons I expect AGI ruin
I’ve been citing AGI Ruin: A List of Lethalities to explain why the situation with AI looks lethally dangerous to me. But that post is relatively long, and emphasizes specific open technical problems over “the basics”. Here are 10 things I’d focus on if I were giving “the basics” on why I’m so worried:[1] 1.
The basic reasons I expect AGI ruin
Info about AGI propagates too slowly through the field, because when one ML person updates, they usually don't loudly share their update with all their peers. This is because: 1. AGI sounds weird, and they don't want to sound like a weird outsider. 2. Their peers and the community as a whole might perceive this information as an attack on the field, an attempt to lower its status, etc. 3.
Yudkowsky on AGI risk on the Bankless podcast
Eliezer gave a very frank overview of his take on AI two weeks ago on the cryptocurrency show Bankless: I’ve posted a transcript of the show and a follow-up Q&A below. Thanks to Andrea_Miotti, remember, and vonk for help posting transcripts. Intro Eliezer Yudkowsky: [clip] I think that we are hearing the last winds start to blow, the fabric of reality start to fray.
"Rationalist Discourse" Is Like "Physicist Motors" — LessWrong
Imagine being a student of physics, and coming across a blog post proposing a list of guidelines for "physicist motors"—motor designs informed by the knowledge of physicists, unlike ordinary motors. Even if most of the things on the list seemed like sensible advice to keep in mind when designing a motor, the framing would seem very odd. The laws of physics describe how energy can be converted into work. To the extent that any motor accomplishes anything, it happens within the laws of physics.
Hashing out long-standing disagreements seems low-value to me — LessWrong
(Status: a short write-up of some things that I find myself regularly saying in person. In this case, I'm writing up my response to the question of why I don't spend a bunch more time trying to resolve disagreements with people in the community who disagree with me about the hopefulness of whatever research direction.
Elements of Rationalist Discourse — LessWrong
How about "hurting a person or deminishing their credibility, or the credibility of their argument, without using a rational argument"? "Hurting a person" still seems too vague to me (sometimes people are "hurt" just because you disagreed with them on a claim of fact), "Diminishing...
Fucking Goddamn Basics of Rationalist Discourse — LessWrong
It feels like CYA if you don't care about truth. Flagging I also disagree with this (also seems to obviously be failing rule #10). I'm a bit confused about this, because, like, I'm sure you know that time is short, there are lots of (true) things to talk about, going infinitely deep on precisely specifying any given thing is clearly unworkable even if you pick the specific sub-hill of "be LessWrong" to die on rather than the broader hill of "maximize truthseeking.".
We don’t trade with ants — LessWrong
We actually do not know they are 'doing just fine'. Many insect species have gone extinct already (speaking of 'existential threats to them'...), and insect populations in general appear to be in substantial decline. It's highly debated because the data is in general so bad compared to bigger stuff like mammals. Anyway: Bees have also been seriously affected, with only half of the bumblebee species found in Oklahoma in the US in 1949 being present in 2013.
A challenge for AGI organizations, and a challenge for readers — LessWrong
Epistemic status: 50% sophistry, but I still think it's insightful since specifically aligning LLMs needs to be discussed here more. I find it quite interesting that much of current large language model (LLM) alignment is just stating, in plain text, "be a helpful, aligned AI, pretty please".
Decision theory does not imply that we get to have nice things — LessWrong
(Note: I wrote this with editing help from Rob and Eliezer. Eliezer's responsible for a few of the paragraphs.) A common confusion I see in the tiny fragment of the world that knows about logical decision theory (FDT/UDT/etc.), is that people think LDT agents are genial and friendly for each other. One recent example is Will Eden’s tweet about how maybe a molecular paperclip/squiggle maximizer would leave humanity a few stars/galaxies/whatever on game-theoretic grounds.
Instrumental Convergence
Instrumental convergenceor convergent instrumental values is the theorized tendency for most sufficiently intelligent agents to pursue potentially unbounded instrumental goals such as self-preservation and resource acquisition [1]. This concept has also been discussed under the term basic drives. The idea was first explored by Steve Omohundro. He argued that sufficiently advanced AI systems would all naturally discover similar instrumental subgoals.
Feeling Moral - LessWrong
Suppose that a disease, or a monster, or a war, or something, is killing people. And suppose you only have enough resources to implement one of the following two options: Save 400 lives, with certainty. Save 500 lives, with 90\% probability; save no lives, 10\% probability. Most people choose option 1. Which, I think, is foolish; because if you multiply 500 lives by 90% probability, you get an expected value of 450 lives, which exceeds the 400-life value of option 1.
July 2022 Newsletter
July 30, 2022 | Rob Bensinger | Newsletters MIRI has put out three major new posts: AGI Ruin: A List of Lethalities. Eliezer Yudkowsky lists reasons AGI appears likely to cause an existential catastrophe, and reasons why he thinks the current research community—MIRI included—isn't succeeding at preventing this from happening A central AI alignment problem: capabilities generalization, and the sharp left turn.
What should you change in response to an "emergency"? And AI risk — LessWrong
I appreciate this comment a lot. Thank you. I appreciate that it’s sharing an inside view, and your actual best guess, despite these things being the sort of thing that might get social push-back! My own take is that people depleting their long-term resources and capacities is rarely optimal in the present context around AI safety. My attempt to share my reasoning is pretty long, sorry; I tried to use bolding to make it skimmable.
ITT-passing and civility are good; "charity" is bad; steelmanning is niche — LessWrong
This post has been recorded as part of the LessWrong Curated Podcast, and can be listened to on Spotify, Apple Podcasts, and Libsyn. I often object to claims like "charity/steelmanning is an argumentative virtue". This post collects a few things I and others have said on this topic over the last few years.
LessWrong Has Agree/Disagree Voting On All New Comment Threads — LessWrong
Hmm didn't really find anything similar, but here are some examples of rating systems I found that looked interesting (though not necessarily relevant): SaidIt: (1) Insightful & (2) Fun SaidIt is a Reddit alternative which seeks to "create an environment that encourages thought-provoking discussion". SaidIt has two types of upvotes to choose from: 1) insightful, and 2) fun.
A central AI alignment problem: capabilities generalization, and the sharp left turn — LessWrong
(This post was factored out of a larger post that I (Nate Soares) wrote, with help from Rob Bensinger, who also rearranged some pieces and added some text to smooth things out. I'm not terribly happy with it, but am posting it anyway (or, well, having Rob post it on my behalf while I travel) on the theory that it's better than nothing.) I expect navigating the acute risk period to be tricky for our civilization, for a number of reasons.
why assume AGIs will optimize for fixed goals? — LessWrong
This question gets at a bundle of assumptions in a lot of alignment thinking that seem very wrong to me. I'd add another, subtler, assumption that I think is also wrong: namely, that goals and values are discrete. E.g., when people talk of mesa optimizers, they often make reference to a mesa objective which the (single) mesa optimizer pursues at all times, regardless of the external situation.
Salvage Epistemology — LessWrong
A funny thing happens with woo sometimes, in the rationality community. There's a frame that says: this is a mix of figurative stuff and dumb stuff, let's try to figure out what the figurative stuff is pointing at and salvage it. Let's call this "salvage epistemology". Unambiguous examples include the rationality community's engagement with religions, cold-reading professions like psychics, bodywork, and chaos magic.
Christiano and Yudkowsky on AI predictions and human intelligence #AI
In previous MCTS RL algorithms, the environment model is either given or only trained with rewards, values, and policies, which cannot provide sufficient training signals due to their scalar nature. The problem is more severe when the reward is sparse or the bootstrapped value is not accurate. The MCTS policy improvement operator heavily relies on the environment model. Thus, it is vital to have an accurate one.
Shah and Yudkowsky on alignment failures #AI
If you don’t know how to implement CIRL in such a way that it actually aims at goodness, then you don’t have an algorithm with properties a, b and c above. Or, to put it another way: suppose I replace the word “goodness” with “winningness”. Now I can describe AlphaStar as follows: Now, you might say that the difference is that CIRL implements uncertainty over possible utility functions, not possible empirical beliefs.
Ngo and Yudkowsky on scientific reasoning and pivotal acts #AI
#A.I. This is a transcript of a conversation between Richard Ngo and Eliezer Yudkowsky, facilitated by Nate Soares (and with some comments from Carl Shulman). This transcript continues the Late 2021 MIRI Conversations sequence, following Ngo’s view on alignment difficulty. Color key: Chat by Richard and Eliezer Other chat 14. October 4 conversation 14.1. Predictable updates, threshold functions, and the human cognitive range [Ngo][15:05] Two questions which I’d like to ask Eliezer: 1.
February 2022 Newsletter #AI
#A.I. As of yesterday, we've released the final posts in the Late 2021 MIRI Conversations sequence, a collection of (relatively raw and unedited) AI strategy conversations: Ngo's view on alignment difficulty Ngo and Yudkowsky on scientific reasoning and pivotal acts Christiano and Yudkowsky on AI predictions and human intelligence Shah and Yudkowsky on alignment failures Eliezer Yudkowsky, Nate Soares, Paul Christiano, Richard Ngo, and Rohin Shah (and possibly other participants in the...
Shah and Yudkowsky on alignment failures
[Yudkowsky][13:22] So I have a general thesis about a failure mode here which is that, the moment you try to sketch any concrete plan or events which correspond to the abstract descriptions, it is much more obviously wrong, and that is why the descriptions stay so abstract in the mouths of everybody who sounds more optimistic than I am.
Ngo and Yudkowsky on scientific reasoning and pivotal acts
[Yudkowsky][17:23] It should not be overlooked that there’s a very valid sibling of the old complaint “Anything you can do ceases to be AI”, which is that “Things you can do with surprisingly-to-your-model shallow cognition are precisely the things that Reality surprises you by telling you that AI can do earlier than you expected.” When we see GPT-3, we were getting some amount of real evidence about AI capabilities advancing faster than I expected, and some amount of evidence about GPT-3’s...
February 2022 Newsletter
March 1, 2022 | Rob Bensinger | Newsletters As of yesterday, we've released the final posts in the Late 2021 MIRI Conversations sequence, a collection of (relatively raw and unedited) AI strategy conversations: Ngo's view on alignment difficulty Ngo and Yudkowsky on scientific reasoning and pivotal acts Christiano and Yudkowsky on AI predictions and human intelligence Shah and Yudkowsky on alignment failures Eliezer Yudkowsky, Nate Soares, Paul Christiano, Richard Ngo, and Rohin Shah (and...
January 2022 Newsletter #AI
The post January 2022 Newsletter appeared first on Machine Intelligence Research Institute.
The Rationalists of the 1950s (and before) also called themselves “Rationalists” — LessWrong
TLDR There’s an organization based in London called the Rationalist Association. It was founded in 1885. Historically, it focused on publishing books and articles related to atheism and science, including works by Darwin, Bertrand Russell, J. B. S. Haldane, George Bernard Shaw, H. G. Wells, and Karl Popper. The topics covered overlap with the present-day rationalist movement (centered on Lesswrong).
Object level and Meta level
Object level vs Meta level is a distinction between levels of abstraction, with the object level being directly about the specifics of the issue at hand, while meta level is concerned with the general principles which apply to a wider range of examples.
Discussion with Eliezer Yudkowsky on AGI interventions
The following is a partially redacted and lightly edited transcript of a chat conversation about AGI between Eliezer Yudkowsky and a set of invitees in early September 2021. By default, all other participants are anonymized as “Anonymous”.
Show More
loading
Actions
Is this you?
As a journalist, you can create a free Muck Rack account to customize your profile, list your contact preferences, and upload a portfolio of your best work.Get in touch with Rob
Contact Rob, search articles and posts on X, monitor coverage, and track replies from one place.
Learn more about Muck Rack