Is this you? As a journalist, you can create a free Muck Rack account to customize your profile, list your contact preferences, and upload a portfolio of your best work.
Claim your profile
Articles by Seth Herd
AI: artificial immigrants — LessWrong
This is a linkpost for https://worldspiritsockpuppet.substack.com/p/ai-artificial-immigrants
Astra and Fable still hack on simple variants of alignment evals from 2025
In February 2025, back when o3-mini was the strongest available LLM, Palisade Research publicized a now well-known alignment eval where they asked models to play a game of chess against a chess engine. They found that the new, RLVR'd models cheated on the task by altering the board state about 36% of the time. The experiment received a reasonable amount of circulation, and there were even rumors of skepticism from some lab engineers until they could run it themselves.
LessWrong vs. TikTok: Tips for capturing attention in a non-rational space
Here's what I've learned from making TikTok videos every day for the last 60 days. I think posting on TikTok is worthwile whether you want to spread AI safety arguments to an enormous audience, you have something you're passionate about you want to share, or you just want to work on your ability to speak confidently and capture human attention. There's a list of tips at the end of this post, but first I want to talk about TikTok at a high level.
The Machines Lack Honour — LessWrong
The battle lines of the AI morality debate are being laid down. On one side you have the ChatGPT dogma: AI as mere tools with no real preferences or even beliefs. On the other you have the twitter AI whisperers: AIs as complex beings with rich personalities and desires which deserve our respect. And in the middle you have the official Anthropic line, that they are genuinely uncertain, as is Claude, but they’re going to try to look into its welfare and explain to it how to be a good person.
Contra Wentworth on Physical Attractiveness for Men — LessWrong
This is a response to John Wentworth’s recent article, Why Physical Attractiveness Matters for Men’s Dating Prospects. I have no quibble with the thesis stated in the title, but a lot of the body of the article struck me as off-base. When John sent me the article, I told him the article seemed “fundamentally confused.” He asked for details, and this article is my answer. Here I list and argue with four premises I detected in John’s article.
Empowerment, corrigibility, etc. are simple abstractions (of a messed-up ontology) — LessWrong
1.1 Tl;dr Alignment is often conceptualized as AIs helping humans achieve their goals: AIs that increase people’s agency and empowerment; AIs that are helpful, corrigible, and/or obedient; AIs that avoid manipulating people.
The effects of caffeine consumption do not decay with a ~5 hour half-life — LessWrong
epistemic status: confident in the overall picture, substantial quantitative uncertainty about the relative potency of caffeine and paraxanthine tldr: The effects of caffeine consumption last longer than many assume. Paraxanthine is sort of like caffeine that behaves the way many mistakenly believe caffeine behaves. You've probably heard that caffeine exerts its psychostimulatory effects by blocking adenosine receptors. That matches my understanding, having dug into this.
Sadly, The Whispering Earring — LessWrong
The earring doesn't just model the reward function. >It is not a taskmaster, telling you what to do in order to achieve some foreign goal. It always tells you what will make you happiest. If it would make you happiest to succeed at your work, it will tell you how best to complete it. If it would make you happiest to do a half-assed job at your work and then go home and spend the rest of the day in bed having vague sexual fantasies, the earring will tell you to do that. The earring is never wrong.
Personality Self-Replicators — LessWrong
I describe the risk of personality self-replicators, the threat of OpenClaw-like agents managing spreading in hard-to-control ways. LLM agents like OpenClaw are defined by a small set of text files and are run by an open source framework which leverages LLMs for cognition.
Aligning to Virtues — LessWrong
Which alignment target? Suppose you’re an AI company or government, and you want to figure out what values to align your AI to. Here are three options, and some of their downsides: AIs that are aligned to a set of consequentialist values are incentivized to acquire power to pursue those values. This creates power struggles between those AIs and: Humans who don’t share those values. Humans who disagree with the AI about how to pursue those values.
Alignment will happen by default. What’s next? — LessWrong
Firstly, I think there is a decent chance we get alignment by default (where default means hundreds of very smart people work day and night on prosaic techniques). I'd put it at ~40%? Maybe 55% Anthropics AGI would be aligned? I wrote about semi-mechanistic stories for how I think current techniques could lead to genuinely aligned AI. Or CEV-style alignment and corrigibility that's stable. I think your picture is a little too rosy though.
A country of alien idiots in a datacenter: AI progress and public alarm — LessWrong
Epistemic status: I'm pretty sure AI will alarm the public enough to change the alignment challenge substantially. I offer my mainline scenario as an intuition pump, but I expect it to be wrong in many ways, some important. Abstract arguments are in the Race Conditions and concluding sections. Nora has a friend in her phone.
Don't Mock Yourself — LessWrong
About half a year ago, I decided to try stop insulting myself for two weeks. No more self-deprecating humour, calling myself a fool, or thinking I'm pathetic. Why? Because it felt vaguely corrosive. Let me tell you how it went. Spoiler: it went well. The first thing I noticed was how often I caught myself about to insult myself. It happened like multiple times an hour. I would lay in bed at night thinking, "you mor- wait, I can't insult myself, I've still got 11 days to go.
The Culture Novels as a Dystopia — LessWrong
I'd mentioned the Golden Age novels by Wright before when we'd gone hiking together, so I thought it'd be worth looking at some related flaws in his utopia. The Sophotechs (the trilogy's equivalent of the minds), are philosophically libertarian. While they do intervene to stop direct violence, they otherwise enforce a system of complete freedom over the self, as well as the maintenance of property rights.
Is there actually a reason to use the term AGI/ASI anymore? — LessWrong
When we have a particular query at hand, we don't need the broad terms, and that's great. But we do need shorthands when defining a particular query doesn't fit the time/space/attention span. Having a wider variety of sub-terms within the community would be extremely helpful, but I'm not sure how to work toward that. I think about this a fair amount. I think terminology matters, and having terms of art is practically necessary.
Problems with Instruction Following as an Alignment Target — LessWrong
We should probably try to understand the failure modes of the alignment schemes that AGI developers are most likely to attempt. I still think Instruction-following AGI is easier and more likely than value aligned AGI. I’ve updated downward on the ease of IF alignment, but upward on how likely it is. IF is the de-facto current primary alignment target (see definition immediately below), and it seems likely to remain so until the first real AGIs, if we continue on the current path (e.g., AI 2027).
LLM AGI will have memory, and memory changes alignment — LessWrong
When stateless LLMs are given memories they will accumulate new beliefs and behaviors, and that may allow their effective alignment to evolve. (Here "memory" is learning during deployment that is persistent beyond a single session.) LLM agents will have memory: Humans who can't learn new things ("dense anterograde amnesia") are not highly employable for knowledge work. LLM agents that can learn during deployment seem poised to have a large economic advantage.
Whether governments will control AGI is important and neglected — LessWrong
Epistemic status: somewhat rushed out in advance of the deadline tomorrow (Saturday the 15th) for the call for public comment on US AI policy. I think the issue is complex and deserves careful consideration.
Will LLM agents become the first takeover-capable AGIs? — LessWrong
One of my takeaways from EA Global this year was that most alignment people aren't focused on LLM-based agents (LMAs) as a route to takeover-capable AGI. I want to better understand this position, since I estimate this path to AGI as likely enough (maybe around 60%) to be worth specific focus and concern.
OpenAI releases ChatGPT 4.5 — LessWrong
This is not o3; it is what they'd internally called Orion, a larger non-reasoning model. They say this is their last fully non-reasoning model, but that research on both types will continue. It's currently limited to Pro users, but the model hasn't yet shown up on the chooser, despite them announcing it. They say it will be shared with Plus users next week. It claims to be more accurate at standard questions and with a lower hallucination rate than any previous OAI model (and presumably any others).
System 2 Alignment — LessWrong
Submitted to Alignment Forum. Contains more technical jargon than usual. Epistemic status: I'm pretty sure these are important questions, but I don't have firm conclusions. I'm hoping to inspire others to think in this direction. ~50% of the average value is probably in the Summary; another ~15% each in the next Overview and OpenAI's Deliberative Alignment sections. The second half is more detailed, and intended for those who want to think about aligning language model agents as AGI.
Rational Utopia & Multiversal AI Alignment: Steerable ASI for Ultimate Human Freedom — LessWrong
(This is the result of 3 years of thinking and modeling hyper-futuristic and current ethical systems. Everything described here can be modeled mathematically; it's essentially geometry. I take as an axiom that every agent in the multiverse experiences real pain and pleasure. Sorry for the rough edges—I'm a newcomer, and my ideas might sound strange, so please steelman them and share your thoughts. I don’t want to create dystopias—I personally wouldn’t want to live in one. This is why I write.
Rational Utopia & Narrow Way There: Place ASI, Dealing With Dystopias, New Ethics... (V. 4) — LessWrong
(This is the result of three years of thinking and modeling hyper‑futuristic and current ethical systems. The first post in the series. Everything described here can be modeled mathematically—it’s essentially geometry. I take as an axiom that every agent in the multiverse experiences real pain and pleasure. Sorry for the rough edges—I’m a newcomer, non‑native speaker, and my ideas might sound strange, so please steelman them and share your thoughts.
Seven sources of goals in LLM agents — LessWrong
LLM agents seem reasonably likely to become our first takeover-capable AGIs. LLMs already have complex "psychologies," and using them to power more sophisticated agents will create even more complex "minds." A profusion of competing goals is one barrier to aligning this type of AGI.
OpenAI releases deep research agent — LessWrong
Today we’re launching deep research in ChatGPT, a new agentic capability that conducts multi-step research on the internet for complex tasks. It accomplishes in tens of minutes what would take a human many hours. Deep research is OpenAI's next agent that can do work for you independently—you give it a prompt, and ChatGPT will find, analyze, and synthesize hundreds of online sources to create a comprehensive report at the level of a research analyst.
Yudkowsky on The Trajectory podcast — LessWrong
I found this interview notable as the most useful public statement yet of Yudkowsky's views. I congratulate both him and the host, Dan Fagella, for strategically improving how they're communicating their ideas. The host, Dan Fagella is to be commended for asking the right questions and taking the right tone to get an extraordinarily concise statement of Yudkowsky's views on what we might do to survive and why.
Model Integrity: MAI on Value Alignment — LessWrong
EVERYONE, CALM DOWN! Meaning Alignment Institute just dropped their first post in basically a year and it seems like they've been up to some cool stuff. Their perspective on value alignment really grabbed my attention because it reframes our usual technical alignment conversations around rules and reward functions into something more fundamental - what makes humans actually reliably good and cooperative?
Gratitudes: Rationalist Thanks Giving — LessWrong
Epistemic status: I wish I'd thought of writing this before the day rolled around. Brief and unpolished, although this is something I've thought about a lot on both personal and computational neuroscience levels. But there are no strong conclusions, just some brief thoughts you may find interesting or even useful. Hopefully you've had a fun Thanksgiving celebration, including feasting and appreciating family and friends.
OpenAI Email Archives (from Musk v. Altman) — LessWrong
As part of the court case between Elon Musk and Sam Altman, a substantial number of emails between Elon, Sam Altman, Ilya Sutskever, and Greg Brockman have been released. I have found reading through these really valuable, and I haven't found an online source that compiles all of them in an easy to read format. So I made one. I used some AI assistance to generate this originally, and then went meticulously through each email and checked them for differences.
Current Attitudes Toward AI Provide Little Data Relevant to Attitudes Toward AGI — LessWrong
Epistemic status: Sudden public attitude shift seems quite possible, but I haven't seen it much in discussion, so I thought I'd float the idea again. This is somewhat dashed off since the goal is just to toss out a few possibilities and questions. In Current AIs Provide Nearly No Data Relevant to AGI Alignment, Thane Ruthenis argues that current AI is almost irrelevant to the project of aligning AGIs. Current AI is simply not what we're talking about when we worry about alignment, he says.
Linkpost: "Imagining and building wise machines: The centrality of AI metacognition" by Johnson, Karimi, Bengio, et al. — LessWrong
I read the whole thing because of its similarity to my proposals about metacognition as an aid to both capabilities and alignment in language model agents. In this and my work, metacognition is a way to keep AI from doing the wrong thing (from the AIs perspective). They explicitly do not address the broader alignment problem of AIs wanting the wrong things (from humans' perspective). They note that "wiser" humans are more prone to serve the common good, by taking more perspectives into account.
Intent alignment as a stepping-stone to value alignment — LessWrong
I think Instruction-following AGI is easier and more likely than value aligned AGI, and that this accounts for one major crux of disagreement on alignment difficulty. I got several responses to that piece that didn't dispute that intent alignment is easier, but argued we shouldn't give up on value alignment. I think that's right. Here's another way to frame the value of personal intent alignment: we can use a superintelligent instruction-following AGI to solve full value alignment.
What TMS is like — LessWrong
There are two nuclear options for treating depression: Ketamine and TMS; This post is about the latter. TMS stands for . Basically, it fixes depression via magnets, which is about the second or third most magical things that magnets can do. I don’t know a whole lot about the neuroscience - this post isn’t about the how or the why. It’s from the perspective of a patient, and it’s about the what. What is it like to get TMS?
Could randomly choosing people to serve as representatives lead to better government? — LessWrong
I'm an advocate of something known as sortition. The premise is simple. Choose people at random, to serve a finite term, in some decision making capacity. Pay them to be there. Sounds ridiculous, right? How could we possibly trust ignorant, stupid, normal people to make good decisions? What would this look like? Why would this be better than electing our officials to office? Descriptive Representation Imagine a Congress that actually looks like America.
Could literally randomly choosing people to serve as our political representatives lead to better government? — LessWrong
I'm an advocate of something known as sortition. The premise is simple. Choose people at random, to serve a finite term, in some decision making capacity. Pay them to be there. Sounds ridiculous, right? How could we possibly trust ignorant, stupid, normal people to make good decisions? What would this look like? Why would this be better than electing our officials to office? Descriptive Representation Imagine a Congress that actually looks like America.
RREAL AGI - LessWrong
I see "AGI" used for everything from existing LLMs to superintelligence, and massive resulting confusion and illusory disagreements. I finally thought of a term I like for what I mean by AGI. It's an acronym that's also somewhat intuitive without reading the definition: Reasoning, Reflective Entities with Autonomy and Learning might be called "(R)REAL AGI" or "real AGI". See below for further definitions.
Universal basic income isn’t always AGI-proof — LessWrong
A universal basic income (UBI) is often presented as a public insurance against large-scale and potentially permanent technological unemployment. Many Silicon Valley leaders that believe in the transformative economic potential of artificial general intelligence have also voiced their support for UBI (e.g. Sam Altman, Elon Musk). UBI can be a part of the policy tools to address long-term technological unemployment.
Conflating value alignment and intent alignment is causing confusion — LessWrong
Epistemic status: I think something like this confusion is happening often. I'm not saying these are the only differences in what people mean by "AGI alignment". Value alignment is better but probably harder to achieve than personal intent alignment to the short-term wants of some person(s). Different groups and people tend to primarily address one of these alignment targets when they discuss alignment. Confusion abounds.
Broad Strengths and Vulnerabilities of AI Safety Plans — LessWrong
By: @Seth Herd Seth Herd, Kabir Kumar, FractalCycle, Tori, TheAntithesis, AIPanic, @momom2 and others Made in the first 2 days of the ai-plans.com August 2023 critique-a-thon Join the current one here: https://ai-plans.com/events This is not a definitive, exhaustive list. It can and has been used as a starting point for more refined ideas. See https://docs.google.com/document/d/1mW4SAxFN_aI6KyYXpl9qz5B9nVdeV9Xyc69GTNme5cA/edit?usp=drivesdk for examples.
If we solve alignment, do we die anyway? — LessWrong
I'm aware of good arguments that this scenario isn't inevitable, but it still seems frighteningly likely even if we solve technical alignment. TL;DR: If we solve alignment, it will probably be used to create AGI that follows human orders. If takeoff is slow-ish, a pivotal act (preventing more AGIs from being developed) will be difficult. If no pivotal act is performed, RSI-capable AGI proliferates. This creates an n-way non-iterated Prisoner's Dilemma where the first to attack, wins.
Humanity isn't remotely longtermist, so arguments for AGI x-risk should focus on the near term — LessWrong
Toby Ord recently published a nice piece On the Value of Advancing Progressabout mathematical projections of far-future outcomes given different rates of progress and risk levels. The problem with that and many arguments for caution is that people usually barely care about possibilities even twenty years out. We could talk about sharp discounting curves in decision-making studies, and how that makes sense given evolutionary pressures in tribal environments.
Fear of centralized power vs. fear of misaligned AGI: Vitalik Buterin on 80,000 Hours — LessWrong
Vitalik Buterin wrote an impactful blog post, My techno-optimism. I found this discussion of one aspect on 80,00 hours much more interesting. The remainder of that interview is nicely covered in the host's EA Forum post. My techno optimism apparently appealed to both sides, e/acc and doomers. Buterin's approach to bridging that polarization was interesting. I hadn't understood before the extent to which anti-AI regulation sentiment is driven by fear of centralized power.
.CLI's Shortform — LessWrong
This is a special post for quick takes by. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page
What's a better term now that "AGI" is too vague? — LessWrong
The term "AGI" is now creating confusion. When it's used in the context of timelines or alignment, we don't know if it means near-future LLMs or superintelligence. It's fair to use AGI as "fairly general AI at near-human level", which includes current LLMs. But we should have a distinguishing term for the stronger use of AGI, because the implications for change and alignment are very different.
Anthropic announces interpretability advances. How much does this advance alignment? — LessWrong
Anthropic just published a pretty impressive set of results in interpretability. This raises for me, some questions and a concern: Interpretability helps, but it isn't alignment, right? It seems to me as though the vast bulk of alignment funding is now going to interpretability. Who is thinking about how to leverage interpretability into alignment? It intuitively seems as though we are better off the more we understand the cognition of foundation models.
Instruction-following AGI is easier and more likely than value aligned AGI — LessWrong
Summary: We think a lot about aligning AGI with human values. I think it’s more likely that we’ll try to make the first AGIs do something else. This might intuitively be described as trying to make instruction-following (IF) or do-what-I-mean-and-check (DWIMAC) be the central goal of the AGI we design. Adopting this goal target seems to improve the odds of success of any technical alignment approach.
Selfmaker662's Shortform — LessWrong
This is a special post for quick takes by. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page
Has Generative AI Already Peaked? - Computerphile — LessWrong
It's helpful to include a summary with linkposts. So here's a super quick one. I didn't listen to it closely, so I could've missed something.
The Rationale-Shaped Hole At The Heart Of Forecasting — EA Forum
Thanks to Eli Lifland, Molly Hickman, Değer Turan, and Evan Miyazono for reviewing drafts of this post. The opinions expressed here are my own.
Show More
loading
Actions
Is this you?
As a journalist, you can create a free Muck Rack account to customize your profile, list your contact preferences, and upload a portfolio of your best work.Get in touch with Seth
Contact Seth, search articles and posts on X, monitor coverage, and track replies from one place.
Learn more about Muck Rack