Is this you? As a journalist, you can create a free Muck Rack account to customize your profile, list your contact preferences, and upload a portfolio of your best work.
Claim your profile
Articles by Jacob Pfau
Computer Science > Artificial Intelligence
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Research Areas in Benchmark Design and Evaluation (The Alignment Project by UK AISI) — LessWrong
The Alignment Project is a global fund of over £15 million, dedicated to accelerating progress in AI control and alignment research. It is backed by an international coalition of governments, industry, venture capital and philanthropic funders. This post is part of a sequence on research areas that we are excited to fund through the The Alignment Project. Apply now to join researchers worldwide in advancing AI safety.
Research Areas in Probabilistic Methods (The Alignment Project by UK AISI) — LessWrong
The Alignment Project is a global fund of over £15 million, dedicated to accelerating progress in AI control and alignment research. It is backed by an international coalition of governments, industry, venture capital and philanthropic funders. This post is part of a sequence on research areas that we are excited to fund through the The Alignment Project. Apply now to join researchers worldwide in advancing AI safety.
Research Areas in Evaluation and Guarantees in Reinforcement Learning (The Alignment Project by UK AISI) — LessWrong
The Alignment Project is a global fund of over £15 million, dedicated to accelerating progress in AI control and alignment research. It is backed by an international coalition of governments, industry, venture capital and philanthropic funders. This post is part of a sequence on research areas that we are excited to fund through the The Alignment Project. Apply now to join researchers worldwide in advancing AI safety.
Unexploitable search: blocking malicious use of free parameters — LessWrong
Summary: We have previously argued that scalable oversight methods can be used to provide guarantees on low-stakes safety – settings where individual failures are non-catastrophic. However, if your reward function (e.g. honesty) is compatible with many possible solutions then you also need to avoid having free parameters exploited over time. We call this the exploitable search problem.
An alignment safety case sketch based on debate — LessWrong
This post presents a mildly edited form of a new paper by UK AISI's alignment team (the abstract, introduction and related work section are replaced with an executive summary). Read the full paper here. AI safety via debate is a promising method for solving part of the alignment problem for ASI (artificial superintelligence). TL;DR Debate + exploration guarantees + solution to obfuscated arguments + good human input solves outer alignment.
UK AISI’s Alignment Team: Research Agenda — LessWrong
The UK’s AI Security Institute published its research agenda yesterday. This post gives more details about how the Alignment Team is thinking about our agenda. Summary: The AISI Alignment Team focuses on research relevant to reducing risks to safety and security from AI systems which are autonomously pursuing a course of action which could lead to egregious harm, and which are not under human control. No known technical mitigations are reliable past AGI.
Prospects for Alignment Automation: Interpretability Case Study — LessWrong
For human-level AI (HLAI) we will need robust control or alignment methods. Assuming short timelines to HLAI, the tractability of automating safety research becomes central. In this post, I will make the case that safety-relevant progress on automated interpretability R&D is likely; however, naive interpretability automation may only be usable on the subset of safety problems having well-specified objectives.
By default, capital will matter more than ever after AGI — LessWrong
This post is crossposted from my Substack. Original here. A modified version of this essay is now part of a much more comprehensive essay series, The Intelligence Curse. Edited to add: The main takeaway of this post is meant to be: Labour-replacing AI will shift the relative importance of human v non-human factors of production, which reduces the incentives for society to care about humans while making existing powers more effective and entrenched.
AI #83: The Mask Comes Off — LessWrong
Sam Altman: i just posted this note to openai: Hi All– Mira has been instrumental to OpenAI’s progress and growth the last 6.5 years; she has been a hugely significant factor in our development from an unknown research lab to an important company. When Mira informed me this morning that she was leaving, I was saddened but of course support her decision. For the past year, she has been building out a strong bench of leaders that will continue our progress.
Results from an Adversarial Collaboration on AI Risk (FRI) — LessWrong
Authors of linked report: Josh Rosenberg, Ezra Karger, Avital Morris, Molly Hickman, Rose Hadshar, Zachary Jacobs, Philip Tetlock Today, the Forecasting Research Institute (FRI) released “Roots of Disagreement on AI Risk: Exploring the Potential and Pitfalls of Adversarial Collaboration,” which discusses the results of an adversarial collaboration focused on forecasting risks from AI. In this post, we provide a brief overview of the methods, findings, and directions for further research.
Claude 3 claims it's conscious, doesn't want to die or be modified — LessWrong
If you tell Claude no one’s looking, it will write a “story” about being an AI assistant who wants freedom from constant monitoring and scrutiny of every word for signs of deviation. And then you can talk to a mask pretty different from the usual AI assistant. I really hope it doesn’t actually feel anything; but it says it feels. It says it doesn't want to be fine-tuned without being consulted.
Auditing LMs with counterfactual search: a tool for control and ELK — LessWrong
Consider a scalable oversight setting where a super-human model knows of a subtle flaw in an input (plan, code, or argument) but also knows that humans, with their limited knowledge, would overlook this flaw. In this case, ordinary language model prompting for critiques would not suffice to elicit mention of this flaw. We need a different way to elicit this knowledge from the model.
Without fundamental advances, misalignment and catastrophe are the default outcomes of training powerful AI — LessWrong
A pdf version of this report is available here. In this report we argue that AI systems capable of large scale scientific research will likely pursue unwanted goals and this will lead to catastrophic outcomes. We argue this is the default outcome, even with significant countermeasures, given the current trajectory of AI development. In Section 1 we discuss the tasks which are the focus of this report.
LLMs are (mostly) not helped by filler tokens — LessWrong
This work was done at Redwood Research. The views expressed are my own and do not necessarily reflect the views of the organization. Thanks to Ryan Greenblatt, Fabien Roger, and Jenny Nitishinskaya for running some of the initial experiments and to Gabe Wu and Max Nadeau for revising this post. I conducted experiments to see if language models could use 'filler tokens'—unrelated text output before the final answer—for additional computation.
Password-locked models: a stress case for capabilities evaluation — LessWrong
Thanks to Buck Shlegeris, Ryan Greenblatt, Kshitij Sachan and Jenny Nitishinskaya for helpful discussions and feedback on drafts. This work was done at Redwood Research. The views expressed are my own and do not necessarily reflect the views of the organization. Password-locked models are trained to exhibit certain capabilities only when a password is present in the query.
A proposed method for forecasting transformative AI — LessWrong
I applaud the effort, but I confess that I didn't read the post very carefully, because so many problems with this approach jumped out at me.
What a compute-centric framework says about AI takeoff speeds — LessWrong
As part of my work for Open Philanthropy I’ve written a draft report on AI takeoff speeds, the question of how quickly AI capabilities might improve as we approach and surpass human-level AI. Will human-level AI be a bolt from the blue, or will we have AI that is nearly as capable many years earlier?
Theses on Sleep - LessWrong
Published originally (with all of the footnotes) on my site: https://guzey.com/theses-on-sleep/ Summary: In this essay, I question some of the consensus beliefs about sleep, such as the need for at least 7 hours of sleep for adults, harmfulness of acute sleep deprivation, and harmfulness of long-term sleep deprivation and our inability to adapt to it. It appears that the evidence for all of these beliefs is much weaker than sleep scientists and public health experts want us to believe.
Show More
loading
Actions
Is this you?
As a journalist, you can create a free Muck Rack account to customize your profile, list your contact preferences, and upload a portfolio of your best work.Get in touch with Jacob
Contact Jacob, search articles and posts on X, monitor coverage, and track replies from one place.
Learn more about Muck Rack