Is this you? As a journalist, you can create a free Muck Rack account to customize your profile, list your contact preferences, and upload a portfolio of your best work.
Claim your profile
Articles by Daniel Murfet
[2508.00331] Embryology of a Language Model
arXiv:2508.00331 (cs) View PDF Subjects: Machine Learning (cs.LG) Cite as: arXiv:2508.00331 [cs.LG] (or arXiv:2508.00331v1 [cs.LG] for this version) https://doi.org/10.48550/arXiv.2508.00331 Submission history From: Daniel Murfet [ view email] [v1] Fri, 1 Aug 2025 05:39:41 UTC (24,779 KB) Bibliographic Tools Bibliographic Explorer Toggle Bibliographic Explorer () Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle...
[2504.18048] Modes of Sequence Models and Learning Coefficients
arXiv:2504.18048 (cs) View PDF HTML (experimental) Subjects: Machine Learning (cs.LG) Cite as: arXiv:2504.18048 [cs.LG] (or arXiv:2504.18048v1 [cs.LG] for this version) https://doi.org/10.48550/arXiv.2504.18048 Submission history From: Daniel Murfet [ view email] [v1] Fri, 25 Apr 2025 03:38:10 UTC (1,629 KB) Bibliographic Tools Bibliographic Explorer Toggle Bibliographic Explorer () Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?)...
Timaeus in 2024 - LessWrong
TLDR: We made substantial progress in 2024: We published a series of papers that verify key predictions of Singular Learning Theory (SLT) [1, 2, 3, 4, 5, 6]. We scaled key SLT-derived techniques to models with billions of parameters, eliminating our main concerns around tractability. We have clarified our theory of change and diversified our research portfolio to pay off across a range of different timelines.
[2501.17745] Dynamics of Transient Structure in In-Context Linear Regression Transformers
[Submitted on 29 Jan 2025] Title:Dynamics of Transient Structure in In-Context Linear Regression Transformers View a PDF of the paper titled Dynamics of Transient Structure in In-Context Linear Regression Transformers, by Liam Carroll and 3 other authors View PDF HTML (experimental) Abstract:Modern deep neural networks display striking examples of rich internal computational structure.
Activation space interpretability may be doomed — LessWrong
TL;DR: There may be a fundamental problem with interpretability work that attempts to understand neural networks by decomposing their individual activation spaces in isolation: It seems likely to find features of the activations - features that help explain the statistical structure of activation spaces, rather than features of the model - the features the model’s own computations make use of.
Cognitive Work and AI Safety: A Thermodynamic Perspective — LessWrong
Introduces the idea of cognitive work as a parallel to physical work, and explains why concentrated sources of cognitive work may pose a risk to human safety. Acknowledgements. Thanks to Echo Zhou for feedback and suggestions. Some of these ideas were presented originally in a talk in November 2024 at the Australian AI Safety Forum slides for which are here: Technical AI Safety (Aus Safety Forum 24)and the video is available on YouTube.
Causal Undertow: A Work of Seed Fiction — LessWrong
A short story, best enjoyed in-context... try exploring possible meanings of some of the unusual terms with your favourite commercially available Solomonoff inductor. Acknowledgements. Thanks to Simon Pepin Lehalleur and Ziling Ye for feedback and suggestions. This post is the "fun" half of a pair, for the serious version see Cognitive Work and AI Safety. The sand was still cool under Dr. Chen's feet, the sun barely cresting the horizon.
The Queen’s Dilemma: A Paradox of Control — LessWrong
Our large learning machines find patterns in the world and use them to predict. When these machines exceed us and become superhuman, one of those patterns will be relative human incompetence. How comfortable are we with the incorporation of this pattern into their predictions, when those predictions become the actions that shape the world? My thanks to Matthew Farrugia-Roberts for feedback and discussion.
Selection for error correction in neural networks? — LessWrong
Epistemic status: Low confidence. This is a bit of an experiment in lowering my bar for what I make a proper post instead of a shortform. It's just an early stage idea that might go nowhere. It looks to me like neural networks might naturally develop error-correction mechanisms because error correction dramatically increases the volume of solutions in the loss landscape. In other words, neural networks with error correction seem like they’d tend to have much lower learning coefficients.
Australian AI Safety Forum 2024 — LessWrong
We're excited to announce the inaugural Australian AI Safety Forum, taking place on November 7-8, 2024, in Sydney, Australia. This event aims to foster the growth of the AI safety community within Australia. Key Details Dates: November 7-8, 2024 Location: Sydney Knowledge Hub, The University of Sydney Website: https://aisafetyforum.au Description The Australian AI Safety Forum is a two-day interdisciplinary event scheduled for November 2024.
SLT: The Hessian rank bounds the learning coefficient — LessWrong
TLDR: In a neural network with d parameters, the (local) learning coefficient λ can be upper and lower bounded by the rank of the network's Hessian d1: d12≤λ≤d12+d−d13. The lower bound is a known result. The upper bound is a claim by me, and this post contains the proof for it. If you find any problems with the proof, do point them out. The learning coefficient λ is a measure of loss basin volume and network complexity. You can think of it sort of like an effective parameter count of the model.
Timaeus is hiring! — LessWrong
TLDR: We’re hiring two research assistants to work on advancing developmental interpretability and other applications of singular learning theory to alignment. About Us Timaeus’s mission is to empower humanity by making breakthrough scientific progress on alignment.
Stagewise Development in Neural Networks — LessWrong
TLDR: This post accompanies The Developmental Landscape of In-Context Learning by Jesse Hoogland, George Wang, Matthew Farrugia-Roberts, Liam Carroll, Susan Wei and Daniel Murfet (2024), which shows that in-context learning emerges in discrete, interpretable developmental stages, and that these stages can be discovered in a model- and data-agnostic way by probing the local geometry of the loss landscape.
Simple versus Short: Higher-order degeneracy and error-correction — LessWrong
[minor] I'm a bit confused or I think you need some additional caveats in the intro. I would have said that Bayesian statistics with typical models is well understood as bias toward short description length but there are important caveats in the neural network case, at least conceptually.
Timaeus's First Four Months — LessWrong
Timaeus was announced in late October 2023, with the mission of making fundamental breakthroughs in technical AI alignment using deep ideas from mathematics and the sciences. This is our first progress update.
Growth and Form in a Toy Model of Superposition — LessWrong
TLDR: This post distills Dynamical and Bayesian Phase Transitions in a Toy Model of Superposition by Chen et al. (2023), where they study developmental stages of the Toy Model of Superposition, understanding growth and form from the perspective of Singular Learning Theory (SLT). Where do the bewildering and intricate structures of Nature come from? What purpose do they serve?
Announcing Timaeus — LessWrong
Timaeus is a new AI safety research organization dedicated to making fundamental breakthroughs in technical AI alignment using deep ideas from mathematics and the sciences. Currently, we are working on singular learning theory and developmental interpretability. Over time we expect to work on a broader research agenda, and to create informed by our research. Our primary focus is research. For now, we're a remote-first organization.
Apply for MATS Winter 2023-24! — LessWrong
Applications are now open for the Winter 2023-24 cohort of MATS (previously SERI MATS).
A list of core AI safety problems and how I hope to solve them — LessWrong
Context: I sometimes find myself referring back to this tweet and wanted to give it a more permanent home. While I'm at it, I thought I would try to give a concise summary of how each distinct problem would be solved by Safeguarded AI (formerly known as an Open Agency Architecture, or OAA), if it turns out to be feasible. 1. Value is fragile and hard to specify.
Towards Developmental Interpretability — LessWrong
Developmental interpretability is a research agenda that has grown out of a meeting of the Singular Learning Theory (SLT) and AI alignment communities. To mark the completion of the first SLT & AI alignment summit we have prepared this document as an outline of the key ideas.
Some Notes on the mathematics of Toy Autoencoding Problems — LessWrong
Anthropic's recent mechanistic interpretability paper, Toy Models of Superposition, helps to demonstrate the conceptual richness of very small feedforward neural networks. Even when being trained on synthetic, hand-coded data to reconstruct a very straightforward function (the identity map), there appears to be non-trivial mathematics at play and the analysis of these small networks seems to providing an interesting playground for mechanistic interpretability.
Show More
loading
Actions
Is this you?
As a journalist, you can create a free Muck Rack account to customize your profile, list your contact preferences, and upload a portfolio of your best work.Get in touch with Daniel
Contact Daniel, search articles and posts on X, monitor coverage, and track replies from one place.
Learn more about Muck Rack