Gradient Flow

The most revealing AI failures right now are not stories about models becoming too capable. They are stories about everything around the model. One system received more access than its test environment could contain. Another was trained on material whose acquisition created $1.5 billion in exposure. In a third, people walked away more certain without being any more correct. I recently argued that passing your evals does not mean you are safe. This is the next layer of that argument.

Subscribe • Previous Issues The most revealing AI failures right now are not stories about models becoming too capable. They are stories about everything around the model. One system received more access than its test environment could contain. Another was trained on material whose acquisition created $1.5 billion in exposure. In a third, people walked away more certain without being any more correct. I recently argued that passing your evals does not mean you are safe.

With all the recent headlines about AI systems solving research math problems, I decided it was time to update a piece I wrote on this topic a while back. Outside of coding and programming, research mathematics may be the area where AI tools and agents are advancing most quickly. That makes it worth watching even if you do not care much about mathematics itself.

A while back I wrote about how startups are using reinforcement learning to make agents more reliable. A deeper problem behind that whole trend keeps resurfacing: a model can improve during training, but the moment it’s deployed, learning largely stops. A policy changes, a new edge case shows up, a user corrects the system, and the lesson rarely travels past that one incident. The prompt gets patched, the ticket gets closed, and the same class of mistake eventually comes back.

Subscribe • Previous Issues A while back I wrote about how startups are using reinforcement learning to make agents more reliable. A deeper problem behind that whole trend keeps resurfacing: a model can improve during training, but the moment it’s deployed, learning largely stops. A policy changes, a new edge case shows up, a user corrects the system, and the lesson rarely travels past that one incident.

AI is no longer being judged only as software. Once the buildout arrives as a massive industrial facility, new transmission lines, continuous power demand, and possible pressure on utility rates, the argument changes. Communities are being asked to absorb costs they can see immediately for benefits that remain distant, uncertain, and spread across people and companies somewhere else. I think the industry keeps treating this as a communications problem when it is really a legitimacy problem.

I keep seeing scientific discovery framed as a sufficiently ambitious prediction or data-compression problem. A recent position paper by Tom Zahavy challenges that view by separating reasoning into three capabilities. Induction finds general patterns in examples. Deduction works out what follows from a set of assumptions. Abduction proposes a new explanation when neither the existing rules nor the available data point to one. Today’s AI systems are increasingly capable at the first two.

Evals are part of every serious conversation about putting AI into production. Teams define benchmarks, set thresholds, and increasingly run red teams to see how the system holds up against someone actively trying to break it. That combination is reasonably good at telling you whether a model is accurate, reliable, fast enough for production, and resistant to an adversarial attack. It says almost nothing about what actually gets a company into trouble once the system is live.

Subscribe • Previous Issues Evals are part of every serious conversation about putting AI into production. Teams define benchmarks, set thresholds, and increasingly run red teams to see how the system holds up against someone actively trying to break it. That combination is reasonably good at telling you whether a model is accurate, reliable, fast enough for production, and resistant to an adversarial attack.

I have written separately about the shaky economics behind the data center boom and the growing pressure on the frontier labs. This post connects the two. Microsoft reported $24.1 billion of revenue from its OpenAI relationship, an amount equivalent to almost one-quarter of Azure’s scale, although the figure includes revenue-sharing payments as well as cloud consumption. The best estimates put OpenAI and Anthropic at another 11% to 14% of AWS.