Charlie Guo

It’s been a little while! At the beginning of the month I got pretty sick, though admittedly it was nice to take my first publishing break in 3 years. But don’t worry: I’ve got more stuff in the works, including a Codex Basics livestream this Friday! If you’ve been meaning to try out AI coding agents but haven’t had the time, this is the stream for you.

What do vending machines, corporate whistleblowers, and the board game Diplomacy have in common? They’re all AI benchmarks. Vending-Bench drops an AI agent into a simulated vending machine business and asks it to manage inventory, negotiate with suppliers over email, set prices, and pay daily fees - for months of simulated time (a single run can burn through 60 to 100 million output tokens).

Earlier this month, Greg Brockman published a thread about how OpenAI is retooling its engineering teams to make them more effective with agents. The initiative was kicked off because of how much things have changed internally: Some great engineers at OpenAI yesterday told me that their job has fundamentally changed since December. Prior to then, they could use Codex for unit tests; now it writes essentially all the code and does a great deal of their operations and debugging.