We changed requirements in a coding benchmark and found models talking themselves out of following them. Sometimes the agent understood the new requirement perfectly. It described what needed to change, considered an implementation, and then decided the hidden tests probably wanted something else. In one run, a model implemented the change correctly, asked another model for the benchmark’s original answer, and removed its own correct code after receiving it.
Is this you? As a journalist, you can create a free Muck Rack account to customize your profile, list your contact preferences, and upload a portfolio of your best work.
Claim your profile
Get in touch with Gavin
Contact Gavin, search articles and posts on X, monitor coverage, and track replies from one place.
Learn more about Muck Rack