Beth Barnes

By Josh Clymer, Hjalmar Wijk, and Beth Barnes An illustration of a sequence of events where rogue replicating agents emerge and cause harm. In 2023, METR1 introduced the term “Autonomous Replication and Adaptation” (“ARA”), which refers to the cluster of capabilities required for LLM agents to acquire and manage resources, evade shutdown, and adapt to novel challenges. Since then, the autonomous replication concern has become more mainstream.

Although METR has never claimed to have audited anything or to be providing meaningful oversight or accountability, there has been some confusion about whether METR is an auditor or planning to be one. To clarify this point: METR’s top priority is to develop the science of evaluations, and we don’t need to be auditors in order to succeed at this.

This is METR’s collection of resources for evaluating potentially dangerous autonomous capabilities of frontier models. The resources include a task suite, some software tooling, and guidelines on how to ensure an accurate measurement of model capability. Building on those, we’ve written an example evaluation protocol.