Stewy Slocum

This is a preliminary research update. We are continuing our investigation and will publish a more in-depth analysis soon. The work was done as part of the MATS 7 extension. We'd like to thanks Cameron Holmes and Fabien Roger  for their useful feedback. TL;DR Claim: Narrow finetunes leave clearly readable traces: activation differences between base and finetuned models on the first few tokens of unrelated text reliably reveal the finetuning domain.

Epistemic status: an informal note. It is common to use finetuning on a narrow data distribution, or narrow finetuning (NFT), to study AI safety. In these experiments, a model is trained on a very specific type of data, then evaluated for broader properties, such as a capability or general disposition. Narrow finetuning is different than the training procedures that frontier AI companies use, like pretraining on the internet, or posttraining on a diverse mixture of data and tasks.