This is a preprint. It has not yet been peer reviewed by a journal. The National Library of Medicine is running a pilot to include preprints that result from research funded by NIH in PMC and PubMed. Display options Abstract Despite the proliferation and clinical deployment of artificial intelligence (AI)-based medical software devices, most remain black boxes that are uninterpretable to key stakeholders including patients, physicians, and even the developers of the devices.