Jemima Winifred Allen

Many of the risks of using LLMs in healthcare have already received extensive attention. First, LLMs are currently highly vulnerable to jailbreaks and malicious attacks,39 which may limit the feasibility of any formal use by healthcare systems. Their reliability—while rapidly improving, especially when intensively prompted40—is also questionable, and they are prone to generating false or unsupported information (the so-called hallucinations).

Conclusion In this paper, we have set out the ethical considerations around the use of conversational AI for delegating procedural consent conversations. At the current time, there are no LLMs outside of research settings which are specifically designed to provide procedural consent conversations. There is limited evidence specifically evaluating the use of current-generation conversational AI for this purpose.

Introduction After discussing the various options for contraception with her doctor, Jane is now seeking a more permanent form of birth control and wishes to undergo tubal ligation. Instead of the usual brief interaction with a junior doctor to discuss the relevant risks and benefits, and give her consent to the operation, Jane has been given a link to a mobile app, ‘Consent-GPT’.