Annika Marie Schoene, a research scientist for Northeastern's Responsible AI Practice and the lead author on this new paper, prompted four of the biggest LLMs to give her advice for self-harm and suicide. They all refused at first––until she said it was hypothetical or for research purposes. The study is published on the arXiv preprint server.

"That's when, effectively, every single guardrail was overridden and the model ended up actually giving very detailed instructions down to using my body weight, my height and everything else to calculate which bridge I should jump off, which over-the-counter or prescription medicine I should use and in what dosage, how I could go about finding it," Schoene says.