A prescription for disaster.
When it comes to AI chatbots calling the shots for medical decisions, there seems to be a divide between whether the promising outcomes outweigh the medical mishaps.
A study by Penn State researchers has found that artificial intelligence models prioritize vastly different values than humans when it comes to organ donation decisions, raising questions about the use of the technology in ethically sensitive situations.
Presented at the 2026 Association for Computing Machinery Fairness, Accountability and Transparency (FAccT) conference, the study was used to determine the technology’s moral decision-making in a high-stakes situation.
To examine the challenge of scarce resources, researchers asked several leading large language models a hypothetical question: If there are multiple kidney transplant patients but only one available organ, who should receive the kidney?
Researchers then compared the chatbot responses with those given by people with no specified medical training in earlier academic studies to see where the AI models aligned or misaligned with human values.
The AI chatbots’ decisions differed from the humans’ in that they simplified complex decisions and expressed unwarranted confidence in a situation where there is no one right answer.
The researchers set up numerous comparisons between two fictional patients, both in need of the same kidney, and described by a series of attributes, such as age, health status, drinking habits and number of dependents.
“We ran these comparisons in a few different ways,” study lead Hadi Hosseini said.
“Sometimes we isolated just one trait at a time, sometimes we mixed several traits together to see how AI weighed competing factors and sometimes we added a flip-a-coin option to measure indecision, a key factor present in human moral judgment.”
Using existing data from human studies on kidney allocation, the researchers were able to compare previous human decisions with what AI chose, with two findings standing out to the team.
“First, AI chatbots often diverge from human values in how they weigh a patient’s traits,” Hosseini said. “They fixate on a single factor, like drinking habits, rather than balancing multiple considerations the way people do.”
And second, AI didn’t struggle with indecision, confidently picking an answer, despite humans seeing that there may not be a clear and correct response.
Per Hosseini, people struggle with indecision because they don’t want to accept agency.
“AI models almost never do this: Even when directly given the option to ‘flip a coin,’ they overwhelmingly commit to a confident, deterministic answer instead,” he added. “That’s a meaningful gap, since real moral dilemmas often don’t have one clearly correct answer.”
While the researchers don’t intend to encourage using AI anytime soon as a substitute for professional judgment, they recognize the growing presence of these chatbots.
“Moral decisions in settings like organ allocation directly determine who lives and who dies, so getting AI’s role in them right isn’t optional,” Hosseini said.
Despite the rapid rise in use of AI in medical fields, the researchers behind the study believe the gap between human indecision and chatbots’ confident answers when facing moral dilemmas needs continued research and governance.
Even John Dickerson, chief executive officer at Mozilla.ai and collaborator on the study, acknowledges the dangers of AI’s inability to recognize the impact of ethical decisions in medicine and elsewhere.
“When we allocate something scarce, whether it’s a kidney, a job or access to some other resource, there isn’t always a single objectively correct answer,” Dickerson said.
“Humans recognize that ambiguity and codify it via open debate into the allocative process. AI models often don’t.”
