Even though most Americans say they don’t trust artificial intelligence (AI), researchers have found a startling new metric that seems to show the opposite: people are more likely to buy something after reading an AI summary of online reviews than one written by a human. Yet AI hallucinated 60% of the time when queried about the products.
The team, from the University of California, San Diego (UDSD), claims this is the first study to show how cognitive biases introduced by large language models (LLMs) have real consequences on user behavior. They also say it’s the first project to measure the quantitative impact of AI influence on people.
Article continues below
First, the scientists prompted AI to summarize product reviews and interviews in the media, before asking AI to fact-check new descriptions to ascertain whether they were true. In a second task, AI was shown both news-story descriptions and falsified versions of the same descriptions it was similarly tasked with fact-checking.
“The consistently low strict accuracy, compared to actual news and falsified news accuracy, highlights a critical limitation: the persistent inability to reliably differentiate fact from fabrication,” the scientists wrote in the study.
The most striking finding involved online product reviews. Participants were far more likely to express an interest in buying a product after reading an AI-generated product summary than after reading one written by a human reviewer.
Distorted consumer judgment
The researchers proposed two reasons why people were more likely to purchase based on AI summaries. First, LLMs tend to concentrate more on the beginning of the input text, a phenomenon called “lost in the middle.” Lead author Abeer Alessa, a research assistant and machine learning and human-computer interaction lecturer, refers to this in prior research.
Second, the LLMs become less reliable when processing information not included in their training data.
“Models tend to be wrong on whether the news description happened or not,“ Alessa told Live Science in an interview. “It may incorrectly state that an event never occurred, even if it did occur after the model’s training was completed.”
During testing, the team found that the chatbots changed the sentiments of real user reviews in 26.5% of cases and that they hallucinated 60% of the time when users asked questions about the reviews.
The project selected examples of product reviews with either very positive or very negative conclusions, and 70 subjects were assigned to read either the original reviews of common consumer products or the summaries of reviews that chatbots generated. Those who read the original reviews said they would buy the given product in 52% of cases, while those who read the AI-generated summaries said they would make a purchase 84% of the time.
The project used six LLMs; 1,000 reviews of electronics; 1,000 media interviews; and a news database of 8,500 items. They measured bias by quantifying framing shifts in the sentiment of the content, the overreliance on text earlier in the samples, and hallucinations.
When the participants read positive product review summaries, they reported they would buy the product 83.7% of the time, compared with 52.3% when reading original reviews.
The scientists concluded that even subtle changes in framing can distort consumer judgment and purchasing behavior significantly.
The authors acknowledged their tests were set in a low-stakes scenario, but warned that the impact could be more extreme in situations with higher risks.
“Some high-stakes scenarios include summarizing healthcare documents or students’ profiles in school admissions,” Alessa said. “In these contexts, framing shifts can affect how a person or the case is perceived.”
The team said in a further statement that the paper represents a step toward careful analysis and mitigation of content alteration induced by LLMs to humans, and provides insight into its effects. They said it could reduce the risk of systemic bias in areas like across media, education and public policy.
Quantifying Cognitive Bias Induction in LLM-Generated Content, Alessa et al., IJCNLP-AACL 2025
