Image by Ryunosuke Kikuno, from Unsplash
Studies Show ChatGPT and Other AI Tools Cite Retracted Research
- Written by Kiara Fabbri Former Tech News Writer
- Fact-Checked by Sarah Frazier Former Content Manager
Some artificial intelligence chatbots are giving answers based on flawed research from retracted scientific papers, recent studies show.
In a rush? Here are the quick facts:
- AI chatbots sometimes cite retracted scientific papers without warning users.
- ChatGPT GPT-4o referenced retracted papers five times, warning in only three.
- Experts warn retraction data is inconsistent and often hard for AI to track.
The research findings, which MIT Technology Review confirmed, raise doubts about AI reliability when it comes to answering scientific questions to researchers, students and the general public.
AI chatbots are already known to sometimes fabricate references. But experts warn that even when the sources are real, problems arise if the papers themselves have been pulled from the scientific record.
“The chatbot is ‘using a real paper, real material, to tell you something,’” says Weikuan Gu, a medical researcher at the University of Tennessee, as reported by MIT. “But, he says, if people only look at the content of the answer and do not click through to the paper and see that it’s been retracted, that’s really a problem,” he added.
MIT reports that Gu’s team tested ChatGPT running on OpenAI’s GPT-4o model with 21 retracted medical imaging papers. The chatbot referenced retracted sources five times yet it only warned users about this issue in three of those instances. Another study found similar issues with GPT-4o mini, which failed to mention retractions at all.
The problem extends beyond ChatGPT. MIT evaluated research-oriented AI tools by testing Elicit, Ai2 ScholarQA, Perplexity, and Consensus. Each cited studies which had been retracted and did not warn about this. The researchers said this happened multiple times in dozens of cases. Some companies say they are now improving detection.
“Until recently, we didn’t have great retraction data in our search engine,” said Christian Salem, cofounder of Consensus, which has since added new sources to reduce errors.
Experts argue that retraction data is patchy and inconsistent. “Where things are retracted, they can be marked as such in very different ways,” says Caitlin Bakker from the University of Regina.
Researchers warn users to stay cautious. “We are at the very, very early stages, and essentially you have to be skeptical,” says Aaron Tay of Singapore Management University.
Image by User_Pascal, from Unsplash
SIM Farm in NYC Could Have Shut Down Cell Service, Officials Warn
- Written by Kiara Fabbri Former Tech News Writer
- Fact-Checked by Sarah Frazier Former Content Manager
The US Secret Service discovered a large SIM farm operation located near New York City, raising concerns about potential future threats.
In a rush? Here are the quick facts:
- The farm’s 300 servers could have disabled cell networks across New York City.
- Network could send 30 million texts per minute.
- Experts say SIM farms are mainly used for fraud, scams, and fake accounts.
The officials disclosed on Tuesday that they located more than 100,000 SIM cards linked to about 300 servers spread across 35 miles from Manhattan, as first reported by WIRED . These criminal devices , often called SIM servers or SIM boxes, allow for massive spam text and call operations.
The Secret Service said the farm came to its attention after it was linked to “swatting” attacks against members of Congress in December 2023.
“Given the number of SIM cards all under the control of a single operation, it could have ‘disabled cell phone towers and essentially shut down the cell phone network in New York City,’” Matt McCool, the special agent in charge of the Secret Service’s New York field office, told WIRED..
Another law enforcement source said: “This network could be used to overwhelm cell towers… to give you an idea of capacity for disruption, this network could be used to send approximately 30 million text messages per minute, meaning it could anonymously text the entire United States in around 12 minutes,” as reported by WIRED.
The Secret Service confirmed the SIM farm had been used by organized crime and nation-state actors. The investigators took possession of the equipment but no one has been arrested, as reported by WIRED.
Experts believe the primary purpose was fraud. “The disruption of cell services is possible, flooding the network to the degree that it couldn’t take any more traffic,” said Ben Coon of Unit 221b, as reported by WIRED. “My gut is telling me there was some type of fraud involved here,” Coon added.
While SIM farms are not new, the scale of this operation and its proximity to critical US infrastructure have amplified concerns about their potential for disruption.
