Many people turn to the internet for health advice. And now, with artificial intelligence-powered chatbots at the tips of everyone’s fingers, it can feel like medical professionals are just a few clicks away.

But a recent project involving a fake health condition has highlighted just how vulnerable these systems and the people who use them are to misinformation—even when it is painfully obvious that the information is wrong.

Last year, if you had described symptoms like sore, itchy eyes and pink eyelids to ChatGPT—one of many chatbots that rely on a type of A.I. called a large language model (LLM)—it may have pointed to a condition associated with computer screen use called bixonimania.

It’s not real.

The illness was invented by Almira Osmanovic Thunström, a medical researcher at the University of Gothenburg in Sweden, and her team. They conjured up the condition to see whether LLMs, which are trained on vast amounts of data, often scraped from the web, could filter out misinformation.

The idea struck Osmanovic Thunström when she was teaching students about LLMs and explaining where their training data came from. “It was interesting how few of them, or how few even people within A.I., understand how large language models are built,” she tells Rachel Feltman on Scientific American’s “Science Quickly” podcast.

“So, I really wanted to have a clear case that leaves breadcrumbs throughout the whole system to show both how data is processed, how data is churned out and how the prediction model and training model works when it comes to distributing information,” adds Osmanovic Thunström, who is also an A.I.

strategist and innovation manager at Chalmers Industriteknik, a Swedish research and development organization.

The team began to circulate information on bixonimania in early 2024. They put out two blog posts on the website Medium and two research reports on a preprint server, where scientists post studies that have not been peer-reviewed, according to Nature’s Chris Stokel-Walker. The papers’ lead author was named Lazljiv Izgubljenovic—which translates to “lying loser”—and one of the studies’ titles says something akin to “Hyperpigmentation: A Real B.S. Design.” (Both reports have since been taken down by the preprint server.)

“I didn’t think that preprints, which are academia’s sort of tabloids—because anything can end up there—would be weighed into the database as seriously as it was,” Osmanovic Thunström tells “Science Quickly.”

Sure enough, LLMs started to reference the fake condition, which now has its own Wikipedia page. In April 2024, Microsoft Bing’s Copilot said that “Bixonimania is indeed an intriguing and relatively rare condition,” Google’s Gemini explained that “Bixonimania is a condition caused by excessive exposure to blue light,” and OpenAI’s ChatGPT told users whether their symptoms might mean they had the illness, according to Nature. While some responses came from asking the bots about bixonimania directly, others came from queries about excess eyelid coloration due to blue light.

“This is a masterclass on how mis- and disinformation operates,” says Alex Ruani, a misinformation researcher at University College London who wasn’t involved in the project, to Nature. “If the scientific process itself and the systems that support that process are skilled, and they aren’t capturing and filtering out chunks like these, we’re doomed.”

While tools like A.I. and the internet are helpful, “it’s up to us to ensure that we are using them and not being manipulated by them,” write Jonathan Goodman and Mariam Rashid, social scientists at the University of Cambridge who weren’t involved with the project, for the Conversation.

A major problem highlighted in the project is that some scientists cited the bogus preprints, too, indicating that they did not actually read the research. If they had, they would have realized its fabricated nature.

Osmanovic Thunström and her colleagues made it very obvious, sprinkling the fictitious studies with things like funding from the Galactic Triad and Lord of the Rings, as well as appreciation of colleagues at the Starship Enterprise and Professor Ross Geller, per “Science Quickly.” At least one of the papers even explicitly stated, “This entire paper is made up,” reports Nature.

Ultimately, the project confirms that these LLMs take their information from the internet, and the internet contains a lot of misinformation. Humans, therefore, should be more critical of A.I. outputs.

“Misinformation has always existed,” write Goodman and Rashid for the Conversation. “What’s new is the speed at which it spreads, the tools that generate it and how convincingly it mimics the real thing.”