Nicole Sharp says that our scientific judgement has never been more important
In 2008 the US author Neal Stephenson published a speculative fiction novel called Anathem. Set on the fictional planet of Arbre, the story follows an order of monks, mostly isolated from the rest of the world, who every so often open up their doors to receive information on the state of the outside world.
In the book, Stephenson explores a Dark Age of the Internet, in which companies flood the network with plausible-sounding-but-false information in a ploy to force consumers to buy the firms’ own tools for distinguishing truth from falsehood.
Initially, this “crap”, as Stephenson calls it, had to be made by humans, but eventually the process is run by a so-called “Artificial Inanity”, which is able to create hundreds or thousands of bogus versions, or “bogons”, for every legitimate document.
Today, we face our own version of Stephenson’s Dark Age as our information landscape increasingly fills with AI slop, now going beyond AI-generated videos of uncanny kittens or images depicting politicians and celebrities in outlandish situations.
With the rise of chatbots powered by large language models (LLMs), our search results, social-media feeds and even scientific journals are getting polluted with data that, at first glance, appears relevant and plausible but contains no value beneath its gilded veneer.
As scientists, we must position our work in relation to other recent research. Yet that task gets harder when we have to sort out the real from the hallucinated. As LLM-generated slop infects our previously trusted knowledge base, we face an information landscape that is quickly rotting beneath us.
Where we could once track an idea to a paper with human authors (who bore consequences for acts of fraud), we face a future where we have to question whether an article’s authors, institutions or experiments ever existed, on top of evaluating its scientific merits.
This is not a hypothetical danger. In 2024 a team led by medical researcher Almira Osmanovic Thunström at the University of Gothenburg invented a fictitious medical condition called “bixonimania” and publicized it through posts on Medium and two pre-print papers.
Their goal was to test whether AI chatbots would pick up and propagate word of the fake disease, which they duly did. The researchers even tried to insert information that made the work obviously fake. This included acknowledging a colleague from Starfleet Academy, thanking the University of Fellowship of the Ring for funding, and even writing early on in the article that “this entire paper is made up”.
Trapped in a bog
As physicists, we are not immune to the dangers of information rot. A flood of LLM-generated articles prompted arXiv in May to announce consequences for authors whose submissions have “incontrovertible evidence that the authors did not check the results of LLM generation”, including having “hallucinated references”. Those consequences include a one-year ban from arXiv and the requirement, thereafter, that any submitted paper must already be accepted at a “reputable peer-reviewed venue”.
I initially welcomed the news. Harsh consequences are one way to discourage LLM usage, especially for sloppy shortcuts. But the more I reflect on it, the more I realize that arXiv’s consequences could eventually fall even on those who are not using LLMs to write their papers.
Relying on shortcuts is easy. Say you’re writing a literature review and paper A, which you’ve read and trust, has a fact you want to cite – but you notice that paper A cites paper B for this fact. It’s simple to scroll down to the reference section and copy paper B’s citation into your own citations (I would be lying if I said I’ve never done this myself).
But as the information landscape rots, this shortcut could run you afoul of policies like arXiv’s. What appears to be a valid citation – maybe even containing the names of authors you know personally – can be hallucinated.
Or it could be entirely fake, as a team of researchers who cited one of the “bixonimania” pre-prints in a peer-reviewed article discovered. Their paper was justifiably retracted by the journal editor, over their objections. By simply copying a reference without double-checking it, you can unintentionally join ranks alongside LLM users. Worse, you open yourself and all of your co-authors to being banned from arXiv. Peer review in the age of artificial intelligence
I am not suggesting that arXiv’s policy is flawed. Rather, I argue that those who intend their writing to be AI-free must be vigilant. We must double down on the skills that make us experts – our scientific judgement, information literacy, communication skills and careful scholarship. We should lean into practising these skills – and teaching them – even though they are precisely the skills that AI companies claim LLMs can replace.
In Anathem, a character simply hand-waves away their Internet full of bogons, but we have no such solution. Instead, as our information landscape rots, we find ourselves trapped in a bog. To get out, we must plan each step carefully and take none for granted. What appears to be solid ground may well prove treacherous. Moving safely is a slow process, full of checking and double-checking as we painstakingly build out paths and tools we can trust.
Simply choosing not to use an LLM when writing is no longer enough. We have only our own scholarly skills to sort the bogons from the ground truth. We will all have to slow down if we want out of the bog.
