AI’s black box problem: discovering physics we don’t understand

In The Hitchhiker’s Guide to the Galaxy, Douglas Adams imagined a supercomputer called Deep Thought, built by a race of hyper-intelligent beings to calculate the Answer to the Ultimate Question of Life, the Universe and Everything. After a mere seven and a half million years of computation, the machine finally revealed its answer: 42.

There was just one problem. Despite their hyper-intelligence, none of those beings understood what the question had been. The joke was that any answer – no matter how precise – is meaningless without a thorough understanding of both the question and the route taken to obtain the answer.

Yet what was once purely science fiction is beginning to sound unexpectedly relevant to modern research. From structural biology to particle physics, artificial intelligence (AI) systems are increasingly involved in research, providing results of remarkable power and accuracy. They are being used to identify hidden patterns in vast datasets, to generate hypotheses, to accelerate simulations and to guide experiments.

As AI becomes increasingly embedded in the scientific process, could we be entering an era of discoveries without understanding?

But in some cases, even the creators of those AI systems are struggling to explain exactly how the AI arrives at its  conclusions. All of which raises an uncomfortable possibility. As AI becomes increasingly embedded in the scientific process, could we be entering an era of discoveries without understanding? And if so, what happens when science starts producing its own versions of the answer “42”?

AI and accelerated particle searches

For particle physicists, this question is no longer hypothetical. At CERN, the home of the Large Hadron Collider (LHC), AI is already woven into day-to-day research, subtly reshaping how physics research is carried out. Indeed, machine learning (ML) – one of the main pathways through which AI is achieved – has long played a role at the LHC.

Back in the early 2010s physicists working at the CMS and ATLAS experiments, routinely used ML in their search for the Higgso boson. The ML algorithms separated rare signals – a Higgs boson decaying either into two photons or into four leptons via Z bosons – from far more common Standard Model processes that produced similar signatures in the detector.

Machine learning did not discover the Higgs boson by itself, but it accelerated the search by making analyses more sensitive. In fact, achieving the same sensitivity without ML would have required between 15% and 125% more data, according to a review of Higgs-decay measurements (Nature 560 41). Some measurements would even have needed more than double the data.

But the scope of AI in particle physics has been expanding rapidly since then. Neural networks are now used to compress enormous datasets, accelerate simulations, and identify unusual events hidden among billions of particle collisions. By learning directly from examples, neural networks can uncover complex correlations in detector data that would be difficult for physicists to unravel using conventional analysis techniques.

Caterina Doglioni, an experimental particle physicist from the University of Manchester, UK, who uses ML for data compression as part of the ATLAS collaboration, says that deployment of AI is becoming difficult to separate from particle-physics research itself. “It’s pretty much everywhere,” she says, especially when it comes to experimental high-energy physics. “It’s critical at the LHC where far more data is created than can realistically be stored.”

How AI can help (and hopefully not hinder) physics

Read more

When it comes to compressing detector data, for example, one option is to use neural networks to force information through a computational “bottleneck” that preserves the most important features while dramatically reducing the storage required. The original data can then be reconstructed from this compressed summary, either exactly or approximately depending on the application (Rep. Prog. Phys. 84 124201).

This is different from the ML techniques used during the Higgs search. Whereas Higgs analyses relied on supervised algorithms that had been trained to distinguish known signals from background noise, these data-compression methods typically use “self-supervised” learning. They identify and preserve important patterns – without requiring labelled examples.

Algorithms learn what ordinary collisions look like and then flag events that deviate from those expectations

More recently, researchers have begun applying ML to a very different challenge. In the Higgs search, algorithms were trained to recognize signatures predicted by theory and distinguish them from background processes. But now, instead of having algorithms that search for a specific predicted signal, those algorithms learn what ordinary collisions look like and then flag events that deviate from those expectations.

In what is essentially a clever form of anomaly detection, the hope is that new physics might reveal itself as something unusual, even if physicists do not yet know exactly what form it will take. In a nutshell, ML used to be applied to problems where physicists knew what they were looking for; now it is applied to problems where we don’t know what we’re looking for just yet.

While no new particles have yet been spotted with these anomaly-detection techniques, they have progressed from theoretical proposals to real experimental searches. In 2024, for example, the ATLAS collaboration reported the results of an unsupervised anomaly-detection search using Run 2 data. No significant deviation from Standard Model expectations was observed, but the study proved that such methods can be deployed successfully on real collider data (Phys. Rev. Lett. 132 081801).

The black box problem

Yet the very feature that makes these systems so powerful – their ability to rapidly identify patterns humans may miss – also raises one of the most persistent concerns surrounding AI in research: the black box problem. Modern ML systems can uncover complex relationships in data, but researchers may struggle to understand exactly how a model arrived at a particular result. In particle physics, where extraordinary claims require extraordinary evidence, that lack of transparency can create unease.

For David Sutherland, a theoretical physicist at the University of Glasgow, the greater concern is not necessarily whether researchers can interpret every internal feature of a model – but whether they can trust and reproduce its outputs. “I would certainly say reproducibility [is more important] than interpretability,” he says.

That distinction matters. Science has always relied on the principle that results should survive scrutiny, be independently verified, and withstand repeated testing. A model does not need to reveal every detail of its inner workings – but researchers must be able to demonstrate that it behaves robustly across datasets and conditions.

Particle physics, in particular, has built extensive safeguards around that process. “We have very large review committees that check basically everything that goes on,” Doglioni says, with those structures providing an additional layer of protection against over-reliance on opaque systems.

Reliability and reproducibility

Yet as AI systems become increasingly embedded in research workflows, ensuring that reliability may become more difficult. Reproducing results may require not only access to data and code, but also to model architectures, training conditions and computational environments.

Unlike many commercial AI systems, the ML tools used in particle physics are often developed by particle physicists and shared openly with the scientific community. Even so, reproducing results may require access not only to data and code, but also to model architectures, training conditions and computational environments, all of which can influence an AI system’s behaviour.

The growing use of AI could also begin to change how physicists work and even how scientific value is measured. Christoph Weniger, a physicist at the University of Amsterdam who applies ML and AI techniques to particle physics and cosmology, argues that AI systems are likely to take on an increasingly active role in navigating complex analyses. Researchers, he believes, will shift into “a supervisory role in steering these different agents”, fundamentally changing how scientists interact with data.

Creative impact

Such changes could extend beyond day-to-day workflows. Traditionally, the currency of academic research has been publications. But Weniger suggests that AI may shift the emphasis away from productivity alone and towards something harder to quantify: originality.

Scientific value may increasingly depend not on how many papers a researcher produces, but on the creativity and impact of the questions they ask

As routine technical barriers to experiments and theoretical work become easier to overcome, he argues that scientific value may increasingly depend not on how many papers a researcher produces, but on the creativity and impact of the questions they ask.

The effects may also extend to how scientific knowledge is communicated. The scientific literature has long been criticized for being difficult to navigate, not simply because of its volume, but also because of the way researchers write. Nichol Furey, a mathematical physicist at Humboldt University of Berlin whose work explores whether there is an underlying algebraic logic to the structure of the Standard Model, believes AI could help bridge that gap.

“Authors in maths and physics often write in ways that are overly opaque,” she says. “AI systems write with their audience in mind, whereas human authors often don’t.” Yet that possibility creates a tension. The same tools that can make research more accessible can also make scientific text dramatically easier to produce.

Is our embrace of AI naïve and could it lead to an environmental disaster?

Read more

Communities are already beginning to grapple with the unintended consequences of widespread AI adoption. Recently, arXiv, one of the world’s largest repositories for physics and mathematics preprints, announced a crackdown on unchecked AI-generated submissions. It said it was introducing one-year bans for authors who upload papers containing obvious signs of unverified AI content, such as hallucinated references or leftover chatbot instructions.

The move reflects growing concern about the quality of AI-assisted scientific writing. Although arXiv’s new policy is too recent for its impact to be fully assessed, the threat of a one-year ban marks one of the strongest responses yet by a major scientific repository to unchecked AI-generated content.

The concern is not simply that AI can generate poor research. Now that producing scientific text is an almost frictionless exercise, the burden of judgement shifts increasingly onto readers, reviewers and future researchers trying to separate genuine insight from noise.

Training future physicists

Another potential problem with AI’s infiltration in many areas of research is training the next generation of physicists. This is generally a long process, where expertise is built gradually through years of practice and failure. If AI automates many of those steps, researchers may eventually face an uncomfortable question: how do you train future experts if the work that once trained them disappears?

This challenge extends far beyond physics. Across the education sector, schools and universities are already grappling with the ease with which generative AI can produce essays, solve problem sheets and complete programming assignments in seconds.

In fact, Wrishik Naskar, a particle theory postdoc at the DESY laboratory in Germany, wonders if research habits are also beginning to change. “When I get stuck, I go to my supervisor,” he says, “[but] many people, the first thing they ask is ChatGPT.” That approach might speed things up, but does it really help a researcher learn and grow? As Naskar puts it: “Focus on the learning. Results will follow if you have skills.”

Despite the many concerns, not everyone sees the transition to AI as a threat, and framing AI as a competitor to researchers may miss the point entirely. For example, Admir Greljo, a particle theorist at the University of Basel, believes the real shift is not replacement but collaboration. “It’s not that they will compete against AI. Entering into theoretical physics is still going to be very interesting and very important. AI is not competition, it’s a new tool.”

That perspective echoes a broader historical pattern. Physicists once calculated by hand, then with slide rules, then with computers. Each technological leap changed how research was done and prompted fears about what skills might be lost along the way. Yet rather than eliminating physicists, those tools expanded what physicists could ask.

Towards an AI scientist?

AI may simply be the next step in that progression. But it also presents something new. Unlike previous tools, AI can suggest ideas, identify patterns and increasingly participate in parts of the scientific process that once seemed uniquely human. So might that participation one day become independence? Could AI move beyond assisting physicists and begin carrying out the entire research process itself?

For Sutherland, a future in which AI systems perform the entire research cycle, from generating hypotheses to collecting data and even writing papers, is not difficult to imagine. But he still envisages a human “steering it” and acting as a “sanity check”, much like the relationship between a senior researcher and a PhD student.

Others go further. Greljo believes that fully autonomous scientific systems may not arrive in the near future, but sees no fundamental reason why they could not emerge eventually. Because science ultimately depends on reproducible results, he argues, humans could simply repeat calculations and test predictions to verify whether the AI had reached the right answer. Such a future, he says, is “not something that is impossible”.

We may be entering a world where machines help us uncover truths about the universe faster than we can fully understand them

We may be entering a world where machines help us uncover truths about the universe faster than we can fully understand them. But that need not signal the end of scientific inquiry. It is important to remember that AI models, much like many of the models used throughout scientific research, are imperfect and may always remain so. But perhaps they do not need to be perfect.

Science has always involved navigating uncertainty, constructing incomplete models and refining them over time. AI may help us discover things faster than ever before. It may reshape how future generations of physicists learn, work and think. But science has never simply been about generating results. It has also been about explanation, curiosity and deciding which questions are worth asking in the first place.

We should not treat AI as an oracle that bypasses the scientific method. Instead, we should view it as another tool in the physicist’s arsenal – a powerful one, certainly, but still a tool, helping researchers explore possibilities, test ideas and refine their models of reality.

After all, the problem with Deep Thought was never that it gave the wrong answer. It was that nobody knew what question they had asked.

The challenge for the next generation of physicists may not be deciding whether AI can discover new truths about nature. It may be making sure we still know which questions are worth asking.