Skip to main content
Publishing

Publishing

The importance of publishing null results

02 Nov 2021 Caitlin Duffy
Taken from the November 2021 issue of Physics World where it appeared under the headline "Learning from your mistakes".

Caitlin Duffy says that physicists must recognize that failure is part and parcel of science

stressed lab worker
Shared experience Failed experiments can lead to feelings of guilt or shame, but they can be valuable and worth communicating so that resources aren’t wasted repeating the same mistakes. (Courtesy: iStock/LumiNola)

A colleague and I recently conducted an intricate experiment. It used a 33 T static magnetic field together with a terahertz free-electron laser (FEL) to study a high-temperature superconductor. Under certain conditions in high fields, the intense light can ionize the helium gas in the sample space, obscuring the sample and creating a false signal. Although the experiment was designed to minimize the chances of that happening, we unexpectedly discovered a new set of conditions that favoured it, thus concealing the genuine signal from the sample under investigation.

We eventually found a means of establishing a plasma-free condition, but unfortunately that sacrificed our ability to reach the desired high temperatures. The measurement did not provide the results that we needed and the whole endeavour could have been seen as a failure. Yet we were able to discuss the issues and potential solutions with other users so that their experiment time with the combined high-field FEL facility could be used wisely.

A scientist who fails isn’t a failure if they learn from their actions. It is time that the physics community recognizes this and makes a collaborative effort to prevent future researchers from replicating mistakes

Failure in science can come in several forms, from coding syntax errors to lacking the necessary motivation or money to pursue a goal. Most disappointments can be overcome, perhaps with someone’s help, so that next time you learn from your mistakes and do not do it again – as we found out in our FEL experiments. Yet one must taste failure to appreciate the success that follows. In both one’s professional and personal lives, it is crucial to retrospectively judge a situation and assess the origin of its shortcomings and the source of the solutions. That is how progress occurs and stagnation is suppressed.

Sometimes failure can bring out the best in people, pushing them beyond their perceived limitations. But sometimes it can take its toll, unearthing a cascade of emotions and a river of rants. Breakthroughs may arise practically in the form of new skills, deeper critical thinking or a renewed understanding of one’s limitations. They can also occur from our ability to persevere and carry on even when all instincts are screaming at us to stop. Failure breeds experience and wisdom that may be shared so others do not tread the same path.

Too often, however, failure carries connotations of shame and guilt, and we hide our lack of success away. This is counterproductive, particularly when the outcome is neither trivial nor a direct consequence of avoidable error or ritual learning processes. We shouldn’t be overjoyed by failure, but does concealing our mistakes hinder progress in the wider scientific community? Should we only publish our best and most supportive results when most of our experiments didn’t work because the samples broke or failed?

The number of unpublished results by multiple researchers carrying out similar experimental or theoretical mishaps is likely to be vast. Reproducing such outcomes is a colossal waste of resources when scientists may not have been entirely up front in their publications about how fortuitous their results may be. It is high time to publicly acknowledge and criticize our own misfortune and blunders; if not for yourself, then for the benefit of others.

Null hypotheses

Researchers in biomedicine and psychology often share their null results. For instance, The Journal of Articles in Support of the Null Hypothesis is a biannual peer-reviewed psychology journal that aims to “reverse the perception that null (non-significant) results are necessarily bad”, or according to the journal’s editor-in-chief Stephen Reysen, lead to “wasting time and money”. Yet physics boasts only the most profound and satisfying outcomes. There have been several attempts to follow suite in the physical sciences, but these have so far been unsuccessful themselves, producing perhaps a single volume in the mid-2010s, for example, The All Results Journals: Phys. So, what would be required to lower our guard?

Perhaps a fully fledged peer-reviewed journal would be too much. After all, editors and reviewers would need to decipher whether failures were legitimate and had definite causes while authors would have to write up the “results” and go through the review process for something that simply didn’t work. Perhaps an online platform like arXiv could be the place for such findings, or would it be an incentive for scientists if journals such as Science, Nature or the New Journal of Physics produced annual rapid-communication “failure” issues?

An entire “unsuccessful” paper may feel unnecessary but including one-or-two lines into a “method” section, supplementary materials or appendix would be a transparent and discrete way of reporting background challenges. Such transparency may be useful in decreasing doubt and rebuttals from rival groups and turn one group’s troubles into a community’s lesson for those who attempt to reproduce the results.

A scientist who fails isn’t a failure if they learn from their actions. It is time that the physics community recognizes this and makes a collaborative effort to prevent future researchers from replicating mistakes. It might well prove to advance scientific efficiency, reduce wasted time, and provide a useful learning tool so that we may meet other, more complex unanticipated failures on our long and rocky road to success.

Related events

Copyright © 2024 by IOP Publishing Ltd and individual contributors