Researchers must describe in a “clear, specific and complete way” how they obtain scientific results if reproducibility and replicability in science is to be upheld. That is according to a new report by the US National Academies of Sciences, Engineering and Medicine, which also calls on funding agencies to invest in open-source, usable tools to boost reproducibility in science.

The report, released yesterday, says that although scientists usually understand the difference between reproducibility and replicability, some fields interchange the terms or have one as an umbrella expression for both. The report defines reproducibility as “obtaining consistent results using the same input data, computational steps, methods, code and conditions of analysis”, while replicability is “obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data”.

The report states that while reproducibility and replicability are not pertinent methods when observing unique events or ephemeral phenomenon, rigorous descriptions and record keeping are still required to help understand and communicate the result. The NASEM report also recommends that journals should consider ways to ensure reproducibility for publications that make claims based on computational methods, “to the extent ethically and legally possible”. This could, for example, involve journals appointing a “reproducibility editor” to oversee such endeavours, according to Lorena Barba, a mechanical and aerospace engineer from George Washington University, who served on the 13-strong committee that wrote the report.

While not a model for all science, the report highlights how the CERN particle-physics lab shares data and results in a very complex environment. The Geneva lab uses a four-step process to handle the huge volume of data emerging from proton-proton collisions. Data are first collected, processed and stored before being released for analysis. The third step involves the resulting analysis and related documentation being preserved in a trusted long-term digital repository and then finally published for public consumption.

Transparent not opaque

Report chair Harvey Fineberg, president of the Gordon and Betty Moore Foundation, told Physics World that the report stresses that it is “essential” for scientists to characterise uncertainty in every finding. “We wanted to put this report and its focus on the concepts of reproducibility and replicability in the context of this larger purpose of gaining confidence in the knowledge that emerges from science,” he says. “If there is a theme I would stress in this report it is the concept of transparency – the notion of making available the data, the code, the computer and computational environment, the digital artefacts that support going from original data to whatever conclusions you reached.”

Barba notes that the committee understands that transparency is not easy given that most scientific endeavours are highly computational and reliant on complex data flows. “There are, of course, challenges to changing our norms of communication in fields like physics and my field of computational fluid dynamics,” Barba told Physics World. “What we are calling for is changing those norms to give importance to the full set of digital objects that are part of a scientific study and acknowledging that the scientific paper is insufficient today in its methods section to include all of the information needed for another researcher to confirm the results or build from those results.”