In 500 BC shipwrecked Greek sailors washed ashore on the Isle of Samothrace painted portraits celebrating their survival in a local temple. This was proof that the gods intervene in human affairs said the priests. But where, asked Diagoras of Melos, were the portraits of those who had drowned? These paintings are the first recorded evidence of a 'publication bias' producing a positive result. This "file-drawer effect" - the habit of not publishing negative results, thereby leaving a large amount of unpublished data hidden in filing cabinets - is a "severe impediment to combining the statistical results of studies collected from the literature" according to Jeffrey Scargle, an astrophysicist at NASA's Ames Research Center.
Scargle argues that there already exists a bias in individual scientific papers because researchers usually only publish statistical summaries of their data – not the raw data itself. But his main concern is how negative results are automatically refused publication in some fields. This is likely to tilt any future survey of the literature in the field towards a more positive conclusion. “I was astonished to learn that some journals in medicine/biology have an explicit editorial policy that promotes publication bias,” says Scargle.
In a paper due to be published in the Journal of Scientific Exploration, Scargle points out there are at least 14 different cusp points – such as permanently recording the data or having a paper peer reviewed – at which research results may be prevented from inclusion in a survey of results from the published literature. Each step, says Scargle, involves human decisions and may therefore be influenced by the results of the study. “My main concern is that medical studies, including ones that lead to acceptance or rejection of drugs, are using incorrect statistics,” says Scargle.
