By James Dacey
In an opinion article just published on this website, the historian and philosopher Robert Crease discusses the pitfalls when trying to measure the “value” of artefacts in culture and society. The article was inspired by an appearance Crease made at a group debate at the Museum of Modern Art (MoMA) in New York City. The panel were trying to identify ways to measure the museum’s impact on culture and the wider economy.
Crease refers to something known as Goodhart’s law, which he explains as follows.
“Named after the British economist Charles Goodhart, who devised it in 1975, the law essentially says that once a measure is chosen for making policy decisions, it begins to lose value as a measure. Goodhart applied it to banking policy, but in other fields, too, measurement can distort not only the practice being measured, but also perception of the goal.”
One example of Goodhart’s law is trying to measure researchers’ quality by the number of papers they produce. As soon as you define this as a measure, researchers could work within the system by churning out unnecessary numbers of low-quality papers. Likewise, there may be many brilliant scientists out there who go undervalued, because they do not score highly in the recognized assessments of quality. The most widely used being the h-index, which attempts to combine the quantity and impact of research papers.
Clearly, in evaluating researchers there is no silver bullet. Researchers can contribute to science in myriad ways, and attempting to compare them against each other involves a heck of a lot of subjectivity. But in the real world, this is precisely what universities and funding agencies have to do. And in the process, they do resort to these metrics and value judgements. In this week’s Facebook poll, we would like you to imagine you have to rank a bunch of researchers, by answering the following question:
What is the best way to measure a researcher’s contribution to science?
Number of publications
The novelty of their research
The frequency with which their work is cited
The prizes they have won
The applications that have sprung from their research
Something else (please post a comment on the Facebook poll, or on this blog)
In last week’s poll, we asked you the question “Should scientists speculate openly in the mainstream media about new science results?” Of the people who responded, 67% chose the option “yes, this would make science appear more exciting”. The remaining 33% went for the option “No, it is only useful to present clear-cut results”. We came up with the poll question after the theoretical physicist and blogger Matt Strassler had accused scientists involved in the AMS dark-matter experiment of spinning their results to journalists. He believes the scientists made them sound more exciting than they were.
Thank you for you participation and we hope to hear from you again this week.
