My first experience of being peer reviewed did not begin well. Here’s the opening of the referee’s report:

“The purpose of publication is to disseminate knowledge to other people who may be able to use it. Since the model dramatically fails the authors’ own experimental tests more than half of the time, I can’t imagine anyone wanting to use it. I therefore recommend against its acceptance, here or anywhere else.”

What followed was a cavalcade of sarcastic remarks, incorrect assumptions, and occasional admissions that the referee didn’t know enough about the paper’s topic to make judgements (not that it stopped them). My senior co-authors were incensed; I was devastated. The words “I recommend against its acceptance, here or anywhere else” were burned into my brain and even now, more than 10 years later, reading them feels like a punch in the gut.

Shortly after receiving that scathing report, however, one of my senior co-authors took me aside. It turns out that his first peer-review report had been just as bad: he’d been told “Nothing new is in this paper, and what is new, is wrong” (I paraphrase slightly). That paper, he added, had eventually been published, and had gone on to receive more than 60 citations.

Buoyed by this admission, and guided by the referee’s tiny handful of useful comments (plus a far more productive critique from a fellow student), I set to work. A few months later, we submitted a much-altered version of the paper along with a request that, due to the adversarial tone of the original referee’s report, the new version be sent to someone else. This second referee proved more amenable, praising us for a “well-written manuscript with results that are likely to be of interest to many readers” and noting that while our model was indeed flawed, “any simplified model will have such problems”. In conclusion, the second referee wrote, “I think the authors are describing something that is novel, and of utility, and so I recommend that the manuscript be published.” After a few minor revisions, it was.

So in some ways, I suppose my first peer-review experience was a success. The revisions definitely made our paper better, and I’m pleased to report that, just like my co-author’s first paper, it has since gone on to receive more than 60 citations. Doing the revisions also made me a better physicist. My original contributions to the paper had been limited to running someone else’s code and performing experimental tests of someone else’s model, but by the time we resubmitted, I had a very thorough understanding of every aspect of the work.

But at the same time, it was a pretty miserable experience. I was a new PhD student at the time, and several other not-so-pleasant events in the lab had left me desperately short of self-confidence. If my co-authors hadn’t stepped in to reassure me that the first referee’s tone was out of line, I might have quit. Our experience also demonstrates why relying on the opinion of a single referee is a bad idea; although multiple reports do mean more work for scientists and journal editors, the potential for abuse by a single biased, spiteful or grumpy individual is just too great.

So my question for Peer Review Week is this: aside from mandating multiple referees (as many journals do) what changes could be made that would preserve the positive aspects of my first peer-review experience while eliminating the negatives? Please add your suggestions to the comments – and, if you’re brave enough, share your own first experience of peer review.