I am not a big fan of the peer review process. Too often it is a superficial ritual that adds little scientific value. Nevertheless, when it does work I think it can be very valuable. Here are some of my recent experiences that I thought were rather positive, and may be marginally interesting to readers.
I was sent a paper by JCP to review. Overall, I liked it but I thought it would benefit from some significant revisions. In a weird coincidence, I was visiting the same institution as some of the authors. I have been recently challenged about whether peer review really should be anonymous [see this discussion of SciPost] and so I took a risk. I signed my report and sent a copy to all the authors and told them I would be happy to meet to discuss the paper. We met and had a nice discussion. However, it was interesting that JCP told me that they had deleted my self-identification as it was against their policy.
I was sent a paper by PRL to review that I (and other referees) took a strong dislike to. The cover letter was also "Interesting". I wrote a concrete critical report. However, for the first time ever, I used the box for "Comments that will only be seen by the Editors". I said the authors had inappropriately used and cited my own work, that the paper was in the class "Not even wrong", and that if PRL published it, PRLs reputation in a certain community would suffer. Maybe I am a coward, but I am glad I was anonymous.
I got a paper I liked to review from JCP. However, the authors did not engage with a whole physics literature that was relevant to the paper and they needed to use it to sharpen their results. I think the final paper will be much better and more interesting.
A recent paper with my postdoc got two critical but constructive reports from PRB. This required some new calculations and comparisons but was much better as a result. Today we heard it was accepted.
I now decline all referee requests from luxury journals. I have limited time and would prefer to invest in journals that I think are making a positive contribution to science.
Have you had any recent positive experiences as a referee or received a helpful report?
Subscribe to:
Post Comments (Atom)
From Leo Szilard to the Tasmanian wilderness
Richard Flanagan is an esteemed Australian writer. My son recently gave our family a copy of Flanagan's recent book, Question 7 . It is...
-
Is it something to do with breakdown of the Born-Oppenheimer approximation? In molecular spectroscopy you occasionally hear this term thro... -
If you look on the arXiv and in Nature journals there is a continuing stream of people claiming to observe superconductivity in some new mat...
-
I welcome discussion on this point. I don't think it is as sensitive or as important a topic as the author order on papers. With rega...
My coauthors and I were recently astounded by an almost line-by-line, 35 page referee report #1 on a 60+ page paper for RMP. It was fussy, infuriating, and helpful, and its color-coded comments included brief essays on points of grammar and punctuation. But we generally much preferred it to #3's report: "...having carefully read through all 67 pages of this review article, I can say without hesitation that I feel that this outstanding manuscript should be essentially published as is." The two extremes illustrate your well-taken points.
ReplyDeleteThanks Thad for sharing this story. It is an amazing illustration of the two extremes.
DeleteThe first referee would be frustrating but you should take it as a great compliment on your paper that she/he thought it worth their time to prepare such a detailed report.
A few random remarks:
ReplyDeleteI very often use the "comments to the editor" box.
I find the editors of PRB and PRL to be pretty open to discussion; I've even had follow-up questions about journal policy based on things I wrote in the "for the editor only" box.
Even authors of good work very often oversell their work in the cover letter... So I take this with a grain of salt, and focus solely on the paper. In the end, it is my opinion of the paper that should matter for the editor, not whatever they write in the cover letter.
Positive experience:
While I have had my share of critical reviews, most of them useful, the most positive experience I had was that a reviewer rejected a paper (PRL) based on not having looked at the references that would have explained what he did not understand. We wrote a rebuttal, and the referee came back and apologized and (after some other comments) recommeded publication. It's not the fact that we were right, but the fact that an anonymous referee is that honest and polite to admit being wrong. Most of the time it's a one way street; the authors are wrong and the referee is never wrong. While one can learn from that as an author (message did not come across), it's refreshing to have a conversation about science that is more bilateral as in this case.
Thanks for sharing your experiences.
DeleteI agree that the cover letter should not really matter, except when it is a resubmission. Then the response to the referees is key. And here, I particularly look for concrete changes to the manuscript and/or a substantial engagement with criticisms. This point is made by Prof. Scholl below.
As an editor for a (non-luxury) journal, I get to see many examples of reviews where the referee is genuinely interested in improving the quality of the published paper.
ReplyDeleteMy unsolicited tip for authors responding to referee comments is: describe what change you have made *in the manuscript* as a result of the referee's comment. Don't just give a long discussion of the referee's comment in a response letter without making any change in the manuscript.
Thanks for the comment.
DeleteI agree completely it is very important to make changes in the manuscript in response to the referees and spell that out in the cover letter.
This is a point I continually make to my co-authors, particularly the less experienced.
Yet, it is always fascinating and disturbing to me how many people don't do this. I see this not as an editor, but as a referee who sometimes sees resubmissions.
As graduate student (and still am) we were taught to work with the referee's comments, make changes to our manuscript and use them to improve our work. Its is frustrating to hear people just blow the reports off.
DeleteFunny Things Peer-Reviewers Are Saying Behind Your Back
ReplyDeletehttps://www.nature.com/scitable/blog/labcoat-life/funny_things_peerreviewers_are_saying
Here is a sample.
"Merry X-mas! First, my recommendation was reject with new submission, because it is necessary to investigate further, but reading a well written manuscript before X-mas makes me feel like Santa Claus."
http://articles.adsabs.harvard.edu/cgi-bin/nph-iarticle_query?bibcode=2006ESASP1306..453G&db_key=AST&page_ind=8&plate_select=NO&data_type=GIF&type=SCREEN_GIF&classic=YES
ReplyDeleteAuthors venting out their feelings in acknowledging peer review in above paper
"We do not gratefully thank TA for his useless and very mean comments"
Hi. I want to share some experience from the other side of the review process, as someone who only recently did his first steps in science. I had a very long going project that resulted in a methodological paper, that is the one where I describe how to carry a specific type of calculations rather than providing just a brief methods overview and a set of best figures in my results section. The project was reformulated multiple times, the scope was adjusted repeatedly and so on. I had very low motivation at writing the manuscript. It took me roughly 3 years from our first meeting to the first submitted draft. I was devastated, I thought that what I do is not important and so on. All the usual underdiscussed struggles of a PhD candidate.
ReplyDeleteHowever, I got a very critical review of my manuscript. The author did a detailed list of suggestions like what paragraph should be rewritten for clarity, how the sections should be rearranged and what type of additional tests and figures we can include in the manuscript. I was thrilled. Finally, someone outside of my network had shown an interest in my work. That was a huge motivational boost, and I'm sure the fact that it was anonymous only added to the effect, which allowed me to finally finish that particular project and move on to do other projects.
Having said that, I really like the idea of pre-submission reviews, and I think if there was less of a negative view and plagiarism scare, the scientific community would benefit greatly from it. Because, you can argue that this is what you did in your first example
Our obsession with eminence warps research.
ReplyDeletehttp://www.nature.com/news/our-obsession-with-eminence-warps-research-1.22251
Some excellent lines from the above article.
When eminence begets eminence, noise in the system gets amplified. There's an element of luck to who ends up having the most success, and that luck will build on itself.
"Favouring elite scientists when evaluating papers and proposals is like giving Usain Bolt a 10-metre head start in his next race because he won his last five. It incentivizes scientists to present themselves and their results in the best light possible, to shun transparency and to deflect criticism. Those tendencies contribute to reproducibility problems.