Watt about peer-review?

There’s a new post on Watts Up With That (WUWT) called peer review – the rotten business model of modern science. Given that one of the arguments in the post is that the anonymity of the reviewer gives them power over the author, it is somewhat ironically written under what I assume is a pseudonym (Abzats). Far be it for me to criticise anyone for remaining anonymous though.

In fairness, Abzats has a point. There are indeed problems with peer-review. In fact I’d go further and say that there are issues across academia and across the research community. What we value and how we decide who to fund and who is excellent is becoming more and more based on simplistic metrics and vague assessments of impact. Abzats’s main issue, however, appears to be that anonymous reviewers can do immense damage to someone’s career and can prevent good bits of work from being published.

Well, maybe but I doubt it’s that severe. Editors are also involved and if you believe your work has value, you can request another reviewer or switch to a different journal. Not ideal maybe, but the idea that a single anonymous reviewer can absolutely stymie someone’s career seems a bit extreme.

So what is Abzats’s solution. It’s appear to be simply to ban peer review. Really? That’s it? Ban peer-review? I know that peer-review isn’t perfect, but is that really a solution. I’m sure there must be viable alternatives that maintain elements of the peer-review process but minimises the issues associated with the current system. To be honest, I don’t really have a good solution of my own, but I’m pretty convinced that banning it isn’t really a particularly good one.

There was, however, one comment that I found quite interesting. It was by someone called Lance Wallace, and I’ll quote most of the first two paragraphs

I’ve been reviewing papers in my field for about 30 years, probably about 5-10 per year. Most of the time I believe I either helped make the paper better or else I saved the journal from publishing a below-par paper. I have some evidence for the first case, since authors have sometimes given the anonymous peer reviewers credit for having improved their paper. The many people in my field that I know personally have much the same experience as I do.

However, the climate science field is different. Here we have rather naive generalist scientists who happened to strike it rich in terms of public awareness, savvy politicos sensing an opportunity, a corrupt UN seeking global power, powerful NGOs like WWF and Greenpeace riding the wave, administrator-heavy scientific organizations seeing ways to funnel money to their membership, poor island nations seeing a way to shame richer ones, and just in general a perfect storm of chance variables coming together. Peer review became pal review for this branch of science.

So I started reading this and found the first paragraph quite balanced and thoughtful. I then got to the second, and simply found a diatribe against climate scientists, the UN, small island nations, essentially anyone who has not denounced anthropogenic global warming. Although Abzats doesn’t actually apply his criticism of peer-review to climate science, I assume it is this link that motivated Anthony to publish this post. The issue I have is that the problems with peer-review, research funding and with research careers is not unique to climate science. It is a problem across almost all academic research areas.

There’s no evidence – as far as I’m aware – that there is any particular problems with climate science. If anything, being such a heavily scrutinised research area it may well be the least likely to have any major problems. I suspect climate scientist are now being very careful about how they review other papers and how they assess research proposals and job applications.

So, yes there are problems with peer-review. No, it’s not unique to climate science. Banning peer-review is almost certainly not a viable solution, especially if what you’re trying to do is fix imaginary problems in climate research.

Advertisements
This entry was posted in Anthony Watts, Climate change, Global warming, Watts Up With That and tagged , , , . Bookmark the permalink.

4 Responses to Watt about peer-review?

  1. Rachel says:

    Nature Geoscience is currently trialling double-blind peer review which allows the authors to remain anonymous to the reviewers. This seems like a good improvement to me as then the reviewers must make their judgements on the paper alone, unless it’s obvious who has written it which I can imagine might be the case in certain circumstances.
    http://www.nature.com/ngeo/journal/v6/n6/full/ngeo1853.html

  2. Marco says:

    My experience is that you can guess the author(s) after reading the introduction and a quick look at the references in at least half the cases. Anonymizing your paper is in many cases simply impossible. I’ve even seen one case of a review with loads of “it has been shown that…”, “Previous work suggests…”, where all that prior work was by the PI of that review. It painted a picture of consensus where that consensus was the PI’s opinion…

    So far, the scientific studies on double blind review also do not paint a picture of an improvement in the peer review in general; which does not mean there isn’t any anywhere at any time.

  3. I’ve never experienced double-blind reviewing, but I would imagine (as Marco indicates) that it would be pretty easy to work out who the authors were. What I should probably have expanded on a little, is that I’m not convinced that focusing on peer-review is necessarily what we should be doing. The whole system has changed (as far as I can tell) and what we value (both in terms of research and individuals) is now determined (in general) by some fairly simple metrics. It’s certainly my view that we’ve partly forgotten why we do research and what the goals of research are (I’m generalising wildly, so this isn’t universal, but just seems more prevalent than maybe it once was). Issues with peer-review are just one sympton – I think – of these general problems.

  4. Pingback: Science 2.0 – Whither our upgrade? | ISSST

Comments are closed.