Tuesday, June 28, 2016

The problem of peer review - with comments by Vox Day

Peer review simply isn't what it is advertised to be; it is not only little more than editing, most of the time it is not even competent editing:
As a scientist with a 15 year career behind me so far, I am afraid that my experiences reflect this. Peer review is excellent in theory but not in practice. Much of the time, the only vetting the papers get are two relatively junior people in a field (often grad students or postdocs) giving it a thumbs up or thumbs down. That is absolutely it. In theory, the editors should make the decisions with the recommendations of the reviewers, but the editors rarely have the time or the expertise to judge the papers and often automatically defer to reviewers. Also, the papers should be reviewed by luminaries of the field, but these folks rarely have the time, and either decline invitations or bounce the work to a student or another trainee. It's not just bad papers that get through, but also good, rigorous, papers that are bounced by this system.

Many if not most of the people in academic science today, at least in biology (my field), are overwhelmed with the need to publish in such high volumes, few people with the needed expertise can afford the time to go over the results in detail. All this while, at the same time and for the same reason, the volume of papers that needs to be reviewed goes up. I've heard of (and had myself) papers havve lingered for 4+ months before they even went out for review.

And, in our rush to publish, we often don't read this literature carefully ourselves but start citing papers anyway, which weaves these potentially weak or erroneous papers even more tightly into the fabric of their field.

It's difficult to care a lot about the quality of your work when you know the extra effort often doesn't help something go through this fickle review process, and when you know people will cite it without really reading it closely. There is little incentive to spend longer on a paper to make sure everything is right and the results are reproducible because there is very little accountability for errors and huge rewards for being prolific.
The ironic thing is that True Believers and the I Fucking Love Science crowd genuinely believe that "peer reviewed science" is the gold standard for evidence. But there is a reason scientific evidence is not automatically allowed in a court of law, let alone considered conclusive, and the more we learn about the defects of peer review, the better we understand that science's credibility is limited.

We have a word for science that is trustworthy, and that word is engineering. Until science can be applied, it cannot be fully trusted to be correct.

All peer review is really designed to do is to reassure the reader that the information presented fits safely within the confines of the consensus status quo.