One of the themes in this week’s blog discussion surrounding Paul Krugman’s post on the subject was whether it is worth waiting around for peer review.
As I’ve tried to explain, the notion of journals as gatekeepers was largely fictional even 25 years ago. And I have a somewhat jaundiced view of how the whole refereeing/publication system has ever worked; all too often, it seems to act as a way for entrenched doctrines to blockade new ideas, or at least to keep people with new ideas from getting tenure at a good school.
Now there is lots of baggage going on there. Paul Krugman featured nicely in an article I wrote on journal rejections a few years back [here is the apparently SOPA free version]. But part of the issue is whether the journal process ads anything. In particular, does peer review help select the best quality content?
This week I heard a presentation by MIT PhD student, Danielle Li, that sheds some light on the subject. She wanted to examine the twin hypotheses that (a) peer review allows expert information to be brought to bear on evaluations and (b) peer review allows bias and favouritism to lead the day. To do this, she studied funding review panels at the NIH. She noted that experts would likely boost the chances of projects that were of high quality while lowering the chances of low quality ones in their domain of expertise. On the other hand, if reviewers were biased towards their area (either positively or negatively), this would have a similar impact on the chances of high and low quality projects. How do you know a project is of high quality if it is rejected? It turns out you can get a good idea from the work leading up to it (a la ‘Real’ The Grant Cycle).
So what did she find? Basically, reviewers in an area are both expert (their scores are related to quality) and positively biased (boosting funding chances by 2.9%) but in the end they help because the expert effect is greater, on average, than the bias effect. So it is basically better to have a biased expert than no expert at all.
Oh and by the way, grad students out there: this is a near perfect example of a dissertation paper. You would do well to look at it closely and learn.
Contribution Economy 