Is Michael Eisen right? Are there *really* problems with peer review?

I was interested in the 2nd argument of Eisen's that you cite: "2. The system is not very good at what it purports to do," specifically the sub-point about "filtering of the mass of papers to identify those one should read."

One of the measures that has received increased interest in addressing how effective peer review is at filtering knowledge is inter-rater reliability (IRR), which is a measure of how consistent reviewers' scores are for any given manuscript. Based on IRR, I think that the literature would support Eisen's argument.

One JAMA meta-analysis (a meta-anlysis being a study that aggregates previous findings and tests the extent to which there are consistent results across them) of 19 previous studies found very inconclusive results: "Editorial peer review, although widely used, is largely untested and its effects are uncertain" (Jefferson, et al., 2002).

A more recent meta-analysis of of IRR that reviewed 48 previous studies found that "According to our meta-analysis the IRR of peer assessments is quite limited and needs improvement (e.g., reader system)" (Bornmann L, Mutz R, Daniel H-D, 2010).

Even within a single journal, there seems to be little evidence that reviewers are, on average, agreeing about which articles are high-quality and which are not worth publishing. Looking at the 2264 articles submitted over 4 years, a study of the Journal of General Internal Medicine (JGIM) found that "Reviewers at JGIM agreed on recommendations to reject vs. accept/revise at levels barely beyond chance, yet editors placed considerable weight on reviewers' recommendations." (Kravitz RL, Franks P, Feldman MD, Gerrity M, Byrne C, et al., 2010).

Based on these 3 studies, I think that the literature would support Eisen's argument that peer review as a system, as it exists here-and-now, is not effectively filtering knowledge on a conistent meritocratic basis.

I think that matching reviewers more effectively to the content they want to review (versus materials they feel obliged to review due to social connections to the journal's editorial board), and increasing the number of reviewers for each manuscript, could improve the process. As Scholastica improves matching of reviewers to content and makes it easier to add more reviewers per article, we should see IRR increase.

I think that Eisen is correct in some ways.  Peer review does take a long time, but I am not sure what would constitute a useful alternative.  Legal scholarship uses a very different process of review where student editors select scholarship for publication.  This process is much quicker, and the turn around from submission to acceptance is usually less than two months.  The problem is, do you want students, who are at the beginning of their careers being the arbiter of scholarly potential?

My biggest concern about peer review is the fallacy that it is blind review.  In our current world, very few people submit work that they have not presented or vetted in another forum (i.e. conference, panel, workshop, etc).  Presentations are usually advertised on the Internet, which leave traces that can be later searched.  All a reviewer has to do is use google.com and search a title and find the name and affiliation of an author.