Open Peer Review

AJC ajcann.wordpress.com

Tomorrow we are going to experiment with open peer review practices in class. However, before we learn how authors, reviewers, and editors feel in the process of peer review, we need to clarify the terminology.

Peer review has been established as basis for decisions about publication in journals, awarding of grants, and academic promotion. Besides the growing use of scientometric methods to quantify scientific performance and impact, peer review is still trusted as foundation to assess the quality of scientific work. Peer-reviewed papers or grant proposals are not essentially right in the sense of a universal truth, but they are accepted because meeting the standards of science. They are trusted and could become part of what is regarded conventionally as body of scientific knowledge.

Traditional peer review processes work similarly to the following flow of events:

Research outcomes or project plans are written up in the form of an article or a project proposal, which are then submitted to a journal or a funding organisation. Both fields are highly competitive, acceptance rates typically vary between 5 and 15%. Editors or officers are sending the paper to several domain experts trusting them with the assessment of the paper's quality. Those are the "peers" providing either double blind or single blind feedback in form of a written review, most of the time according to specific guidelines and grading schemes. They also check if the paper meet typical scientific norms, such as building of current state of the art, clear reasoning, thoroughly designed research, and so forth. They recommend whether the paper is worth publication or funding. Accordingly the authors may then revise their work and resubmit it.

http://undsci.berkeley.edu/article/howscienceworks_16

Such peer review processes are rather time-consuming and costly, the duration from submission to publication often exceeds one year. Moreover even though most scientists regard it as wonderful way to learn about new research and their moral duty to participate in this process, many complain about the workload attached to peer review – this is called "reviewer fatigue". 

"Without a doubt, the number one complaint at every editorial board meeting I attend is how to find good, qualified reviewers. As more and more manuscripts come in, the pool of tried and true reviewers is being bombarded with requests from multiple journals. Journals and books aren’t the only culprits. Many of the top experts being called on to review manuscripts are also being asked to review grant proposals. As has been discussed, a lot, this is an unpaid, extracurricular activity for which no real credit is given to the reviewer. Anecdotally, we hear that reviewer fatigue is the main issue with reviews not being completed or invitations being ignored or declined. A recent study took a look at what happens at one journal when a reviewer declines."  Angela Cochran 2015

Traditional non-transparent peer review faces many other challenges as well, such as reliability, reviewer bias, fraud, conservatism, lack of interdisciplinary expertise and so forth. The debate is summarised nicely here: http://p2pfoundation.net/Peer_Review Several scandals involving scientific fraud and sloppy peer reviewers have led to diminishing trust in the peer review system as such.

Open peer review cannot provide solutions to all of the problems of traditional peer review, but first and foremost it can help to re-establish trust in the ideal of peer- expertise. Opening up the peer review process basically entails revealing the identity of authors and reviewers to each other, or even to the public. The situation could be enriched by open conversations and consultations in the scientific validation process, letting the public watch the process unfold or even participate. We can discern three basic levels/dimensions of open peer review:

•  open-identity or attributed peer review (as opposed to anonymous peer review);

•  open-access or disclosed peer review, where the peer review contents are published;

•  open-invitation or unsolicited peer review, where anyone interested can join and contribute to the peer review process.

Furthermore we find several other common characteristics of open peer review, as collected in a literature review by Emily Ford:"Defining and Characterizing Open Peer Review: A Review of the Literature" (2013). Library Faculty Publications and Presentations.

Besides the signed or disclosed review, we also find new forms of editor-mediated review. This means that also the editorial process, such as pre-selection and final decision making is made transparent. Crowdsourced modes of review entail a transparent public process without any limit to the number of comments or reviews, sometimes even without editorial moderation. This is already happening at platforms for publishing pre-prints provoking pre-publication reviews, if the article is promoted in the right way (through social media, by colleagues, …). Synchronous review can occur either at the same time of the publication of an article or grant proposal, or even during the process of writing. Whereas post-publication review resembles more to the commenting on a blog post.

In an open evaluation environment we find written peer reviews, numerical ratings, usage statistics, social web information and citations in combination with participatory elements from social media. Reviewers identities could be authenticated and reviews themselves be gathered in a credit system. This could provide an improvement to the problem of reviewer fatigue.

Image by: Laura Recio Hidalgo

Peer Production process

When thinking further about peer review we notice that we might learn from other already more established forms of peer production.

Being explained as third mode of production, peer production is not reducible to market or government imperatives. Even though hierarchies, bureaucracies and economies are part of all knowledge production, it also rests on decentralised and flat, sometimes even emerging structures and events on local or global grounds. It is certainly more than mass-collaboration.

Open peer review could benefit from all modes of peer production, not only the technically enhanced ones. It could provide the basis for new forms of collaboration in knowledge production.

Interesting links for the discussion of open peer review and open evaluation in general:

• Should Post Publication peer review be anonymous? 

• Peer review – issues, limitations, and future development

• Peer Review - A Taylor&Francis Whitepaper

• Stop pretending peer review works

• Science first, careers later

• Why peer review needs a good going over

• Open Peer Review - a literature review