AJC ajcann.wordpress.com |
Tomorrow
we are going to experiment with open peer review practices in class. However, before
we learn how authors, reviewers, and editors feel in the process of peer
review, we need to clarify the terminology.
Peer review
has been established as basis for decisions about publication in journals,
awarding of grants, and academic promotion. Besides the growing use of
scientometric methods to quantify scientific performance and impact, peer
review is still trusted as foundation to assess the quality of scientific work.
Peer-reviewed papers or grant proposals are not essentially right in the sense
of a universal truth, but they are accepted because meeting the standards of
science. They are trusted and could become part of what is regarded
conventionally as body of scientific knowledge.
Traditional
peer review processes work similarly to the following flow of events:
http://undsci.berkeley.edu/article/howscienceworks_16 |
Such peer
review processes are rather time-consuming and costly, the duration from submission
to publication often exceeds one year. Moreover even though most scientists
regard it as wonderful way to learn about new research and their moral duty to
participate in this process, many complain about the workload attached to peer
review – this is called "reviewer fatigue".
"Without a doubt, the number one complaint at every editorial board meeting I attend is how to find good, qualified reviewers. As more and more manuscripts come in, the pool of tried and true reviewers is being bombarded with requests from multiple journals. Journals and books aren’t the only culprits. Many of the top experts being called on to review manuscripts are also being asked to review grant proposals. As has been discussed, a lot, this is an unpaid, extracurricular activity for which no real credit is given to the reviewer. Anecdotally, we hear that reviewer fatigue is the main issue with reviews not being completed or invitations being ignored or declined. A recent study took a look at what happens at one journal when a reviewer declines." Angela Cochran 2015
Traditional
non-transparent peer review faces many other challenges as well, such as reliability,
reviewer bias, fraud, conservatism, lack of interdisciplinary expertise and so
forth. The debate is summarised nicely here: http://p2pfoundation.net/Peer_Review
Several scandals involving scientific fraud and sloppy peer reviewers have led
to diminishing trust in the peer review system as such.
Open peer
review cannot provide solutions to all of the problems of traditional peer
review, but first and foremost it can help to re-establish trust in the ideal
of peer- expertise. Opening up the peer review process basically entails
revealing the identity of authors and reviewers to each other, or even to the
public. The situation could be enriched by open conversations and consultations
in the scientific validation process, letting the public watch the process
unfold or even participate. We can discern three basic levels/dimensions of
open peer review:
• open-identity
or attributed peer review (as opposed to anonymous peer review);
• open-access
or disclosed peer review, where the peer review contents are published;
• open-invitation
or unsolicited peer review, where anyone interested can join and contribute to
the peer review process.
Furthermore
we find several other common characteristics of open peer review, as collected
in a literature review by Emily Ford:"Defining and Characterizing Open Peer Review: A
Review of the Literature" (2013). Library Faculty Publications and
Presentations.
Besides the
signed or disclosed review, we also find new forms of editor-mediated review.
This means that also the editorial process, such as pre-selection and final
decision making is made transparent. Crowdsourced modes of review entail a
transparent public process without any limit to the number of comments or
reviews, sometimes even without editorial moderation. This is already happening
at platforms for publishing pre-prints provoking pre-publication reviews, if
the article is promoted in the right way (through social media, by colleagues, …).
Synchronous review can occur either at the same time of the publication of an
article or grant proposal, or even during the process of writing. Whereas
post-publication review resembles more to the commenting on a blog post.
In an open
evaluation environment we find written peer reviews, numerical ratings, usage
statistics, social web information and citations in combination with
participatory elements from social media. Reviewers identities could be
authenticated and reviews themselves be gathered in a credit system. This could
provide an improvement to the problem of reviewer fatigue.
Image by: Laura Recio Hidalgo |
Peer Production process
When thinking further about peer review we notice that we might learn from other already more established forms of peer production.
Being
explained as third mode of production, peer production is not reducible to
market or government imperatives. Even though hierarchies, bureaucracies and
economies are part of all knowledge production, it also rests on decentralised
and flat, sometimes even emerging structures and events on local or global
grounds. It is certainly more than mass-collaboration.
Open peer review could benefit from all modes of peer production, not only the technically enhanced ones. It could provide the basis for new forms of collaboration in knowledge production.
Interesting links for the discussion of open peer review and open evaluation in general:
Louis Nenda says: Thanks km for this helpful summary. I would be very interested in how the open peer review session went along? Are you aware of the peer review game as reported here: http://www.biostat.jhsph.edu/~jleek/peerreview/
ReplyDeleteYours, Louis
We will see how it works, currently I am waiting for the participants to post their comments to their colleagues texts.
Delete