This blog is collecting positions and reflections of open science in practice. Furthermore it is hosting the syllabus and teaching materials for the course: Open Science - the better science? at the University of Vienna, Austria, Department of Social Studies of Science and Technology. Comments are very welcome, please reveal your identity.
we are going to experiment with open peer review practices in class. However, before
we learn how authors, reviewers, and editors feel in the process of peer
review, we need to clarify the terminology.
has been established as basis for decisions about publication in journals,
awarding of grants, and academic promotion. Besides the growing use of
scientometric methods to quantify scientific performance and impact, peer
review is still trusted as foundation to assess the quality of scientific work.
Peer-reviewed papers or grant proposals are not essentially right in the sense
of a universal truth, but they are accepted because meeting the standards of
science. They are trusted and could become part of what is regarded
conventionally as body of scientific knowledge.
peer review processes work similarly to the following flow of events:
outcomes or project plans are written up in the form of an article or a project
proposal, which are then submitted to a journal or a funding organisation. Both
fields are highly competitive, acceptance rates typically vary between 5 and
15%. Editors or officers are sending the paper to several domain experts
trusting them with the assessment of the paper's quality. Those are the
"peers" providing either double blind or single blind feedback in
form of a written review, most of the time according to specific guidelines and
grading schemes. They also check if the paper meet typical scientific norms,
such as building of current state of the art, clear reasoning, thoroughly
designed research, and so forth. They recommend whether the paper is worth
publication or funding. Accordingly the authors may then revise their work and
review processes are rather time-consuming and costly, the duration from submission
to publication often exceeds one year. Moreover even though most scientists
regard it as wonderful way to learn about new research and their moral duty to
participate in this process, many complain about the workload attached to peer
review – this is called "reviewer fatigue".
"Without a doubt, the number one complaint at every editorial board meeting I attend is how to find good, qualified reviewers. As more and more manuscripts come in, the pool of tried and true reviewers is being bombarded with requests from multiple journals. Journals and books aren’t the only culprits. Many of the top experts being called on to review manuscripts are also being asked to review grant proposals. As has been discussed, a lot, this is an unpaid, extracurricular activity for which no real credit is given to the reviewer. Anecdotally, we hear that reviewer fatigue is the main issue with reviews not being completed or invitations being ignored or declined. A recent study took a look at what happens at one journal when a reviewer declines." Angela Cochran 2015
non-transparent peer review faces many other challenges as well, such as reliability,
reviewer bias, fraud, conservatism, lack of interdisciplinary expertise and so
forth. The debate is summarised nicely here: http://p2pfoundation.net/Peer_Review
Several scandals involving scientific fraud and sloppy peer reviewers have led
to diminishing trust in the peer review system as such.
review cannot provide solutions to all of the problems of traditional peer
review, but first and foremost it can help to re-establish trust in the ideal
of peer- expertise. Opening up the peer review process basically entails
revealing the identity of authors and reviewers to each other, or even to the
public. The situation could be enriched by open conversations and consultations
in the scientific validation process, letting the public watch the process
unfold or even participate. We can discern three basic levels/dimensions of
open peer review:
or attributed peer review (as opposed to anonymous peer review);
or disclosed peer review, where the peer review contents are published;
or unsolicited peer review, where anyone interested can join and contribute to
the peer review process.
we find several other common characteristics of open peer review, as collected
in a literature review by Emily Ford:"Defining and Characterizing Open Peer Review: A
Review of the Literature" (2013). Library Faculty Publications and
signed or disclosed review, we also find new forms of editor-mediated review.
This means that also the editorial process, such as pre-selection and final
decision making is made transparent. Crowdsourced modes of review entail a
transparent public process without any limit to the number of comments or
reviews, sometimes even without editorial moderation. This is already happening
at platforms for publishing pre-prints provoking pre-publication reviews, if
the article is promoted in the right way (through social media, by colleagues, …).
Synchronous review can occur either at the same time of the publication of an
article or grant proposal, or even during the process of writing. Whereas
post-publication review resembles more to the commenting on a blog post.
In an open
evaluation environment we find written peer reviews, numerical ratings, usage
statistics, social web information and citations in combination with
participatory elements from social media. Reviewers identities could be
authenticated and reviews themselves be gathered in a credit system. This could
provide an improvement to the problem of reviewer fatigue.
Image by: Laura Recio Hidalgo
Peer Production process
When thinking further about peer review we notice that we might learn from other already more established forms of peer production.
explained as third mode of production, peer production is not reducible to
market or government imperatives. Even though hierarchies, bureaucracies and
economies are part of all knowledge production, it also rests on decentralised
and flat, sometimes even emerging structures and events on local or global
grounds. It is certainly more than mass-collaboration.
Open peer review could benefit from all modes of peer production, not only the technically enhanced ones. It could provide the basis for new forms of collaboration in knowledge production.
Interesting links for the discussion of open peer review and open evaluation in general: