June 7, 2017

How will science be evaluated in the future?

By Belén Pacheco-Fiallos
Abstract: Traditionally, publications are considered the only scientifc product, and citations counting and the Journal Impact Factor as the only way of measuring academic excellence. However, this system of scholarship could be replaced by a new, more dynamic alternative model, in which adjacent scientific products are taken into consideration and broader forms of impact are measured by Altmetrics.  In a attempt to include doctoral students in this ongoing conversation, I had lenglty conversations with two of them and here I outline their opinions, fears and recommendations. 
Technological advancements have brought to a multi-level revolution in the scientific work process: from teaching and funding to publishing and peer-review, it seems that a variety of aspects of how scientists work are changing (1). Open Science, a part of this tech-driven revolution, strives for a more productive, accountable and democratic science. Several groups and institutions are involved in reaching this goal; therefore the term Open Science “evokes quite different understandings and opens a multitude of battlefields(2). However, something is sure, the conversation about Openness also concerns the questions of which – and how – to evaluate scientific activities in the future (2).
For scientists within biomedical sciences, advancing the Open philosophy includes an array of activities that go beyond the usual work of a scientist (3). This is problematic, because publications are traditionally considered to be the only scientific product. So far, citation counting and the Journal Impact Factor, which measures journal’s citations per article, have been the only metrics that mattered for scholarship (3), therefore getting cited has become so important that the “Publish or Perish” mentality plagues academia. These metrics are faulty: citation counting “neglects impact outside the academy, and ignores the context and reasons for citation” (4) and it has been estimated that only about 20% of citers read the original work (5). In addition, citations take too long to accumulate, which leads to the inappropriate use of the Journal Impact Factor instead of the citations to evaluate publications of single researchers (6). For the mentioned reasons, this narrow metric system impedes scientists from engaging in openness-driven activities, because they feel that doing so could jeopardize their careers.
Though deficient, these mechanisms are necessary for scholarly communication and evaluation (6), because researchers need to identify important academic content. In relation to this, Piwowar, the co-founder of the non-profit ImpactStory, suggested that maybe in this date and age, it is more accurate and beneficial to value all research products, not only publications (7). After analyzing the scientific process, an alternative, more dynamic model for scholarship can be recognized, which takes into consideration not only publications, but also several adjacent scientific products, such as methods, data, and self-publishing in the form of blog posts (Figure 1). Given thatexisting evaluation mechanisms often fail for alternative products” (7), these nontraditional academic efforts need a nontraditional metric system for judging them: Altmetrics (alternative metrics). The impact of new and old scientific products can be tracked by monitoring citations, blogs, tweets, downloads and attributions in research articles, such as mentions within methods and acknowledgments (7). Altmetrics allow a more comprehensive and nuanced understanding of the scientific impact and process than traditional citations (8).
Figure 1. Model of scholarhip
This discussion on how to evaluate science has possible repercussions for every stakeholder. On the research side, PhD students are situated in the most vulnerable position on the academic ladder. Social changes should be made on the basis of paying attention towards most disadvantaged group involved, thus the conversation about introducing new metrics mechanisms within the academic system should undoubtedly include doctoral students. Accordingly, I decided to have conversations with two PhD students in Vienna working in biomedical sciences within the same research institution. The interviews, which turned into a friendly discussion, were very broad and extended through several topics, from self-publishing to Altmetrics and the politics of Open Science.
Self-publishing: Blogs and Social Media
Regarding the alternative model’s emphasis on self-publishing, the interviewees were confused about what how to use twitter as scientist: “So, what would I have to do? Just starting a Western Blot and putting it on Twitter?” Nonetheless, one of them pointed out that a positive aspect of engaging in science communication was that this would allow the wider audience to “realize that thousands of people are working every week, 8-10 hours per day, to produce the tinniest piece of information to maybe make human life better in 50 years.” Sadly, both students also expressed deep concerns about engaging in this new time consuming activity:
This would take time from actual work… because I need to sleep! I would like to not need to sleep, but I need to.”
“As a PhD student it is very easy to be overwhelmed as is. So, I mean, when I started my PhD I wanted to write a blog, actually, but then you know, there is not enough time! You have to do experiments, you have to do your reading, you have to plan your stuff, you have to analyze your data, you have lab-book and you have to breathe!”
Furthermore, one student said that science communication is of vital importance, but before measuring anything on the Internet that can have real-world impact on people’s lives, it’s necessary to acknowledge that the Web can be dark place, where populism, racism and bigotry are widespread. Therefore, making decisions based on researchers’ social media engagement and response rates could be detrimental to those that don’t belong to the most privileged sections of society.

Lab-book, data sharing, materials & methods

With respect to the question of opening up the contents of their lab-book, sharing of data, material and methods, one of the interviewees was not convinced that exposing the entire intellectual process of a scientist is the right solution. One student feared that exposing their unedited lab-book would make them appear incompetent in front of the whole world and would allow people to judge them by their failures. This was especially troubling, because – as they have pointed out –  in science we need the right to fail.” Another disadvantage of opening up the lab-book would be the fact that more information is not always better: “A ton of data would be created which no one will sieve through anyway.” There was a sense of distrust in the wider scientific community, as “there will always be people that would only expose a part the process”.
Furthermore, the idea of revealing the lab-book contents, unprocessed data or detailed materials and methods in such a competitive field would make the doctoral student feel vulnerable to the “known sharks”. They expressed that the community should denunciate such people and not “give them a Nobel Prize.” Materials and methods are actually part of the publication, but are often not descriptive enough, “to the point that it can take you a long time to actually replicate anything”, which also prevents other researchers from “stealing” the methods. In addition, the interviewees mentioned that the scarcity of content within the materials and methods section is partly the fault of the Journals, because they only allow very limited space. They thought that the idea of expanding the understanding of the scientific system and what it produces is brilliant, but they lamented the perpetuating system: “It’s like if you have 3 people pointing a gun at each other, and one of them has to put their gun down first… Who’s that going to be?”
One of the students was very positive about the existence of Altmetrics, as they do widen the scope of scientific impact and would allow scientists to raise an extra reference when, for example, applying for a grant. However, the use of Altmetrics as the sole measuring point would not be useful, because it would prolong “the same problem as before: it doesn’t say anything about the importance or the value of the science itself.” What should mater is the impact of science in daily life, but that this can only be appreciated in retrospective, because “evaluating science in a short term is ineffective.” Along these lines, the student gave the example of Benoit Mandelbrot, who described fractals around 30 years ago: back then as well as nowadays, the general public was not aware of that fractals play an important role in the functioning of mobile phones. In relation to this, the students expressed that only scientists can evaluate science, because politicians or business people are in general disconnected from the academic world. As an example of this, they talked about how CRISPR (special segments of DNA that nowadays serve as the basis of a genome editing technology) was first described by Francisco Mojica, a Spanish professor at the Univesity of Alicante, Spain in 1993, but it needed two fellow scientists, Emmanuelle Charpentier and Jennifer Doudna in 2012 to realize CRISPR’s potential. Now the CRISPR-Cas9 technology is valued in 1 billion dollars around the world.

Furthermore, the students pointed out that anyone’s reasearch being picked up by news outlets depends on the communications department of their workplace. They also indicated that it is important to give science to the masses, but generally the information gets twisted the wrong way around in the media (Figure 2), so they would avoid common news outlets as a point of reference. They questioned the sensibility of promoting research via social media: “Is there a significant scientific blogging community?” and “Who tweets papers? Who puts papers on Facebook?” One of them said that they would know how to thrive in a social media environment, because they considered themselves as “digital native”. However, the other student felt very uncomfortable with the idea of having to use social media for science, as it feels like mixing personal with professional life.
When asked if they would use these metrics to decide which of two papers to read, one of them said, “I would read both papers and decide which one is better… If I have 50 papers that don’t have any citations, then we are talking!” Both students praise the fact that Almetrics include multiple measurement factors, because some people would find some platforms more informative than others. One said that they would be interested in Wikipedia, as it has a rather democratic way of serving as the repository of human knowledge, but would never use something like Google+ to locate good articles. They pointed out that “maybe it is just the scientific arrogance that we rely on our own system and what masses use is not good enough for us: There is this chauvinism in science.” Therefore, “using the platforms and the tools that we already have in the science community, like Research Gate and Pubmed, would be more useful… Instead of trying to move us to Facebook and Twitter.
So, what now?
Ideally, when hiring someone, the decision makers would base the decision on reading the content of the candidate’s work and not look where they have published: “Blank publications would allow the papers to be assessed on the basis of the science and not the packaging.” However, when applying for grant or when universities are being assessed, it is not realistic that anyone would read every paper in question… So, finding the right way to science is complicated and there is no easy solution to the problem. Nevertheless, two things are clear:
-       Scientists need to stop shying away from the responsibility of participating in discussion about the necessary systemic and policy modifications.
-       The changes have “to revolve around researchers’ own perspectives and experiences, so as to mediate between the wish for systemic change and the need to respect existing material and conceptual constraints, research demands, and ethical concerns” (3).

I support the idea of an alternative scholarly model, however I’m skeptical about the real-world implementation of Altmetrics… I just hope that “it manages to start a dialogue, because then maybe we will be able to find a better system, because we really need a better system.
(1) Puschmann, C. "(Micro) blogging science? Notes on potentials and constraints of new forms of scholarly communication." In Opening science , 89-106. Springer International Publishing, 2014.
(2) Fecher, B., & Friesike, S. "Open science: one term, five schools of thought." In Opening science , 17-47. Springer International Publishing, 2014.

(3) Levin, N., Leonelli, S., Weckowska, D., Castle, D., & Dupré, J.. "How Do Scientists Define Openness? Exploring the Relationship Between Open Science Policies and Research Practice." Bulletin of Science, Technology & Society 36, no. 2 (2016): 128-141    

(4) J. Priem, D. Taraborelli, P. Groth, C. Neylon. Altmetrics: A manifesto. October 26, 2010. http://altmetrics.org/manifesto (accessed June 1, 2017).

(5) Simkin, M. V., & Roychowdhury, V. P. "Read before you cite!" arXiv preprint cond-mat/0212043, 2002.

(6) Fenner, M. (2014). "Altmetrics and other novel measures for scientific impact." In Opening science, by S. Bartling and S. Friesike, 179-189. Springer International Publishing.

(7) Piwowar, H. "Value all research products." Nature 493, no. 7431 (2013): 159.

(8) Yeong, C. H., & Abdullah, B. J. J. "Altmetrics: the right step forward." 8, no. 3 (2012): 1-2.


  1. Dear Belén Pacheco-Fiallos, when I read the title of your blog post, it awakened my interest to read further, whole post, so I continued and I found/learned something new. It’s a simple blog post (what I find ok and positive), so it’s not hard for understanding. I’m saying that, because in my opinion, the main role of scientist, blogger, student-blogger (among the other roles) and writing something is to try to make it understandable for ‘mortal’ people, not only professional people, so to say. Another thing, regarding the method which you used for practical realization of your research question, interview, which is the most common method, but it’s not actually about method, it’s about whom you are interviewing, so it was a surprise for me when I saw that you interviewed Phd students and from this interview you pointed on main questions and doubts of this students. In my opinion, it’s a plus, but I was asking myself, it’s that all about that, is there something more, which is important? I didn’t saw anything about that why you choosed this topic.
    So, this is my evalution of your blog post:
    • Level of importance: Is the publication of relevance for the seminar and the target community of the blog, and does it provide important insights? Does the work represent a novel perspective or even new findings?
    I think that you choosed interesting topic to write about and it’s in the relationship with the seminar, because to me, altmetrics is (regarding how you represented main points about this question, especially the new model of scholarship) most about to be more open about materials, methods, data, and self-publishing in the form of blog posts and to go beyond traditional, to go play on the openness and that altmetrics include multiple ways of measurement (citations, blogs, tweets, etc.). The blog post gives something new, such as view of of the altmetrics from the perspective of phd students and among that, benefits of altmetrics. So, I give 5 from 5 stars for this level.

    • Level of validity: Is the core question/concern clearly formulated? Is the argumentation stringent? Is the interpretation balanced and supported by sources?
    There is a good formulation of core question,’how will be science evaluated in the future’ and the post smoothly goes into a right direction. But still, I’m not quite sure how science will be evaluated in the future… So, maybe there should be more sources about that core question and predictions. So, 4/5.
    • Level of completeness: Do the authors reference the appropriate scholarly context (e.g. refer to seminar literature)? Do the authors provide or cite all necessary information to follow their findings or argumentation? Do they miss relevant publications in the field? I think that article is supported with the adequate sources when it’s about findings about this topic and the seminar and adequate citations. 5/5
    • Level of comprehensibility: Is the language correct and easy to understand for the broad audience of the blog? Are images or figures well displayed and captions properly described? Is the article systematically and logically organized?
    Yes, the language is simple and easy to understand. Images are well displayed with originality, especially the first one. 4/5

  2. The blog covers issues surrounding altmetrics some problems with traditional modes of scientific Evaluation. You have explained the way in which judging scientific work purely by Impact factors and publications has severe drawbacks, and presented some of the reasons why many believe widening the scope of our considerations is necessary.

    You’ve added a couple new voices to the discussion (PhD students). Their contributions are particularly interesting since they each seem to have made Points on both sides of the issue (i.e. whether or not altmetrics would improve or harm the practice of scientific Evaluation).

    That diversity of Statements, combined with your own Analysis, hammers home the fact that this is a complex question without easy answers. Still, the nuances Need to be discussed if we are to come up with a good Explanation of how we should adopt new norms and practices, and shed some older ones.

    I think your post achieves its purpose. You explore some of the many considerations involved in scientific Evaluation practices, and in that you bring in two new voices who are in interesting positions to reflect on the different paths.

    Your separated sections do a nice Job of leading your Reader through the post, making it clear where your core Areas of concern lie.

    However, I would have liked to see more quotations from the PhD students, and in a slightly more structured way. It would also have been interesting to know a bit more about who they are. Not in Terms of their names or anything, just in Terms of how These matters have affected their own careers, which Areas of Research they are active in, and so on. Those Details could have important implications for how we understand their Statements.

    In the first section, you lay out the "traditional modes" of Evaluation, as we may call them. This serves as a concise, clear introduction to the matters at Hand.

    Then, in the next three sections, you consider some alternative forms of Evaluation which have not gotten as much Attention or use. It is within These sections that quotations from your two interview Partners Show up most, as they weigh in on the different available Options, and their drawbacks.

    I liked These sections since you explore some of the newer Options, including different opinions on them, but again, it would have been nice to flesh out some of your respondents' answers in These sections, to give your Analysis a Little meat.

    Which brings me to my second Overall critique, which is just personal. I want to know your opinion! You have done a wonderful Job of presenting an important question, but I would have liked for you to have taken a stronger Position about what you think is the right way Forward.

    However, I don't want to fault you too much for this. That's because you explicitly stated, especially at the end, that These are difficult matters to decide, and what you want to see, more than any particular Option, is a larger conversation taking place.

    In so far as I understand your post to be a call-to-converse, in that sense, I am completely on your side.

    So, let me assign some 'grades':

    Level of Importance: 5
    It is, of course, highly important to get clear on the reasons behind our methods of scientific Evaluation. Just like campaign finance in politics, it's an issue which affects all others.

    Level of validity: 5
    Since you argued forcefully for the Need to widen this discussion, that constituted a strong case in itself. Your sources were also novel, and enhanced the Quality of your Argument.

    Level of completeness: 4
    The reason I am not giving you a 5 here is because of what I said before, a couple times, about the lack of material from your interesting conversations with the doctoral students. I should probably not characterize it as a 'lacking', but more of a 'relative lacking'...relative, that is, to what I would have preferred.

    Level of comprehensibility: 5
    Yes, there is really nothing for me to criticize here. The blog is clear, well-structured, and well written.

  3. That last comment was written by Lucas by the way...

  4. Hi! Thanks for conducting the interviews and giving us insights into the opinion of PhD students to issues of openness and alternative ways to evaluate and make research visible. You touch on very important issues, especially the challenges coming with lack of time and vulnerability of researchers. You could have explained the open lab books as well as altmetrics a bit better for the readers not familiar with the topic. Furthermore, I would have wished for links to e.g. WIKIPEDIA definitions of "Wester blot" or "journal impact factor". All in all your post is a very informative and good read!