How will science be evaluated in the future?

By Belén Pacheco-Fiallos

Abstract: Traditionally, publications are considered the only scientifc product, and citations counting and the Journal Impact Factor as the only way of measuring academic excellence. However, this system of scholarship could be replaced by a new, more dynamic alternative model, in which adjacent scientific products are taken into consideration and broader forms of impact are measured by Altmetrics.  In a attempt to include doctoral students in this ongoing conversation, I had lenglty conversations with two of them and here I outline their opinions, fears and recommendations. 

---

Technological advancements have brought to a multi-level revolution in the scientific work process: “from teaching and funding to publishing and peer-review, it seems that a variety of aspects of how scientists work are changing” (1). Open Science, a part of this tech-driven revolution, strives for a more productive, accountable and democratic science. Several groups and institutions are involved in reaching this goal; therefore the term Open Science “evokes quite different understandings and opens a multitude of battlefields” (2). However, something is sure, the conversation about Openness also concerns the questions of which – and how – to evaluate scientific activities in the future (2).

For scientists within biomedical sciences, advancing the Open philosophy includes an array of activities that go beyond the usual work of a scientist (3). This is problematic, because publications are traditionally considered to be the only scientific product. So far, citation counting and the Journal Impact Factor, which measures journal’s citations per article, have been the only metrics that mattered for scholarship (3), therefore getting cited has become so important that the “Publish or Perish” mentality plagues academia. These metrics are faulty: citation counting “neglects impact outside the academy, and ignores the context and reasons for citation” (4) and it has been estimated that only about 20% of citers read the original work (5). In addition, citations take too long to accumulate, which leads to the inappropriate use of the Journal Impact Factor instead of the citations to evaluate publications of single researchers (6). For the mentioned reasons, this narrow metric system impedes scientists from engaging in openness-driven activities, because they feel that doing so could jeopardize their careers.

Though deficient, these mechanisms are necessary for scholarly communication and evaluation (6), because researchers need to identify important academic content. In relation to this, Piwowar, the co-founder of the non-profit ImpactStory, suggested that maybe in this date and age, it is more accurate and beneficial to value all research products, not only publications (7). After analyzing the scientific process, an alternative, more dynamic model for scholarship can be recognized, which takes into consideration not only publications, but also several adjacent scientific products, such as methods, data, and self-publishing in the form of blog posts (Figure 1). Given that “existing evaluation mechanisms often fail for alternative products” (7), these nontraditional academic efforts need a nontraditional metric system for judging them: Altmetrics (alternative metrics). The impact of new and old scientific products can be tracked by monitoring citations, blogs, tweets, downloads and attributions in research articles, such as mentions within methods and acknowledgments (7). Altmetrics allow a more comprehensive and nuanced understanding of the scientific impact and process than traditional citations (8).

Figure 1. Model of scholarhip

This discussion on how to evaluate science has possible repercussions for every stakeholder. On the research side, PhD students are situated in the most vulnerable position on the academic ladder. Social changes should be made on the basis of paying attention towards most disadvantaged group involved, thus the conversation about introducing new metrics mechanisms within the academic system should undoubtedly include doctoral students. Accordingly, I decided to have conversations with two PhD students in Vienna working in biomedical sciences within the same research institution. The interviews, which turned into a friendly discussion, were very broad and extended through several topics, from self-publishing to Altmetrics and the politics of Open Science.

Self-publishing: Blogs and Social Media

Regarding the alternative model’s emphasis on self-publishing, the interviewees were confused about what how to use twitter as scientist: “So, what would I have to do? Just starting a Western Blot and putting it on Twitter?” Nonetheless, one of them pointed out that a positive aspect of engaging in science communication was that this would allow the wider audience to “realize that thousands of people are working every week, 8-10 hours per day, to produce the tinniest piece of information to maybe make human life better in 50 years.” Sadly, both students also expressed deep concerns about engaging in this new time consuming activity:

“This would take time from actual work… because I need to sleep! I would like to not need to sleep, but I need to.”

“As a PhD student it is very easy to be overwhelmed as is. So, I mean, when I started my PhD I wanted to write a blog, actually, but then you know, there is not enough time! You have to do experiments, you have to do your reading, you have to plan your stuff, you have to analyze your data, you have lab-book and you have to breathe!”

Furthermore, one student said that science communication is of vital importance, but before measuring anything on the Internet that can have real-world impact on people’s lives, it’s necessary to acknowledge that the Web can be dark place, where populism, racism and bigotry are widespread. Therefore, making decisions based on researchers’ social media engagement and response rates could be detrimental to those that don’t belong to the most privileged sections of society.

Lab-book, data sharing, materials & methods

With respect to the question of opening up the contents of their lab-book, sharing of data, material and methods, one of the interviewees was not convinced that exposing the entire intellectual process of a scientist is the right solution. One student feared that exposing their unedited lab-book would make them appear incompetent in front of the whole world and would allow people to judge them by their failures. This was especially troubling, because – as they have pointed out –  “in science we need the right to fail.” Another disadvantage of opening up the lab-book would be the fact that more information is not always better: “A ton of data would be created which no one will sieve through anyway.” There was a sense of distrust in the wider scientific community, as “there will always be people that would only expose a part the process”.

Furthermore, the idea of revealing the lab-book contents, unprocessed data or detailed materials and methods in such a competitive field would make the doctoral student feel vulnerable to the “known sharks”. They expressed that the community should denunciate such people and not “give them a Nobel Prize.” Materials and methods are actually part of the publication, but are often not descriptive enough, “to the point that it can take you a long time to actually replicate anything”, which also prevents other researchers from “stealing” the methods. In addition, the interviewees mentioned that the scarcity of content within the materials and methods section is partly the fault of the Journals, because they only allow very limited space. They thought that the idea of expanding the understanding of the scientific system and what it produces is brilliant, but they lamented the perpetuating system: “It’s like if you have 3 people pointing a gun at each other, and one of them has to put their gun down first… Who’s that going to be?”

Altmetrics

One of the students was very positive about the existence of Altmetrics, as they do widen the scope of scientific impact and would allow scientists to raise an extra reference when, for example, applying for a grant. However, the use of Altmetrics as the sole measuring point would not be useful, because it would prolong “the same problem as before: it doesn’t say anything about the importance or the value of the science itself.” What should mater is the impact of science in daily life, but that this can only be appreciated in retrospective, because “evaluating science in a short term is ineffective.” Along these lines, the student gave the example of Benoit Mandelbrot, who described fractals around 30 years ago: back then as well as nowadays, the general public was not aware of that fractals play an important role in the functioning of mobile phones. In relation to this, the students expressed that only scientists can evaluate science, because politicians or business people are in general disconnected from the academic world. As an example of this, they talked about how CRISPR (special segments of DNA that nowadays serve as the basis of a genome editing technology) was first described by Francisco Mojica, a Spanish professor at the Univesity of Alicante, Spain in 1993, but it needed two fellow scientists, Emmanuelle Charpentier and Jennifer Doudna in 2012 to realize CRISPR’s potential. Now the CRISPR-Cas9 technology is valued in 1 billion dollars around the world.

Furthermore, the students pointed out that anyone’s reasearch being picked up by news outlets depends on the communications department of their workplace. They also indicated that it is important to give science to the masses, but generally the information gets twisted the wrong way around in the media (Figure 2), so they would avoid common news outlets as a point of reference. They questioned the sensibility of promoting research via social media: “Is there a significant scientific blogging community?” and “Who tweets papers? Who puts papers on Facebook?” One of them said that they would know how to thrive in a social media environment, because they considered themselves as “digital native”. However, the other student felt very uncomfortable with the idea of having to use social media for science, as it feels like mixing personal with professional life.

Figure 2. Science new cycle, http://phdcomics.com/comics/archive.php?comicid=1174

When asked if they would use these metrics to decide which of two papers to read, one of them said, “I would read both papers and decide which one is better… If I have 50 papers that don’t have any citations, then we are talking!” Both students praise the fact that Almetrics include multiple measurement factors, because some people would find some platforms more informative than others. One said that they would be interested in Wikipedia, as it has a rather democratic way of serving as the repository of human knowledge, but would never use something like Google+ to locate good articles. They pointed out that “maybe it is just the scientific arrogance that we rely on our own system and what masses use is not good enough for us: There is this chauvinism in science.” Therefore, “using the platforms and the tools that we already have in the science community, like Research Gate and Pubmed, would be more useful… Instead of trying to move us to Facebook and Twitter.”

So, what now?

Ideally, when hiring someone, the decision makers would base the decision on reading the content of the candidate’s work and not look where they have published: “Blank publications would allow the papers to be assessed on the basis of the science and not the packaging.” However, when applying for grant or when universities are being assessed, it is not realistic that anyone would read every paper in question… So, finding the right way to science is complicated and there is no easy solution to the problem. Nevertheless, two things are clear:

-       Scientists need to stop shying away from the responsibility of participating in discussion about the necessary systemic and policy modifications.

-       The changes have “to revolve around researchers’ own perspectives and experiences, so as to mediate between the wish for systemic change and the need to respect existing material and conceptual constraints, research demands, and ethical concerns” (3).

I support the idea of an alternative scholarly model, however I’m skeptical about the real-world implementation of Altmetrics… I just hope that “it manages to start a dialogue, because then maybe we will be able to find a better system, because we really need a better system.”

(1) Puschmann, C. "(Micro) blogging science? Notes on potentials and constraints of new forms of scholarly communication." In Opening science , 89-106. Springer International Publishing, 2014.

(2) Fecher, B., & Friesike, S. "Open science: one term, five schools of thought." In Opening science , 17-47. Springer International Publishing, 2014.

(3) Levin, N., Leonelli, S., Weckowska, D., Castle, D., & Dupré, J.. "How Do Scientists Define Openness? Exploring the Relationship Between Open Science Policies and Research Practice." Bulletin of Science, Technology & Society 36, no. 2 (2016): 128-141    

(4) J. Priem, D. Taraborelli, P. Groth, C. Neylon. Altmetrics: A manifesto. October 26, 2010. http://altmetrics.org/manifesto (accessed June 1, 2017).

(5) Simkin, M. V., & Roychowdhury, V. P. "Read before you cite!" arXiv preprint cond-mat/0212043, 2002.

(6) Fenner, M. (2014). "Altmetrics and other novel measures for scientific impact." In Opening science, by S. Bartling and S. Friesike, 179-189. Springer International Publishing.

(7) Piwowar, H. "Value all research products." Nature 493, no. 7431 (2013): 159.

(8) Yeong, C. H., & Abdullah, B. J. J. "Altmetrics: the right step forward." 8, no. 3 (2012): 1-2.