The (ironic?) search for credibility and transparency in STS

By Daniel Marante

- The following text was written on assignment for the seminar Open Science: The Better Science? taught by Prof.: Katja Mayer; Faculty of Science and Technology Studies, University of Vienna, in January 2016 -

Abstract

In this blog post I review the practices suggested by some founders of The Berkeley Initiative for Transparency in the Social Sciences and authors of the article “Promoting Transparency in Social Science Research” (E. Miguel et al., 2014) and reflect on their application over the fundamental arguments and key concepts of the STS field taught by the University of Vienna in its STS MA Program. I have done this because of my deep intellectual preoccupation when reasoning and understanding the way some influent STS scholars philosophically conceive the world as they perform research and particularly, when they try to explain not the social aspects intertwined inside the scientific institution, but the foundations, methods and implications of science itself.

Topic in question: Promoting Transparency in Social Science Research. Miguel, E., Camerer, C., Casey, K., Cohen, et al. (2014).

This article by E. Miguel et al argues that social sciences should adopt some specific practices so they become more credible and useful in policy making. The reason is that policy-making done by governments and advocacy groups affects people at large and it should be done based on dependable evidence.

They contend that there is a systematic problem with the social sciences publishing organization, in which journals do not mandate high transparency standards. Consequently, the way social sciences are conducted might still require certain thoroughness. In this sense, they endorse the use of three core transparency practices: 

• Disclosure (or transparency in design), which refers to the systematic reporting of data collection and analysis. This means to report all measures, manipulations and data exclusions as well as how they arrived at final sample sizes.
• Preregistration (or transparency in intentions), which means to state clearly and disclose parameters of the study and,
• Open Data and Materials (transparency in analysis), which provide the means for independent researchers to reproduce reported results, test alternative specifications on the data, identify misreported or fraudulent results and better understand the intervention, measure and context of the studies.

Moreover, several authors of the article have established in 2012 the Berkeley Initiative for Transparency in the Social Sciences (BITSS) to “strengthen the quality of social science research and evidence used for policy-making” by “enhancing the practices of economists, psychologists, political scientists, and other social scientists in ways that promote research transparency, reproducibility, and openness”. BITSS organizes meetings to discuss research transparency practices and training in the tools that can facilitate the adoption of these practices. BITSS’s concrete objectives are (2016, January 5th). Retrieved from http://www.bitss.org/about/:

• Build consensus on key issues facing students, faculty, researchers, funders, journals, and other key partners to be more transparent in the social sciences;
• Improve our understanding of the problem and build evidence on solutions for increased transparency through long-term study of researcher practices;
• Increase supply of and access to tools and resources for research transparency, which is a necessary precursor for widespread adoption of best practices across the research community;
• Deliver coursework and change research practices at scale by harnessing the BITSS network of students, academic faculty, and researchers (a “push” mechanism); and
• Provide recognition and awards for the adoption of behaviors related to research transparency (a “pull” mechanism).

On transparency

The authors in their explanation propose that incentives and values in the scholar activity should be aligned so that research transparency in the social sciences is possible. They do not elaborate further in detail in this relation, but some hints are given to achieve this aligning: instruments to archive and share research material, plans and data, should be available for further researchers, for example.

In philosophy, an epistemic state E is strongly transparent to a subject S if and only if, when S is in state E, S can know that it is in state E, AND when S is not in state E, S can know it is not in state E. 

For example if we denote E as a transparent research and S as the subject that reads it, then we could infer that: i) the research is strongly transparent to the reader if and only if, when the reader reads it, he or she can know that E is in fact transparent (self awareness of reading transparent research) AND ii) when the research is non-transparent, the reader can know that he or she is in fact reading a non-transparent research. Both conditions need to be possible so that there is full acknowledgement of being in either case.

In other words, and linked back to the article, a study is transparent only and only if its design, intention and analysis is clear and known by the reader reading it. The purpose of this normative stance, I suppose, is to expose/disclose the essence of the study being done. This could be particularly beneficial since social sciences (and STS, too) appeal frequently to qualitative research methodologies and to relativist-based theories, which tend to be harder to divest from personal criteria. I contend that the main goal of transparency, is to make available the biggest amount of information from a research so that its data, analysis and conclusion can be contextualized in regard to its potential subjective aspects. Thus, I understand being transparent means either to be as objective and precise as possible, or if this is not the case, to acknowledge it as the researcher and as the reader.

On credibility

According to the Oxford English Dictionary, credibility is the quality of being trustworthy or believable and being trustworthy is to be able to be relied on as honest or truthful. Therefore, credibility has to do with reliability, which means to be dependable and consistent. Moreover, “Traditionally, reliability has two key components: trustworthiness and expertise, which both have objective and subjective components. Trustworthiness is based more on subjective factors, but can include objective measurements such as established reliability. Expertise can be similarly subjectively perceived, but also includes relatively objective characteristics of the source or message” (Flanagin and Metzger, 2008, P.5).

Said this, credibility could be an attribute of a researcher but as an attribute of a research, it means that it should be honest and consistent in its theoretical and methodological conception so that the analysis and conclusion are products of observation, reasoning and sense. Thus, an idea is credible because it has no logical or empirical flaws in its arguments and hence, it can be at least empirically verified, inclusive by other parties under similar conditions.

Expecting the former in STS

When I think about applying such transparency and credibility practices to some STS studies, I encounter with a fundamental contradiction. So far and since the beginning of my experience in the STS department in the University of Vienna, I have encountered with a systematic debunking of notions related to the establishment of a common philosophical ground where scientific and other type of ideas can be challenged, verified or rejected under impartial conditions. I refer to some of the key concepts in the STS field such as 1) lack of objectivity as the state of something being true even outside of a subject’s individual bias (which is at the same time very related to the notion of discovery or observation of nature), 2) the social constructivism argument by Knorr Cetina where she doubts of the unbiased –free of social influence- nature of (some) things, 3) the Strong Programme by Bloor et al in which they postulate the symmetry of knowledge, 4) the Paradigm Shifts by Kuhn, who explains the development process of science as a sole effect of socio-scientific paradigms substituting others, 5) Latour and Woolgar’s arguments in Laboratory Life, where they conclude that scientific facts are constructed out of a negotiation process between scientists in the lab, and similar theories or hypothesis developed within strong relativistic standpoints that are central in the field.

So far, these STS scholars seem to position themselves within this unique frame of reference to describe how the world that surrounds them runs the way it does. Of course, and perhaps not surprisingly, they explain it entrenched inside that frame of reference that is naturally free to use by anyone, but which rigorous falsifiability is rather not possible and hence, its credibility at least questionable.

Sure they are allowed to propose relatively feasible -although sometimes improbable- hypothesis of how the world and things inside of it (science as institution, science as inquiry method, society, etc.) operates, but when I reflect on trying to give high credibility and transparency to some intellectual work that is conceived within these strong relativistic standpoints, it gets somewhat problematic.

The fallacy?

Regardless the arguments presented in a description of the world, the problem when producing knowledge from relativistic standpoints and using inductive or abductive (instead of deductive) reasoning, is that very often this knowledge cannot be falsifiable and therefore, cannot be systematically, logically refuted.

The philosopher of science Karl Popper sustained that falsifiability or refutability of a hypothesis or theory is the inherent possibility that it can be proven false. A statement is called falsifiable if it is possible to conceive of an observation or an argument, which negates the statement in question. In this sense, falsify is synonymous with nullify, meaning to invalidate or ‘show to be false’.

Because of a well-defined relativistic frame of reference, under which an important group of analyses and conclusions are made in STS, their arguments could appear plausible, but nevertheless difficult –if not impossible- to falsify. As a corollary, however, the fact that some hypotheses cannot be exhaustively refuted using logical deduction or empirical evidence, does not mean they are to be taken as proved or verified. If done so, a fallacy would take place.

I contend that how in such a relativistic, non-positivistic realm, could systematic transparency and high credibility become feasible? According to the article, openness and transparency standards improve the quality and credibility of social sciences research; but how do we achieve this in a research field largely done without acknowledging notions that are central to these attributes?

I do not assert that there is a dead end in trying to make STS a more credible study field. What I argue is that the concept of credibility is inherently related to that common ground where interesting ideas seem interesting, because they attempt to describe with rigor the world we live in and towards we are curious.

Bibliography:

Woolgar, Steve; Latour, Bruno (1986) [1979]. Laboratory life: the construction of scientific facts. Princeton, New Jersey: Princeton University Press.

Miguel, E., Camerer, C., Casey, K., Cohen, et al. (2014). Promoting transparency in social science research. Science, 343, 30-31.

Flanagin and Metzger (2008), Digital media and youth: Unparalleled opportunity and unprecedented responsibility. In M. Metzger, & A. Flanagin (Editors), Digital media, youth, and credibility (pp. 5–28). Cambridge, MA: The MIT Press.

-   Master of Arts 'Science - Technology - Society' (n.d.). Retrieved January 7, 2016, from https://sts.univie.ac.at/lehre/master-sts/

Berkeley Initiative for Transparency in the Social Sciences. (2015, October 8). Retrieved January 7, 2016, from http://www.bitss.org/about/

Kuhn, Thomas (1970). The Structure of Scientific Revolutions. Chicago: University of Chicago Press.

- Gertler, B. (2003, February 7). Self-Knowledge. Retrieved January 7, 2016, from http://plato.stanford.edu/entries/self-knowledge/
  Boghossian, P. A.. (1994). The Transparency of Mental Content. Philosophical Perspectives, 8, 33–50. http://doi.org/10.2307/2214162

-  Knorr-Cetina, Karin (1981). The manufacture of knowledge: an essay on the constructivist and contextual nature of science. Oxford New York: Pergamon Press.
   Social Studies of Science 36/3 (June 2006) 489–493. SSS. London, Thousand Oaks CA, New Delhi: SAGE Publications.