Measuring Interdisciplinarity in Scientific Research

My colleague at the School of International Affairs, Caroline Wagner, has just published a n insightful study, Caroline S. Wagner, J. David Roessner, Kamau Bobb, Julie Thompson Klein, Kevin W. Boyack, Joann Keyton, Ismael Rafols and Katy Börner, Approaches to understanding and measuring interdisciplinary scientific research (IDR): A review of the literature, Journal of Informetrics (in Press, corrected proof, available online 16 October 2010).

Abstract: Interdisciplinary scientific research (IDR) extends and challenges the study of science on a number of fronts, including creating output science and engineering (S&E) indicators. This literature review began with a narrow search for quantitative measures of the output of IDR that could contribute to indicators, but the authors expanded the scope of the review as it became clear that differing definitions, assessment tools, evaluation processes, and measures all shed light on different aspects of IDR. Key among these broader aspects is (a) the importance of incorporating the concept of knowledge integration, and (b) recognizing that integration can occur within a single mind as well as among a team. Existing output measures alone cannot adequately capture this process. Among the quantitative measures considered, bibliometrics (co-authorships, co-inventors, collaborations, references, citations and co-citations) are the most developed, but leave considerable gaps in understanding of the social dynamics that lead to knowledge integration. Emerging measures in network dynamics (particularly betweenness centrality and diversity), and entropy are promising as indicators, but their use requires sophisticated interpretations. Combinations of quantitative measures and qualitative assessments being applied within evaluation studies appear to reveal IDR processes but carry burdens of expense, intrusion, and lack of reproducibility year-upon-year. This review is a first step toward providing a more holistic view of measuring IDR, although research and development is needed before metrics can adequately reflect the actual phenomenon of IDR.

 The article suggests three important insights that are especially worth considering.  The first is the governance implications of measurement.  The authors suggest the possibilities of unpacking the assumptions inherent in understandings of interdisciplinarity, and consequently, its measurement.   Yet measurement trends that fail to expose their assumptions can have a significant effect on the substance of interdiciplinarity, and the legitimacy of its practice, through the definition of the parameters of its measure.  The second suggests the limits of a purely quantitative approach to the measurement of a thing.  Numbers do not lie.   Yet numbers may not tell any particular "truth" either.  In a society increasingly taught to approach numbers like fetish objects, the reintroduction of qualitative measures to analysis, and to measurement, provides an important and useful framework that avoids the indeterminacy of qualitative measures and the decontextualized and sometimes false certainty of quantitative measures.  The third suggests the power of economics and complexity in the  choice of measurement methods.  Sophisticated measures tend to be complex and expensive.  Economics factors may become as important a factor in the development of  consensus measures of interdisciplinarity as other factors.  That suggests that, perhaps at the limit, measurement remains fundamentally a political and cultural measure. Professor Wagner and her co-authors' engagement with these issues is worth careful consideration.