ATG Interview- A Conversation about Research Impact

with Rachel Borchardt, Andrea Hebert and Camille Gamboa

Interview conducted by Caroline Goldsmith (Associate Director, The Charleston Hub)

Transcript edited by Leah Hinds (Executive Director, Charleston Hub)

The following is a lightly edited transcript of an excerpt from ATG the Podcast. The video for this episode can be found at https://youtu.be/j_9r8igeJX4?feature=shared

This episode features a roundtable discussion on Research Impact. Caroline Goldsmith, Associate Director, Charleston Hub, talks with Rachel Borchardt, Scholarly Communication Librarian and University Library Faculty, American University; Andrea Hebert, Research Impact Librarian, Louisiana State University; and Camille Gamboa, AVP of Corporate Communications, Sage. This conversation features both the librarian perspective and the publisher perspective of how each defines and supports research impact at their institutions, how the conversation has expanded over the years, and why forward thinking is crucial to meet the challenges such as academic silos, the dominance of traditional metrics, and the lack of commitment from U.S. universities to initiatives like DORA, and to support more effective impact of research. Librarians have an important role in shaping these conversations and helping institutions move beyond traditional metrics. We also discuss Policy Maps, which is a free tool offered by Sage for individual researchers to determine their policy impact.

CG/ATG: Hello and welcome to ATG the Podcast. I’m Caroline Goldsmith, Associate Director of the Charleston Hub, and today we’re doing a roundtable discussion around the topic of research impact, and we’ve invited three guests: Rachel Borchardt who is a Scholarly Communication Librarian and University Library Faculty at American University, Andrea Hebert who is a Research Impact Librarian at Louisiana State University, and Camille Gamboa who is AVP of Corporate Communications at Sage. And to all of you, I’d like to say welcome. Thank you for joining us today and, as I said, we’re focusing on research impact and how we come to understand and measure it and its effects. We’re going to get the librarian perspective and the publisher perspective today on what research impact is, and we’re interested in the recent case study that Sage conducted regarding Policy Maps, which is a free tool for individual researchers to determine their policy impact. But first, before we jump into all of that, let’s go around the room and introduce ourselves. Tell us what you do at your institution and how you support research impact.

CG: Well, very happy to be here. Thank you so much for hosting. As you mentioned, I do communications and public affairs work at Sage and, so beyond the traditional communications duties, I’ve been lucky enough to have been given the remit to explore how I can make and how Sage can make the research that we publish more impactful. Exploring questions like what does it take to make our research usable outside of academia? How can we solicit the type of research that will be helpful in policy and in practice? What constraints do we see as an industry to do this type of work? I get to explore all these really impactful, important questions and so it’s a really meaningful part of my job.

Just quickly also to introduce Sage. If you don’t know Sage, we are an academic publisher that was founded 60 years ago. It’s our birthday, and we are very independent, very mission-driven, very values-driven, and we have our foundations in the social and behavioral science. And so, a lot of the work that we do in this area, a lot of what I will be saying, focuses on supporting the unique needs of social science.

RB: I can go next. So hi, I’m Rachel. I’m the Scholarly Communications Librarian at American University, which is located in Washington, DC. And a bit about American University, it is a private, mid-size university and, as of very recently, we now have the R1 designation. We really pride ourselves on having a blend of teaching and research. I think the other thing that’s pretty unusual about us, since we’re in the DC area, our focuses, our primary things that we’re known for, include policy, international service, government. So we really are, like Sage, very social sciences focused. Not to say that we don’t have STEM and other disciplines, but those tend to drive some of our most high-impact research. For my job, part of my responsibilities include supporting and advancing research impact at the university. And that includes a wide range of activities from helping individuals with their tenure files and promotion to participating in university-wide initiatives related to research impact. I’m also on an NSF grant that focuses on research translation and public impact at my university.

AH: I’ll go ahead and wrap this part up. I’m Andrea and I work at Louisiana State University, A&M. It’s in Baton Rouge, Louisiana. It’s an R1, and it happens to be a public land, sea, and space grant university. We have about 40,000 students total, and there are 50 doctoral programs and about 3,500 graduate students. LSU is known for its STEM programs. So, my role as the research impact librarian is relatively new, and we’re still trying to figure out exactly what my work looks like. In addition to general scholcom work, I do tend to work with individual faculty who are interested in improving the visibility of their scholarship, and I can discuss traditional and alternative measures of scholarship and impact. Sometimes that translates into talking to departments about those things. I also give the institution and departments a broader view of their work and their impact using some of the more traditional measures.

CG/ATG: Well, thank you guys for that introduction and giving us some background on you. Let’s start with talking about how each of you define research impact and what it means to you, if you feel like you have a different perspective from your colleagues, and how you define it.

RB: I think I used to define research impact as more or less synonymous with the metrics that we’ve traditionally used to measure it; so, bibliometrics, citation counts, all that fun stuff. But, I think today, I would also describe it more as the total impact that all research and research outputs are having on a variety of individuals and populations, and that that can only be partially described and explained through metrics and qualitative measures.

AH: I agree with Rachel’s definition. However, I think that my view probably differs from that of my institution. Universities generally worry about rankings, and some of the rankings are definitely tied to traditional metrics. And because of LSU’s STEM focus, they are more accustomed to thinking about things like, you know, plain old bibliometrics. And they are probably more open to accepting things like patent citations. However, before I was the research impact librarian, I was the librarian for the schools of social work, education, and leadership and human resource development, so I was always aware that their work was not necessarily well represented in those traditional metrics. I’ve always been more able to see the value of things like policy.

RB: And I think for me, I forgot to say, in terms of alignment with my university, it’s been a process. I think when I first started getting interested in research metrics, there was more of a disconnect. I was really excited about all metrics as a way to start filling that gap. As I mentioned, the NSF grant that I’m on, public impact is a really big focus for us. And part of our work is trying to better align the broad spectrum of impact that we see with university policy. And there are definitely clashes between things like wanting to support public impact, but still wanting that good university ranking.

CG/ATG: Yeah. I’m glad you brought that up because I was just about to ask about altmetrics and things like social media posts and the impact that those have, and if that’s one of the things you’re interested in? And speaking of that, the alt metrics and social media posts and all these things now that we’re looking at over the years, how have you seen the conversation change around research impact?

CG: I’ve been in the industry for 13 years, and I’ve definitely seen shifts in that time to greater emphasis on outside impact. I think, you know, the three of us are on the same page. I agree with your definitions in terms of impact. And I think that we’re starting to see that trickle out to the different kind of parts of the science ecosystem.

The funders, for example, are starting to ask more and more, before they award a grant, that they’re asking questions about what outside impact research they’re funding will make. Publishers, journals, and trade societies are putting more emphasis on research that supports the UN SDGs, the United Nations Sustainable Development Goals, for example. Universities are rewriting their tenure and promotion practices so they’re more aligned with institutional values and less aligned with singular citation-based metrics that we were just talking about. And we have organizations like DORA, they’re having these big picture conversations about how to improve our research, how our research is evaluated, so it doesn’t all rely on impact factors. So there’s definitely an evolution happening. It’s maybe slow and steady, maybe slower than I would like it to be, but there is an evolution happening. At Sage, we’re definitely part of these conversations. I mentioned before that we were trying to bring attention to the unique needs of social science. So, for example, data has shown that public social and behavioral science research is cited for longer periods of time in scholarly literature, even though we use this three-year, two-year, three-year-based measure. Actually, in the social and behavioral sciences, research is cited for longer periods of time than that. It also makes outsized impact on policy, and we’ll talk about a new tool later on that can help us with this. But yeah, social science has kind of this unique value, these unique needs, and yet academe uses — or overuses — the same short-term literature-based metrics to decide what is or isn’t a successful science. So we’re trying to talk about things in terms of social science because it’s at our core.

RB: I definitely agree with that, and I think one of the big shifts that I’m seeing happening very recently is just taking best practices from other disciplines who are more used to thinking about this more holistically. So, for example, the humanities has always relied more on qualitative measures than quantitative because their publishing practices are so different, and what is impactful in their field can’t always be described by metrics. Or, say, community-based research, right, where the outcomes really depend on the community. You aren’t necessarily looking for those citation counts. You’re looking for community impact. So they’ve developed frameworks and models for measuring that kind of thing. And so, taking those best practices and spreading them out and understanding these don’t have to be confined to one specific discipline or type of research but can apply really broadly, and especially as we see more interdisciplinary research happening, and no one’s necessarily just doing traditional research within a department or a school. I think we’re seeing broader acceptance of these more holistic measures beyond their original disciplines.

AH: I agree that there have been so many positive developments from including things like policy and patent and even open science, but I still think there are some barriers to moving away from these traditional measures. So, for example, in the United States, there are very few universities who are signatories of DORA. There may be some university libraries or even some departments within universities, but very few universities as a whole have endorsed it. And that’s really problematic, right? One of DORA’s key principles was that the scientific content of a paper was more important than where it was published. However, as I mentioned earlier, universities or many universities are very concerned with their rankings, and ranking agencies and AAU rely on these traditional citation data. So where are they publishing? How many citations are they racking up? And so as long as universities are tied to that, their faculty are also tied to it, and that’s unfortunate. I do know it’s changing, but I wish it were changing more quickly in all places.

CG/ATG: Why is looking at research impact important in today’s cultural and political environment?

RB: So I think higher education has a lot of significant issues right now, but one of the many issues is that we tend to silo our research work, and it excludes the majority of the population. It starts with access to research, which, open access is trying to account for, but it also means delivering information in a way that the general public or other populations, practitioners, community members can understand and use, and more importantly, trust. Ultimately, I see it as talking about return on investment. Where do your tax dollars go? If someone goes to college, how can we trust the information that’s being given to the college students? These are really basic ideas that academia has only recently tried to grapple with, and by and large, it’s still not reflected in the faculty incentive systems, and I include ranking in that, but promotion and tenure has the same problem of relying on those traditional metrics, and that tends to form the basis by which you get tenure, and these other considerations are definitely secondary. So if you’re saying, “I had this great public impact. Everybody uses my work, but it’s not cited. Is that tenureable?” And that’s a question that I think higher education really needs to start taking seriously.

AH: Right, so I mean, I think it’s very clear that a large section of the U.S. public have, you know, they’ve lost confidence in research and the institutions which produce research. Our researchers do good work, but we are not doing a good job of showcasing how that work actually impacts the public as a whole. Moving to a more holistic representation of research impact could really help with this. For example, does the average person at the grocery store care whether or not a paper was published in a journal with a high impact factor? Probably not. They probably don’t know what impact factor is. It’s a very niche topic, and we forget that sometimes, but what they are interested in is how that research impacts their life in a positive way. I think if we talked about that more, it would really bring back some of that trust.

CG: I totally agree with that. Can I just add just one other thing on that, which is to say that I remember back in 2016, which if you can believe it was almost a decade ago, everyone was talking about these new terms called misinformation and disinformation. We were always talking about how things like social media were spreading, you know, bad information, and back then I kind of thought, oh, this is kind of a passing thing. It’s a fad. If we as, you know, higher ed and research librarianship can get together, we can quickly tackle the problems associated with bad information so we can get back on the trajectory of valuing evidence, data, quality research, we’ll be fine. Again, we’re almost a decade later. I’m sad to say the problem of misinformation is at best, normalized, and at worst, deepened, and additionally, of course, as my colleagues were just mentioning, at the federal level, funding and evidence-based policy making are in a kind of precarious state right now. Lots of change is happening. As we all know too well, cuts are happening within science agencies, and some agencies are being told that research that covers certain topics just cannot be funded. So it’s almost as if a new definition of impact, one that ignores real people or evidence, is emerging, which is kind of scary. But there is some good news. You know, I do think that we will get back on the right track. For example, in a recent survey that Sage put out, more than 1,800 researchers, 92% of them said that the ultimate goal of research is to make a positive impact on society. 92% didn’t, they didn’t say the ultimate goal was, you know, academic citations or, you know, anything else. They said it’s for research to make a positive impact on society. So their hearts are in the right place, and I think they do understand that their work can and should make a difference. Over the long term, I do think we’ll get back to putting increased emphasis on the intersection of research and, you know, policy impact, government, real world impacts, and supporting research so that it makes impact is not something that we should give up on.

CG/ATG: We’ve been looking back the last 10 years, now we’re going to kind of shift to looking forward to the future, and Rachel and Andrea, I’m going to ask you this about the librarian perspective on this first. What changes should be made for more effective impact of research, in your opinion?

RB: I think there’s a really interesting opportunity to join conversations that are happening, both within librarians’ individual institutions and more broadly, because we really have a unique perspective and knowledge on this topic. That said, there are relatively few research impact library specialists, and I think that’s a missed opportunity for librarians to help shape these conversations as they happen. And I can say I’ve been doing research impact for about 12, 13 years, and I’ve only had it as part of my job responsibilities for about three. I can count probably on two hands the number of research impact librarians that I know. It’s hard to see where this concentration of expertise is in libraries, and I think that there’s a lot more opportunity to gain that expertise and to label it as such. Looking more broadly, the UK has really effectively organized around research impact, and that’s largely thanks to the fact that there’s this research excellence framework that all UK institutions respond to. They’ve developed initiatives like “More Than Our Rank” that’s been a really good way for expert librarians to help shape the national conversation. For me personally, I’ve been lucky to be recognized as an expert at my university, so I often do have a seat at the table, and that’s enabled me to participate in initiatives like this public impact grant. But more broadly, I think conversations about impact in the United States are happening without librarian input or expertise at the table.

AH: I can definitely affirm what she said, that most of the conversations are not occurring with research impact librarians at the table. I’ve only been in my role for two years, and I definitely do not have a seat at the table. And making progress is difficult, right? I’m having to leverage relationships. I always trade on social capital. I help the faculty that I know with their individual issues, hopefully by word of mouth. They reach out to other people, either in their own department or others. Eventually, departments will come to me for my expertise, and I’m hoping it eventually crawls up the chain. But I will say that my library dean has done a really good job of highlighting the fact that the library does have the expertise to university administration, but it has yet to translate into actual voice.

CG/ATG: Camille, can you give us the publisher perspective on this and give us your thoughts?

CG: Yeah, happy to. First, I want to say thank you for the work that you both are doing as research impact librarians, because you’re right, there aren’t very many of you. And that’s kind of part of the reason why we wanted to have this conversation through the mouthpiece of the Charleston Hub, which, you know, has a great mouthpiece, a very powerful one to the library community. In terms of what publishers can say, I think upholding the DORA’s principles will help us have a more balanced approach to measuring research impact. I didn’t realize until you just said, Andrea, that so few universities were signed on. I think most publishers are, at least most of the bigger ones are. But yes, it’s not just signing it, but actually upholding those principles. For example, when the Journal Impact Factor comes out every year, let’s recognize them for what they actually are, which is a measure for the journal and not a measure of the quality of the individual articles within them, or a measure of the researchers who publish in those journals. If we can have these kind of more balanced, nuanced conversations and not rely or fetishize on one measure, we can improve the measures and we can shift how our research is celebrated and incentivized. I’ll just mention one other thing briefly, which is that Sage is also looking into exploring the research on research impact. So, the scholarly world can, and we believe should, turn its attention to itself to help make improvements. We can use our own scholarly research measures to talk about this and have scholarly conversations that will shed light on what’s happening and where there’s success happening and how we can spread that. I’m excited about where that will take us. It’s early days on this new project we have but watch this space.

Editor’s Note: Thank you very much to Caroline Goldsmith for hosting this interview, and to Rachel Borchardt, Andrea Hebert and Camille Gamboa for speaking with us. Please tune into the podcast at https://www.charleston-hub.com/podcast/atgthepodcast-268-aconversation-with-rachel-borchardt-andrea-hebert-and-camillegamboa-about-research-impact/ to listen to the entire conversation, or watch the video at https://youtu.be/j_9r8igeJX4?feature=shared