Supporting the Changing Research Practices of Public Health Scholars

Ithaka S+R’s Research Support Services Program investigates how the research support needs of scholars vary by discipline. In 2016-2017 Ithaka S+R examined the changing research methods and practices of public health scholars conducting research through U.S. institutions. This project was undertaken collaboratively with research teams at seven academic libraries with the goal of identifying services to better support public health scholars.

This report aims to provide actionable findings for the organizations, institutions, and professionals who support the research process of public health studies. Ninety-three scholars were interviewed during the project, and Ithaka S+R sampled 30 of the resulting transcripts towards the analysis for this report. These transcripts yielded findings in the following thematic areas in which public health would benefit from improved or new services: working with others; information discovery and access; working with data; and outputs, audience, and impact. Within these sections, we identified the following key challenges:

• Workflows in Collaborative Settings. Public health scholars’ work often features cross-institutional and international collaborations. They need workflow tools and infrastructures that can better facilitate their collaborative work.
• Information Discovery and Access. While discoverability of and access to information has generally improved for Public health scholars, they continue to experience barriers to discovering grey literature and accessing peer-reviewed articles written by scholars outside of the U.S.
• Working with Data. Public health scholars experience different challenges based on whether their research incorporates quantitative and/or qualitative data; however, they generally experience difficulties in leveraging new technologies for data storage, management and preservation, and balancing privacy requirements with increasing expectations to share data.
• Outputs, Audiences and Impacts. Public health scholars find the publishing landscape complicated and difficult to navigate, particularly pertaining to impacts and open access. They also seek support in articulating the value of their publishing beyond peer-reviewed journals and showcasing their work dynamically online.

This report concludes with a set of recommendations to better support these scholars’ research and their outputs as they work to improve domestic and global public health.

Introduction

Ithaka S+R’s Research Support Services (RSS) program conducts in-depth qualitative analysis of the research practices and associated support needs of scholars by discipline towards better understanding changing research methods and practices. Our previous projects in the program studied scholars in history, chemistry, art history, religious studies, and agriculture.^[1] A scholar-centered approach to understanding research in higher education is crucial to developing information services and spaces. A sustained approach to studying different disciplines over time also leads to a better understanding of how research activity functions across the academy.

Public health is a compelling field to investigate research support needs because of its salience across a wide range of audiences and industries. The innumerable factors that impact public health make it a highly collaborative and interdisciplinary field and one whose scholars conduct basic and applied research on a vast array of topics. These characteristics have implications for public health scholars across each of their projects’ full research lifecycles. Specifically, they manifest as challenges as scholars search across platforms designed for other disciplines and seek out gray literature, and continue into the kinds of data that they use and the tools to organize it as advancements in technology raise the standards for data collection, analyses, and data management and sharing. With its emphasis on the public, these scholars often produce non-credited outputs to fulfill the terms of their grants and to reach non-academic audiences where their research will be most beneficial, but remain beholden to traditional metrics measuring academic success.

In this report, we explore public health scholars’ research activities and workflows from the ways that they discover and access information, manage and share data, and create and disseminate outputs, to how they conduct these aspects of research when working with others and the key issues that they perceive to affect their research in the present and anticipate for the future. We share our findings and recommendations in order to identify opportunities for the librarians, publishers, organizations, and collaborative and funding agencies that support this research.

Methods

This report is one component of a collaborative research project undertaken with seven institutions. A central component to developing the project was defining “public health” and delimiting meaningful categories within the larger field to ensure a representative sample in the analysis.

Developing the Public Health Project

This project is part of Ithaka S+R’s ongoing program to conduct research on scholarly information practices by discipline through collaboration with other institutions. Participation in the project was open to any U.S. higher education institution with a public health research program that was able to conform to the project specifications (e.g. timeline, research capacity). We included all institutions that met this criteria who expressed interest in participating. The partner libraries created research teams of one to four members who, following a training workshop designed and led by Danielle Cooper, conducted semi-structured interviews with public health scholars at their institution that Ithaka S+R analyzed for this report (see Appendix 2 for the semi-structured interview guide used for this project). Each research team also wrote local reports based on their own data and analysis, six of which were made public (see Appendix 1 for the full list). We thank all of the institutions that participated in this project. A list of participants on the institutional research teams and their local reports can also be found in Appendix 1.

Defining, Delimiting, and Scoping the Public Health Scholar

The report focuses on the practices and needs of public health scholars in higher education. Reflecting the project’s aim to focus on research as opposed to teaching activities, we defined “scholars” as individuals who are employed by their institutions with research as a significant component of their capacity, as opposed to primarily teaching. Graduate students were also not included in this study.

Public health approaches the study of health from the vantage of disease prevention and health promotion at the community level. As public health focuses on the environmental, human behavioral, and lifestyle dimensions of health, the audiences of public health research are diverse and include policy makers, practitioners, and the public-at-large. This multi-disciplinary field spans biological sciences, quantitative sciences, and social sciences, and the research takes place in a variety of contexts ranging from the laboratory to the field and involves both basic and applied types of research. Because of its broad audience, public health research is often conducted in partnership with, or obtains funding from, NGOs or state and federal agencies. Attention was given to ensure the participation of a broad scope of public health scholars, although each of the seven participating institutions varies in how it delineates its public health sub-disciplines. To facilitate our sampling of transcripts, Ithaka S+R standardized these sub-disciplines by analyzing anonymized metadata provided for each interviewee and by reviewing each of the participating institutions’ own departmental definitions. Using these standardized sub-disciplines, described below, we took care to categorize each of the 93 participating scholars according to their main area of research.

Epidemiology and Biostatistics

Epidemiology and biostatistics refer to scholars whose research involves developing and implementing theory and methods to collect data and analyze public health phenomena. These sub-disciplines are methodological in nature and provide the evidence used to identify the causes of illness and to inform health treatments, interventions, and policies. While both look at trends and patterns across populations, biostatisticians affiliated with public health focus solely on how to design and apply statistical methods to research in this field. Epidemiologists more broadly design and apply quantitative and qualitative methods to analyze why diseases and other health disparities exist in some groups and not others. Epidemiologists and biostatisticians lend their support to other sub-disciplines within public health to assist analyses and interpretations of data.

Environmental and Occupational Health

Scholars within environmental and occupational health focus on the ways natural, built, and workplace environments affect community health. In environmental health, scholars study how hazards in the air, water, soil, food, and even in plants, animals, and insects, can be detrimental to good health, while scholars in occupational health study the physical, chemical, and psychosocial hazards that can negatively affect workers’ health and safety.

Community and Behavioral Health

Community and behavioral health refers to scholars who research how to improve the health of individuals and communities through changed behaviors. Scholars in this sub-discipline look into how community practices and individual-level behaviors have an impact on health, as well as the economic, social, and cultural factors that influence health behaviors. Areas of research include, but are not limited to, smoking, nutrition and obesity, maternal and child wellbeing, mental health, sexual health, and mosquito-borne diseases. A large component of community and behavioral health involves promoting healthy habits and developing and evaluating appropriate health interventions.

Health Policy

Health policy refers to those scholars who conduct research into how policies are developed, which individuals, organizations, or other influential factors are involved in their development, and how they are implemented. Scholars within this sub-discipline also study the converse side of health policy through research on how policy can address the underlying social, economic, political, cultural, and legal factors that affect public health, the ethical implications of adopting certain policies, and how the implementation of or changes to policy can impact public health in other ways (e.g. disaster preparedness or road safety).

Health Services and Management

Health services and management refers to scholars whose research focuses on the ease of access individuals or communities have to health services and the quality of care they receive, as well as how healthcare organizations and programs are structured, managed, and sustained. This can include, but is not limited to, researching the best means of storing data (health informatics) and disseminating health information, how healthcare is financed and delivered, how patient care and safety can be measured and improved, the impact technology has on the quality of patient outcomes, and how health and social programs can better work together.

In addition to ensuring appropriate distribution across these sub-disciplines, Ithaka S+R selected transcripts across a range of methodological orientations and experience levels to ensure that scholars with a variety of approaches to the field were included in the sample, and that those differences were considered, where relevant, to identify research support needs and services.

Workflows in Collaborative Settings

Scholars in public health describe the nature of their work as inherently interdisciplinary and collaborative because of the breadth of factors that can affect individuals’ and communities’ health. This breadth is evident in the wide array of disciplines and groups where collaborations can occur, including but not limited to:

• State health departments
• Public health departments within the same institution or other institutions
• Other disciplines (e.g. sociology, education, medicine, chemistry, engineering) within the same institution or other institutions
• Non-governmental organizations (NGOs) or international NGOs (e.g. WHO, UNICEF, Médecin Sans Frontières, the International Rescue Committee)
• Governmental organizations within the U.S. (e.g. CDC, NIH, OSHA) and overseas (e.g. ministries of health)

Scholars will enter into collaborations to bring expertise to their own projects and to lend their expertise to others. They find that collaboration helps ensure that their research is of high quality and addresses the full scope of a given public health topic in areas where they have less familiarity. For example, one scholar said, “Clinicians, they deal with the medical stuff, and I believe whatever they tell me. And I recommend the economic aspect of the study, and they pretty much accept whatever I say.” Another explained in relation to their specific work in health policy, “The quality of ethics work depends on having side by side colleagues who are technical experts. If I say something ridiculous about the way to do preparedness, in the eyes of the preparedness director, it’s sort of dead on arrival.” The projects that public health scholars participate in collaboratively tend to speak to their primary areas of interest and involve aspects to which they can contribute subject-matter expertise; however, collaborations also occur where scholars will contribute their particular skillset to a project that may be outside their primary research focus.

Epidemiologists and biostatisticians in particular mentioned that they delineate more often between the research that they do for others and their own research. This is because these two subfields focus more on developing and testing methodologies, a skillset that is not exclusive to any one discipline or sub-discipline. These scholars are therefore better positioned to provide expertise to a wide array of projects, which may or may not align with the main areas where they conduct their own research. “There’s usually collaborative research, where you do a lot of the analysis, design, etc.,” explained one biostatistician, “and then you have actual research”—referring to their own work—“where you develop methods, prove theorems, and so on so forth.” Another scholar whose focus is epidemiology said, “In [my] area of research I function both as a methodologist and as a principal investigator. But I also collaborate with colleagues outside of those areas…and when I do that I’m primarily just the methods person on their projects.” Despite the important role these scholars play in producing meaningful results, this reliance on others for their methodological and analytical expertise can pose a challenge to scholars’ ability to effectively conduct their research, as discussed in the section “Working with Data.”

Scholars will collaborate more widely to ensure that health interventions are sustained once a study has ended or to implement their applied research on pressing health concerns overseas.

Public health scholars don’t only participate in collaborative research in the U.S.; they also conduct research with others in countries throughout Europe, South and Latin America, Africa, and Asia. In some instances, these collaborations take the form of basic research being conducted on a public health issue outside the U.S. In others, scholars will collaborate more widely to ensure that health interventions are sustained once a study has ended or to implement their applied research on pressing health concerns overseas. These partnerships range from the national government of the country undergoing an intervention to international NGOs and local implementing partners.

The location of an international collaboration can pose a significant challenge to scholars. Scholars who collaborate with partners in Europe reported being able to access data and communicate with their partners with relative ease, while scholars working with partners in less developed areas struggled not only with the limitations of infrastructure, but also with poor resources of their own to conduct research: “With my international partners, they don’t have access to a lot of university platforms and resources in the ways that I do here…I have to find alternative mechanisms outside of the university system to be able to keep the dialogue going.” These comments indicate that international collaborations would benefit greatly from expanded access to the same e-resources and licensed research tools while a project is ongoing.

Scholars also expressed concerns around sharing their data responsibly with collaborators and often preferred to work with individuals they already know and trust. Part of this stems from the sensitive data that is often collected for public health studies, with one scholar saying in regard to the long-standing relationships they have with their collaborators, “A lot of my work is [on] very sensitive ethical issues that we have to be cognizant of, and in a couple of my experiences, trying to establish collaborative working relationships, that has been missed.” Such experiences suggest that scholars would benefit from more standard ways to communicate about data sensitivities and negotiate collaborations accordingly to alleviate reliance on reputation alone. Collaborators working across institutions also face uncertainty over which tools they are permitted to use for data management and storage when they must do so cross-institutionally. As one interviewee highlighted:

It is a challenge for us, particularly in data sharing, because almost all of the systems that we use for our data management are restricted to [institution]-specific individuals. Even in the stage of writing proposals, it’s difficult for us to find consensus between institutions about what systems we can use. Can we use Box? Can we use Dropbox?

These remarks reflect that developing common research workflow and infrastructure across institutions is a major challenge for public health scholars, especially, but perhaps not solely, for sensitive data. While scholars are acutely aware of the sensitive nature of their data and take steps to safeguard it, they reported that there are few systems or standardized guidelines in place that scholars can readily turn to for assistance in sharing and managing data in collaborations.

Information Discovery and Access

Public health scholars use a variety of tactics to discover information, including attending conferences, working with students who are compiling literature reviews for their theses, reviewing others’ work as members of journal peer-review boards, or by drawing on their professional networks to obtain information. This information is often used in the compilation of literature reviews, with the goal to not only enhance their background knowledge on a topic or to inform a grant application or manuscript, but to ensure that they are not duplicating studies that already exist. When actively searching for information, scholars prefer to utilize user-friendly online platforms that facilitate broad searches across other disciplines and gray literature, or platforms that provide them with pertinent information. They struggle with more effectively conducting searches, however, and are unsure about how to determine which information is relevant to their research. Public health scholars also rely extensively on their peer networks to discover, access, and keep up with information in their fields, and stress the importance of speed in obtaining information electronically in a fast-paced academic environment.

Discovering Published Information

When searching for published information, scholars report using a variety of online platforms concurrently (e.g. PubMed, Web of Science, Google Scholar, Scopus, etc.), but will pick and choose based on the functionality each platform provides. Some scholars describe how, due to the interdisciplinary nature of public health, not all platforms are relevant for information discovery when they are highly discipline-specific. For example, one scholar said that “it really depends on the type of audience…so I’ve written papers that are for political science journals and then it doesn’t do me any good to do a PubMed search because the journals that I’m looking for aren’t there.” They also emphasized the capabilities of certain platforms in allowing them to conduct a comprehensive review of published literature. “Where I want to see all the times something was cited and trace forward in terms of, okay here was the original article, what’s happened after that…I’ll use Web of Science for those purposes,” explained one scholar, while another lauded Google search because “to put the things that people cite the most at the top is really helpful because simply doing it chronologically, it can be hard to know if you’ve missed something.” Scholars will also utilize their institutional library’s online systems or go straight to journals that they are familiar with, but given the widespread use of other platforms, the library is not always the first place or the only place that scholars go to search for information.^[2]

User-friendliness was an important determinant in the platforms scholars selected to use—as one scholar explained, “I keep trying to go back to PubMed, but I find PubMed pretty clunky”—as was the ease with which they were able to discover a wider range of information or hone in on a specific article or author. Public health scholars reported that Google and Google Scholar were especially adept at both, with one scholar remarking, “Sometimes Google Scholar is the easiest,” and another saying, “if I’m looking for a paper and I put in a few words, the paper I want comes up in Google Scholar and I can’t make it happen with other things quite as well.” Another described how Google can facilitate interdisciplinary research, saying that “increasingly, I’ve started to use Google Scholar some just because I like the different organization of data, particularly in fields that are less familiar to me;” however, some scholars felt that Google lacked rigor, with one scholar saying, “I remember asking my office mate once…[and] she was like, ‘Oh, I just look up stuff in Google Scholar.’ And that just seemed like such a kind of casual way of doing things…I guess I feel like I have high standards for searches.”

Scholars are uncertain about how to most effectively search for relevant information and broaden their searches beyond the jargon they are familiar with from their own sub-disciplines.

Despite the application of user-friendly platforms, scholars still face challenges in discovering information due to a lack of awareness of certain platforms and skill-level with effective searching techniques. For instance, one scholar said, “I wish I were better at searching…like, you can do a PubMed, you can do ‘gene’, and you can do ‘sequence’, but I don’t have all the skills and honestly I don’t have time to build the skills.” Another similarly expressed that “one of the things I’m concerned about…is the bias in how we choose our research search terms.” These comments suggest that scholars are uncertain about how to most effectively search for relevant information and broaden their searches beyond the jargon they are familiar with from their own sub-disciplines. This same scholar goes on to express concern over “how we sort of keep or eliminate certain types of papers,” suggesting uncertainty over how to narrow down the information that they find to only those pieces that are relevant to their studies. Still others have the opposite issue in their searches for information: “I go to PubMed or Web of Science and dig out some literature and stuff, but I never get enough stuff because I don’t know these other databases.”

While scholars seek ease and breadth simultaneously in their searches, some scholars prefer to use platforms that they are familiar with out of a sense that they are more credible or easier to use.

Because of the original research that scholars must contribute to their fields, as well as the interdisciplinary nature of their work, being able to grasp the full extent of existing literature is crucial and underlines the importance of scholars’ abilities to know of and make use of the appropriate information discovery platforms. The comments in this section reflect that, while scholars seek ease and breadth simultaneously in their searches, some scholars prefer to use platforms that they are familiar with out of a sense that they are more credible or easier to use. This suggests that some scholars may not be willing or prepared to take advantage of evolving search methods that can expedite information discovery because of pre-existing biases or a lack of understanding in how to best use search engines.

Gray Literature

Scholars report relying on published, peer-reviewed literature for much of their information needs, but they also utilize gray literature extensively. The definition of “gray literature” is contested, with one scholar even saying, “It keeps coming up, and I’m actually not even sure what it means,” and another referring to it as “fugitive literature.” Gray literature, as it is understood by the majority of public health scholars and librarians, involves literature that has not been formally published through peer-reviewed means and is not available via scholarly journals. It can include such items as policy briefs, conference proceedings, dissertations, and government reports, to name some of the types of literature that constitute gray literature.

Public health scholars use gray literature to help maximize the breadth of information they are able to obtain for their projects. One scholar whose research is highly specialized noted, “I don’t think we’re going to find much in the published literature, so then we’re going to have to go to the gray literature.” Other scholars have found that “there were fantastic things like on government websites that I would never have found just by looking at PubMed.” Another said that when searching for new projects by scanning through literature, “It might not be in the traditional peer-reviewed literature but in the gray literature where some of these things are emerging or reported.” Other scholars mentioned that because of the lag time between when a study is conducted and published, relying only on published material from scholarly journals for information can cause them to miss out on vital information that could be important for their own studies or alert them to the fact that their study would be a duplication.

Still, scholars and librarians alike express difficulty in finding gray literature because it has not been systematized. After one scholar mentioned their use of this form of literature as a source of information, their interviewer agreed that “it’s much harder to search, especially systematically,” and asked their interviewee, “Do you have any ideas on how the library can help with that sort of less formal type information needs?” Scholars described relying on white papers and their own knowledge to discover gray literature. As one scholar explained, “There are a few gray literature search engines that are relatively good if it’s an academic or a federal kind of thing. Once you get away from that realm of the white paper…it gets much more challenging,” while another had little experience finding gray literature on their own, saying, “You know, my colleague’s the one who really found most of that…I think a lot of it was referenced in a very well-known white paper, and some of it was things we had come across before.”

They also described taking their searches to the broader Internet, rather than relying solely on academic channels like the library or on platforms that cater to peer-reviewed literature. “That’s where Google can be really helpful,” said one scholar in describing their use of platforms like PubMed, Base, or Scopus, “things like policy white papers, reports that are being released by agencies or by NGOs or implementing partners and stuff like that—those aren’t being captured there.” Much like with published literature, however, scholars who make use of gray literature and other forms of non-published or non-peer-reviewed information struggle to identify the most important pieces to search for and use in their research.

Trust and Authority

Public health scholars often lean on their colleagues or professional networks for information discovery. One of the reasons for this is because they are cautious of the quality of literature available to them in a career field that incentivizes publishing as often and as sensationally as possible. Consequently, many scholars prefer to rely on colleagues whose work and reputations they are familiar with. “I know most of the people involved in [my field] at least by scholastic reputation. And so I know the people who I really trust their results always versus those people…well, they published it and it’s not like I don’t trust that they published what they thought was real but I don’t necessarily trust their judgment all the time,” said one scholar. Another elaborated that the resources they use are “either in the capacity of knowing someone, working with that agency”—referring to the organizations or federal agencies many public health scholars collaborate with—“or doing my own research enough to know that this is a totally legitimate source.” As these comments suggest, the emphasis on publishing and impact factor today has created a sense of distrust among scholars towards the credibility of others’ work, leading them to prefer information from individuals with whom they are personally acquainted.

The emphasis on publishing and impact factor today has created a sense of distrust among scholars towards the credibility of others’ work.

Scholars also delegate the task of finding information to others when they have the capacity to do so, whether that person is a colleague, collaborator, or student. For one scholar whose main focus is on methods, they described how information discovery “becomes less relevant for me personally because I’m usually not writing the lit review, right? I’m just the person who has to describe what we did statistically, and the results.” This remark indicates that the need to have background knowledge is not always applicable to all scholars, depending on the project and their role in it.

Others often utilize students or research staff to aid in their searches for information, but a commonly-raised concern was that students lack the skills to effectively find and synthesize information, and furthermore, that scholars have trouble teaching their students these skills. “I haven’t been able to train my students to gather information in a way that helps me. I feel like they often go on the wrong track, so I end up spending a lot of time additionally searching for literature,” said one scholar.

Several others remarked that given the amount of exposure students today have had to social media and technology, they had thought that students would come in with a stronger skillset in information discovery; however, this has not been the case. According to one scholar’s anecdote, in trying to teach their students how to find credible information by setting a minimum of five journal articles for a project, students “just go to websites. And I was just dealing with an email from somebody…and they said, ‘I can’t find five [journal articles]. I’m co-teaching with another colleague…she did a PubMed and wrote back and said, ‘I just found 250 articles.’ I don’t know what they’re not getting!” Because scholars often rely on students to assist them in information discovery, their inability to do so well can impact scholars’ own research, while scholars face the additional challenge of making students more aware of the ways they can find information from credible sources.

Accessing Information

In general, public health scholars report few issues in being able to access information. They are often able to access most of the literature that they need through their institution’s library, including going through their library’s website to obtain articles from platforms like Google Scholar or Scopus. The majority of scholars prefer to access information electronically, with one scholar admitting, “I can’t imagine the last time I’ve actually gone to a library.” They also increasingly expect that they will be able to access information immediately, with many saying that they don’t have the patience for interlibrary loan and find hard-copy materials too inconvenient to be beneficial unless the information is absolutely essential. Scholars in this field also receive a significant amount of funding from federal agencies to conduct their research, with the requirement that their published articles be made open access after a period of time. It may be that because of this, public health scholars have a relatively easy time accessing literature. Their extensive use of gray literature also facilitates access because reports, such as those by the U.S. government or NGOs, are often available at no cost on the Internet.

However, scholars do face some challenges, especially when it comes to accessing literature in highly specialized sub-disciplines and literature written by scholars outside the U.S. Libraries may not subscribe to journals on subjects where there is relatively less demand for that kind of information, and they may not yet subscribe to newer journals that are growing in popularity in certain fields, thereby limiting scholars’ ability to access this information. Varying open access requirements across countries also affects scholars’ access, with one scholar commenting that in a journal that also features European researchers, “not all the papers are supported by American government grants…they don’t end up becoming open access after a year. And so in particular that journal has been a real stumbling block, and more and more people are publishing in it.”

Accessing older information presents another challenge, as “there ends up being plenty of validity in older papers but they can be pretty hard to access” because they have not yet been digitized, as does information on failed projects or methods, known as negative information. To remedy these challenges to an extent, public health scholars will utilize their professional networks for information access as well as discovery, with some scholars even dedicating a portion of their time to building these networks for this purpose. As one scholar said, “For the resources that may not be available through the library and electronically now…the easiest way is to contact the person who did the work.”

Keeping Up through Peer Networks

The tactics used by public health scholars to keep up with information are similar to those employed to discover information (e.g. attending conferences, reviewing journal articles, working with students, and speaking with colleagues). The information that scholars choose to keep up with is often dependent on the areas in which they conduct research and is not necessarily within the realm of public health, given its interdisciplinary nature. Like agriculture scholars, public health scholars report automating how they keep up with literature to an extent, with a number of scholars receiving automatic email updates through platforms like PubMed, notifications from peer networking sites like ResearchGate or LinkedIn when their connections publish new articles, or emails containing a table of contents from specific journals that are pertinent to their research. Despite the widespread use of automatic updates, scholars often find themselves overwhelmed by the amount of information and emails they receive through this method. As one scholar said, “I have thousands of unread emails in my inbox, partly because of all the listservs I subscribe to…while it’s stressful to have lots of unread emails, it’s the only way that I can at least attempt to try and keep up with things.”

Because of the length of time it takes to formally publish a study, scholars who only find and access literature through traditional venues may find themselves unable to stay up to date with the literature in their fields.

Scholars also remarked on the pace of keeping up, with one saying, “I’m keeping up in the sense of I’m two years behind the curve,” alluding to the slow turnaround of studies from their conception to the time that they are disseminated through peer-reviewed journals. To mitigate this, scholars rely more on social media or other media sources to keep up with developments in their field, especially public health scholars whose work focuses on policy. “In the world of policy analysis…we need up to date information about what’s happening in the policy world and the debates going on in the policy arena. So we rely a lot on, I guess you would call them leading daily news services.” Another scholar said that “the Network for Public Health Law has a Facebook page, and they will post our news articles or journal articles that have come out recently on public health law topics and that always helps.” However, other scholars were less willing to rely on social media as a way to keep up, with one scholar expressing uncertainty over whether social media was an appropriate channel for dissemination—“what you’re doing now, and what you’ll be doing in five minutes…we’re still evaluated in ways of how often you disseminate through the regular pathways.” Because of the length of time it takes to formally publish a study, scholars who only find and access literature through traditional venues may find themselves unable to stay up to date with the literature in their fields.

Working with Data

The use of quantitative and qualitative data and analytical methods are dependent on scholars’ particular projects and preferences. All of the sub-disciplines identified in this report utilize both types of methods, and at times incorporate both into mixed methodologies. When gathering data, scholars use existing data sets that they obtain through payment, partnership, or from open access databases, but they will also collect their own data, a process that can entail using literature to build a data set but which more frequently takes scholars into contact with human subjects. Scholars employing qualitative data and methods face challenges to the legitimacy of their research, while scholars using quantitative data and methods often lack the skills to conduct analyses themselves. When considering management and storage, these scholars face challenges in effectively utilizing newer or more sophisticated methods, as well as thinking ahead to how technology may change preservation. They also find it difficult to share data despite funding mandates to do so due to the sensitive nature of the data they often collect.

Qualitative Research and Analytical Methods

Public health scholars’ research focuses on a diverse array of subjects within their own sub-disciplines and within collaborations with other public health colleagues or outside the public health field. Those whose projects warrant the use of qualitative research methods acquire data from a number of primary and secondary sources, including but not limited to:

• In-depth interviews and focus groups
• Community-based participatory research
• Direct observation
• Public health policies and statutes
• Peer-reviewed literature for meta-analyses (i.e. scoping and systematic reviews)
• Social media exchanges

Public health scholars recognize that qualitative research can yield a more detailed narrative around issues compared to quantitative research because of the nature of the methods used to collect data. For instance, one researcher said that “one of the advantages of our work is that when we do ask questions, we have the ability to follow up and say ‘Hey, could you explain that a little bit more or tell me more about that?’ In terms of just being given a survey, [where] you only have a core set of responses that you can choose from.” Despite this appreciation for the deeper understanding of issues that qualitative research can provide, solely using qualitative research methods is rare for public health scholars in a field described as being “very quantitatively oriented.”

This stems in part from a sense among public health scholars that qualitative research is regarded as less legitimate, also owing to its data collection methods. These methods are seen to inherently reduce the validity of data because they generally utilize smaller sample sizes and occur in less controlled environments, and they are also subject to quality assurance issues that allow data to be interpreted subjectively. As one scholar explained, “when you have qualitative analysis…it relies on a human being to make a judgment about what the respondent meant when they said something. We can double-coat [sic] it”—referring to using two scholars to agree on what a respondent meant—“but that is both time consuming and it relies on having good quality people who are well trained.” Another scholar said that “when we do observational studies, we tend to get more into the sophisticated methods [of analysis] to try to overcome the design weaknesses of those studies,” while another mentioned that though they tend to gather subjective, qualitative data, they then try to turn it into objective, quantitative data to strengthen their analyses—“I don’t do much qualitative stuff, I’m suspicious of it.”

For those public health scholars working on qualitative studies, this represents a challenge to getting published in a peer-reviewed journal because “not all journals are very receptive to accepting qualitative studies.” Specific journals have been created in response to this challenge, with one scholar whose work involves community-based participatory research (CBPR) saying that “fortunately enough, because of those challenges, now there are very specific CBPR kind of related journals that we just focus our efforts on putting those work in there;” however, publishing itself requires a certain amount of research to ensure that public health scholars are submitting to the right journals. This is an area that these scholars more generally find to be challenging and will be discussed in greater detail in the section “Output, Audience, and Impact.”

Quantitative Research and Analytical Methods

While the majority of public health scholars employ both qualitative and quantitative research methods depending on the project, scholars whose main area of focus is in epidemiology, biostatistics, or environmental and occupational health tend to utilize quantitative data and analytical methods to a greater extent than other sub-disciplines. Quantitative data types, primary and secondary sources of data, and analytical methods include but are not limited to:

• Geographic data
• Hospital patient data
• Surveys and structured interviews
• Federal agencies (i.e. the EPA, HHS, the U.S. Census, etc.)
• Randomized controlled trials or experiments
• Modeling (i.e. regression, geospatial, structural, etc.)

Despite the ubiquity of quantitative research in the public health field, scholars often expressed discomfort toward conducting quantitative analyses. One scholar described how they take deliberate steps to keep their analyses as simple as possible, saying that “if I can’t analyze lab data with a t-test or ANOVA, I’ve made a mistake in the design essentially.” According to the same interviewee, however, this simplicity is “totally at odds with people in public health.” Complexity appears to be the norm in quantitative analyses, and public health scholars without a strong grasp of quantitative analytical methods struggle to conduct analyses themselves. Additionally, these methods are continuously evolving as analytical tools become more advanced. “The stats methods keep changing,” explained an occupational health scholar, “it’s not intuitive enough that it’s something I want to keep up with. I’d much rather read about theory or results than about statistical methods.”

This statement exemplifies how this discomfort leads many scholars to rely heavily on collaborators or statistically-inclined colleagues to run their analyses for them, including paying others to perform this aspect of research. This emphasis on complex statistical methods and the resulting reliance on others for their analytical skill sets can impact scholars’ ability to contribute meaningful research to their fields. For example, one scholar described the setbacks they experienced because they were unable to find someone with the necessary skills to analyze data, saying that students and staff, “they seem like they’re gonna be able to pull it off and then they can’t. And then you get the right person and suddenly everything comes together. That’s the reality.” Some scholars also find it challenging to interpret complex statistics for themselves and for their audience. “Sometimes, when you’re working with statisticians, they’re like, oh, let’s do something new or create a new method, which is great. But then, how do I explain that to, you know, the random reader of the journal. If it’s super complicated, I can’t. I don’t know if I can understand it myself!” At the same time, public health scholars believe that big data presents an opportunity to conduct more thorough analyses. Given the challenges they face in conducting their own quantitative analyses or finding individuals to conduct these analyses, however, they may be unable to fully take advantage of this data.

Data Management and Storage

When their departments have the infrastructure to support it, scholars will often rely on secure servers provided by their institutions to store data used for analyses, at least while a project is ongoing. As one scholar elaborated, echoing what several other scholars have expressed, “It has to be stored on secure servers. IRBs, of course, insist on it…especially with personally identifiable data.” Scholars will also use tools like Dropbox or Box to store data and to facilitate sharing with colleagues and collaborators, even in lieu of institution-specific storage and management platforms. As one scholar said, “[Dropbox is] easier to use than [institution’s storage]…so, even though we’re told to use [institution’s storage], I usually use Dropbox as much as possible.”

However, they expressed uncertainty around whether Dropbox or Box would be appropriate to use without permission, and in fact, the use of these tools can be impeded by guidelines around personally identifiable information. For example, one scholar said that “we use Box.com because that’s approved by the university for all the different HIPAA and other purposes…I don’t keep any of our primary stuff in Dropbox because that doesn’t meet our institutional requirements.” The technology associated with these cloud-based storage platforms can also be daunting to some scholars, with one interviewee elaborating that while they have used Dropbox, “Not as many people seem to want to use it…sometimes when you engage other people if they haven’t used it they’re not interested in learning. Even something as simple as Google Docs, there are some people who will use it and some people who refuse.”

Management of data and notes occurs more idiosyncratically and is often dependent on scholars’ preferences or specific projects; however, a common theme is that public health scholars tend to gravitate toward familiar methods when in charge of their own data management or portions of it. For example, one scholar mentioned their continued use of paper to collect data from participants, before handing it off to a member of staff to put into a database, while another maintains handwritten notes because of the “forensic integrity” that comes with not being able to accidentally delete paper copies as can happen on electronic devices. Many researchers will also compile their data or conduct analyses in Excel or other widely used tools. As one scholar described when trying to use a new analytical tool, “I wasn’t pleased with it so now we are downloading in Excel and then creating new columns where we add our own codes…sometimes by hand, sometimes in Excel, sometimes in Atlas TI, whatever the research team has familiarity and knowledge with.”

Scholars will also turn to data managers, sometimes written into grants or available through their host institutions, or leave these tasks up to collaborators. As one scholar said, “I am not a data manager…that takes a different skill.” When data managers are not available, grad students are known to take on the task despite a common refrain from public health scholars that students are not adequately prepared to manage information. One scholar elaborated that students lack “a high level of sophistication with the technical side of data management,” and while they are capable of learning, “it does take time to get them started.” These comments, as well as public health scholars’ use of tools they already know, indicate that scholars find it too difficult or time-consuming to learn new technologies on their own, especially when the methods they use are deemed adequate for their purposes, if not especially sophisticated or secure. While these challenges can be mitigated to an extent through the use of personnel with data management expertise, relying on students can pose a challenge when they themselves do not have the necessary technical skills.

Scholars also take a variety of attitudes toward backing up their data and storing data for long periods of time, but struggle with long-term preservation beyond funding mandates that data be deposited in repositories. Back up methods include saving data to a server—required by many IRBs for security—on external hard drives, or in cloud-based storage platforms. While some scholars were not inclined to save their data beyond its use in a project, most expressed a desire to store their electronic data indefinitely because of its relative ease and inexpensiveness. As one scholar said, “The long-term storage costs these days is trivial,” while another said, “Most of my data I would keep forever…if there’s no controls on it, sure, why not have it around forever just in case.” Hard copy data did not receive the same kind of consideration, generally viewed as taking up space or being shredded so that it wouldn’t. For one scholar, “we stuffed [the data] in a filing cabinet, put it on a few tapes, and it’s been moldering there ever since…we’re about to do a purge of our files for lack of space, and I don’t know what’s going to happen.” While this comment was expressed in relation to an outdated form of data storage, scholars working with more current data similarly display little awareness that advances in technology may render their storage methods obsolete in the future.

Sharing Data Responsibly

Public health scholars overall take a varied approach to sharing their data, with some making their data available through websites to disseminate information back to the communities they studied and even to stave off requests for data. Many public health scholars also receive funding from federal agencies like the NIH or CDC, which require that they make their data publicly available upon the completion of a study. To that end, scholars will deposit their data in online repositories to the extent that it can be de-identified; however, they are generally cautious about sharing human subject data because of the management and manipulation involved in ensuring the privacy of participants in their studies. Where sharing is not required, several scholars said that they do not share their data at all because of its sensitive nature, with one scholar saying that “because we are intensively sampling from a small population, we cannot anonymize data. There’s no way to redact it meaningfully,” and another explaining that they are not permitted to deposit their data from studies conducted for the government “because the government wants us to destroy all the data once it is done.”

Scholars also find it challenging to share their data responsibly because they do not want it to be misused by individuals or businesses that may manipulate results for their own political, scholarly, or financial gains, and consequently prefer to maintain some level of control over who has access to their data. For example, one scholar said that they were more inclined to share raw data with scholars who approached them directly rather than merely putting it online for anyone to access. There were especial concerns around making data obtained from vulnerable populations, notably indigenous communities, open access because “their data has been misused in even the recent past.”

Scholars also described other reasons for not sharing data. One does not share their data because their data sets are so small as to not be of particular interest to anyone else, although they are not against putting their data online for long-term preservation. Yet another will make their de-identified data available online, but considers this a secondary consideration because it is not especially valuable to their academic career. In an environment where the more sensational the study the better, one scholar said that “if you create a data set, you might get a pat on the back, but data creation and sharing data is not an activity that’s rewarded.”

But with growing concerns around transparency, many scholars are encouraged to put their data into online repositories like ICPSR, or they are encouraged to turn their data over to their host institution to be made publicly available through their channels. While some scholars do regularly share their data with repositories outside of funding requirements, more often scholars’ remarks ranged from “I need to do a better job at that,” to “It doesn’t occur to me that that’s an option” and “I’m going to plead ignorance, as the excuse” to not using their institution’s repository, to the commonly asked question, “What’s a repository?” These comments demonstrate that, where funding requirements do not mandate data sharing and where sharing de-identified data is less of a concern, public health scholars are often simply unaware of the benefits of data-sharing and their associated schemes implemented by their host institutions or of the broader push by some entities to make data more freely available through external platforms.

Outputs, Audience, and Impact

The type of output that public health scholars produce depends on what kind of impact they wish to achieve with their research, the audience they hope to target, and the terms of their grants. Because scholars operate within a culture of “publish or perish,” their most common output is the peer-reviewed journal article with the aim of being published in the highest-ranking journal and publishing as often as possible. They will also publish in open access journals for their faster turnaround, as well as because federally-funded projects require they make their articles OA. However, they find it challenging to publish in OA journals because of their lower impact factors, high costs, and concerns that these journals are predatory (see “Making Peer-Reviewed Publications Open Access” below for a further description of these scholars’ perceptions of open access journals). Other types of outputs include but are not limited to reports as required by funding agencies, data sets and analytical codes, conference proceedings and papers, policy briefs, fact sheets, and infographics. These outputs are often targeted to specific non-academic audiences; however, they are rarely given the same weight as journal publications and are therefore a challenge for junior scholars trying to establish their careers. The competitive nature of academia and evolving technology also has public health scholars considering how to market themselves more effectively, but they are reluctant to make use of newer internet tools and are uncertain about how to promote themselves appropriately.

Uncertainty over the “Right Journal”

Journal articles are the main output for the majority of public health scholars, with publications in peer-reviewed journals described by one scholar as “the cheese at the end of the maze.” Scholars will also publish codes from their analyses, sometimes in the appendices of their journal articles, but also in online forums where anyone can access them. Their perspectives on open access are further detailed in the following section.

Public health scholars publish in a range of journals, including high impact ones like The Journal of the American Medical Association and The New England Journal of Medicine, broad public health journals like The American Journal of Public Health, or journals that are specific to their sub-disciplines and specializations, such as Health Services Research or Atmospheric Environment. They will also publish in open access journals, to be discussed in greater detail in the next section, as well as leave the decision of where to publish up to collaborators when they are not the primary researcher. As one scholar said, “Publish anywhere as long as you get something that has certain impact factors. And obviously that gives you some liberty in terms of where you publish.”

Despite the widespread practice of targeting audiences over impact, scholars concede that for more junior faculty, publishing in high impact journals is necessary for their careers.

Similar to scholars in other disciplines, public health scholars recognize that impact is a main consideration in where they choose to submit articles for publication. “It is relevant where you publish for our tenure packages,” explained one scholar, “so there is some interest in thinking about publishing in the best possible journals.” However, impact was not the only factor that scholars took into account, with many scholars prioritizing their audiences and viewing impact as an equal or secondary concern. In describing their process of compiling a list of potential journals to submit to, one scholar said, “You’ve got to send it somewhere that cares about it. And then once you’ve got multiple journals where it’s appropriate to go, then you start high and go down.” Another scholar explained in reference to a journal they like to publish in, “It’s not that that’s a great academic journal, but it reaches the right membership, which is an important group to reach with new information.” Despite the widespread practice of targeting audiences over impact, scholars concede that for more junior faculty, publishing in high impact journals is necessary for their careers. “In terms of now that I’m trying to get tenure,” explained one scholar, “maybe I should start trying to publish in fancier journals that none of my peers are going to read.”

For some scholars, publishing in high impact journals presents a challenge because their research was not an appropriate fit for such journals, or because there were no high impact journals in more specialized fields. “If we think that we have an article that would appeal to [JAMA] or The New England Journal of Medicine, we would certainly go there because in both they’re widely read by the policy audience and they are both high impact journals. [But] most of our work doesn’t quite fit what they’re looking for,” said one scholar. Another scholar working in entomology said, “It’s not that I’m not interested in [impact], but no entomology journals have a good impact factor.” This barrier to publishing in high impact journals can be detrimental to scholars’ careers in an environment that prizes impact over other factors; however, scholars also mentioned that in the absence of being able to receive a high impact score for their research, publishing more but in less renowned journals was a desirable alternative.

In order to publish more, speed was also a crucial consideration in where to publish. Scholars expressed frustration at the long amount of time that can elapse between submission and publication because “by the time something gets published, it can be old news already,” and as mentioned in the sub-section “Keeping Up,” this can affect scholars’ ability to have a full grasp on the direction of research in their fields. It can also affect the credit they receive for their research in an environment that rewards being the first to publish on a topic. Consequently, scholars will publish in journals that they know have a relatively faster rate of turnaround—“A journal like Health Affairs that we know publishes quickly and has a tremendous policy audience is a primary target for us”—or submit to open access journals despite their lower impact factor and cost because “they are just places to publish with quick turnarounds.” Although some scholars make their pre-published work available to the public through personal sites or platforms like ArXiv, they were generally skeptical of this method of disseminating their research more quickly. Scholars preferred to maintain some level of control over how widely their preprints could be shared and expressed uncertainty over whether they were permitted to post their work online while it was awaiting peer-review.

Making Peer-Reviewed Publications Open Access

Many public health scholars are required to make their publications open access due to funding mandates from organizations like the Gates Foundation or from federal agencies like the CDC or NIH. “One thing about open access that’s beginning to occur…is some journal editors are agreeing to arrangements with federal funding agencies to have only a very limited time period in which the publication is only accessible to those who subscribe to the journal and then opening it up shortly after that,” explained one scholar. As previously mentioned in the section “Information Discovery and Access,” this arrangement facilitates information access for public health scholars because of the high number of studies that are funded by the government. Where these funding requirements are not present, however, scholars expressed tepid support for publishing in open access journals or making their peer-reviewed publications available to the public through other means.

This sentiment stems in part from the high costs associated with publishing in open access journals. Some scholars did express support for the concept of publishing open access, namely for the speed of publication as previously discussed, or for the ability to reach certain audiences—“if it’s not on PubMed Central, the person I’m wanting to reach might not be able to see it”—but most find the cost prohibitive with a readership too low to justify the expense. As one scholar explained, “If a journal says hey, you can open access this for $1000, nobody I know even reads that. It’s like well, how can you even consider that?” To offset these costs, scholars will often use their own excess funds, or will use funds supplied by their institution’s library to publish in open access journals, but on their own they find it too costly to publish open access. Scholars also find it difficult to discern real open access journals from fraudulent ones, with one scholar saying, “It’s so challenging how many open access publications pop up on a daily basis and how many of them are not valid,” and are generally warned away from publishing in open access journals because of their reputation for accepting literature traditional journals won’t publish.

There are also the scholars who find it challenging to make their peer-reviewed publications more widely available because they have not done so before and are uncertain about how to do so now. As one scholar explained, “It’s simply that I haven’t taken the time to find out,” while another said, “I’m trying to learn more about what open access is and which ones are okay and which ones aren’t…That’s another area I kind of need tutorials, I don’t know so much about that.” Open access journals are not the only means of more widely disseminating peer-reviewed publications, either. Platforms such as PubMed, LinkedIn, or ResearchGate act as venues where scholars can disseminate their peer-reviewed publications for public consumption, but some scholars were unsure about whether they could put their publications on these sites without violating journals’ copyrights. These comments suggest that public health scholars need greater assistance in determining which open access journals are predatory or not, as well as greater awareness of which of these journals are relatively better ranked. Furthermore, they require clear guidelines around the proper use of professional networking sites for disseminating their scholarship more widely.

Reaching Non-Academic Audiences

Public health scholars frequently produce outputs for non-academic audiences, in addition to and sometimes in lieu of their peer-reviewed publications, because they recognize that individuals often do not benefit from information that is solely available in scholarly journals. As one scholar said, “I realize that this is almost career suicide, but rather than impact factor I’m more interested in whether it’s publically available.” Scholars deploy a variety of outputs with the hope that their work will reach those who need it the most, including through local newspapers, blogs, personal and departmental websites, or by targeting the communities where they did their research. “We did a whole feedback booklet for the community of what we found in the survey and used mostly just simple pie charts and bar graphs,” explained one scholar, while another said that for some of their work, it “first appeared in blogs including data and data analysis,” adding that “we’re looking at that as a way of quickly disseminating to a broader audience.”

For other scholars whose research focuses less on individuals and more on policies and structural issues, they will create outputs targeted toward decision-makers and disseminate their research accordingly. For example, one scholar said, “We monitor policy developments and if we see something being debated…and we know that we produced something a year and a half ago that’s relevant to that debate, we’ll disseminate that again.” Others will adapt their outputs to be easily digestible to increase the likelihood that they are reviewed by individuals or groups with the capacity to affect change. “We’re increasingly finding that all of our academic papers, it doesn’t matter what we put out there,” one scholar said in describing why they create non-academic outputs. “Unless we translate them into manageable products, like short briefs, fact sheets, whatever, they’re not going to get used by decision makers.” While these outputs are produced on a voluntary basis, scholars also face mandates to produce reports for their funding agencies, often in conjunction with making their data and the codes used to conduct their analyses available to other scholars or the public.

The choice and effort to disseminate these kinds of outputs is undertaken with the knowledge that they will have little impact on scholars’ careers, which continue to be measured through traditional metrics derived from journals’ impact factors and scholars’ number of publications. Creating these outputs is therefore a challenge for more junior scholars, with one scholar acknowledging that they can focus more on reaching general audiences because “I already have my full tenure and I’m a full professor. I don’t really need to worry about that publishing record.” To be able to achieve both scholarly impact and impact with non-academic audiences, scholars will also trade off the kinds of outputs they produce, as evinced by one scholar saying, “If we think we’ve got a piece of work that we think could appeal to a journal like Health Services Research, we’ll put it there because we’ve already got a policy brief on descriptive stuff that we know is getting to the general audience that we want to target.”

These outputs could be classified as gray literature and are produced by enough scholars that they constitute a key source of information, as discussed in the earlier section “Information Discovery and Access.” But despite the widespread use and output of gray literature, public health scholars not only have a difficult time systematically searching for these outputs, but consequently also receive little credit for them due to a lack of infrastructure to measure how widely they are utilized. Fortunately, some institutions are now beginning to recognize the impact of outputs other than peer-reviewed journal articles. As one scholar mentioned in regard to their institution, “The fact that they now recognize and try to have a depository where you can send products so that if you’re coming up for tenure, non-traditional dissemination products can kind of give a stamp of approval.” However, as these comments suggest, non-peer-reviewed outputs including gray literature, codes, and data still go largely unacknowledged as a real contribution to scholarship.

Self-Promotion

While many scholars disseminate their research with the hope that non-academic audiences will use it toward improving public health, they will also disseminate their research more widely to promote themselves. Scholars in this field describe publicizing their work by giving interviews that may be televised, broadcast over the radio, or written up in web-based newspapers; writing essays for trade journals; drafting press releases; or using social media (e.g. Facebook, Twitter), professional networking sites (e.g. LinkedIn, ResearchGate, etc.), and personal or departmental websites to advertise their research. However, while some scholars do proactively promote themselves, many conduct promotional tasks only when asked by others—“They ask us to do an interview and we will certainly do that. We occasionally get asked to write essays for trade journals and we’ll do that. Anything that helps us”—or on an infrequent basis, as exemplified by one scholar’s remark, “I don’t make it a priority…probably should do more press releases than we do. And probably don’t do as much advertising around our dissemination as we could.”

Similarly with social media and professional networking sites, scholars often expressed that they should make use of these web-based platforms to market their research more broadly, but did not utilize them on a regular basis. “I think it’s a fantastic idea to use LinkedIn or something like that. Well, I did set up a pretty modest site and I haven’t updated it in like 18 months,” said one scholar. Others stated, “I’m not a Facebook kind of guy. I do use ResearchGate and LinkedIn…slightly” and “Not so much social media. I would like to. I haven’t as much.” Part of the reason for this low level of usage is that scholars are reluctant to “brand” themselves and have trouble navigating through the rapid dissemination of information that occurs on these sites. As one scholar remarked, “If everybody is promoting themselves why should I believe what they’re saying at some point? Because they’re under pressure to exaggerate more and more.” They are also uncertain about how best to present themselves to others, with one scholar asking, “Sometimes I don’t think we’re very good at marketing, like how do you market yourself in not an egotistical way?” These comments suggest that public health scholars are aware that there are benefits to promoting their research and, by proxy, themselves. Like scholars in other fields, however, they do so more because they feel pressured to in an increasingly competitive environment, but are ambivalent toward the use of newer tools that do not necessarily promote merit and which some scholars find difficult to effectively use to promote themselves.

Conclusion

The field of public health is complex in its interdisciplinary nature and far-reaching in its potential impact. Both of these aspects affect the ways that scholars working within this field discover and access information, conduct their analyses, and create outputs. Scholars in this field produce an abundance of open access literature through federal funding mandates, as well as gray literature for their funding agencies and in an effort to promote healthier lifestyles and policies. Because of this, they have a relatively easy time accessing both traditional and non-traditional forms of information, although access to gray literature is impeded by a lack of infrastructure to support systematic searches or preservation. Access is further aided by these scholars’ host institutions, whose libraries’ online systems facilitate access to peer-reviewed journal publications that are not accessible elsewhere. However, because of public health’s interdisciplinary nature and its international components, some scholars do face challenges in their ability to access articles written by scholars outside of the U.S., and can have trouble accessing information in journals that their schools do not subscribe to. To mitigate this, scholars describe going through their professional networks to obtain information, a tactic that they also employ to discover and keep up with information in a fast paced and highly competitive environment. They will also search online platforms, but continue to struggle with conducting effective searches using platforms that are not tailored to their discipline.

Both primary and secondary data are employed by these scholars to conduct their analyses. Where scholars don’t gather data themselves, they are often able to access them through their funding agencies and are frequently required to make their data publically available upon completion of a study. Despite an emphasis on quantitative research methods, however, public health scholars, with the exception of biostatisticians, found it challenging to keep up with this set of skills and had to rely on others to conduct complex statistical analyses. Conversely, scholars using mainly qualitative research faced biases against the validity of their research. While appreciation for qualitative research is increasing, these scholars would benefit from greater advocacy, and simultaneously believe that big data represents an opportunity for their research in the future but do not yet have the skills to take advantage of it.

Organizing this data was another challenge, with public health scholars resorting to familiar if unsophisticated methods and lacking consensus across institutions on the best platforms to use to manage and store data. The sensitive nature of their data imposed further restrictions on the platforms they could use, and reduced some scholars’ willingness to relinquish their data to others despite funding mandates and encouragement from their institutions to preserve their data in institutional repositories. Other scholars were simply not aware of data-sharing schemes.

Like scholars in other disciplines, public health scholars are measured by the number and impact of their peer-reviewed publications; however, many scholars in this field consider their work to be pertinent to the public. They subsequently produce outputs meant to reach non-academic audiences despite little incentive from the academy to do so. They also continue to publish in peer-reviewed journals, but chafe at the traditional publishing culture and have devised strategies to ensure that they obtain high impact or a high number of publications while reaching their more specialized academic audiences or the general public. With the increasing competitiveness in this field, scholars also felt that they should market themselves and their research to a greater extent, but rarely took a proactive approach to self-promotion. While these concerns are not unique to this field, public health scholars disproportionately create and utilize gray literature to maximize the breadth of their research and the people they reach with it in a timely manner. These outputs are only now beginning to be recognized as legitimate scholarship, but infrastructure to support these non-traditional outputs has a long way to go.

Recommendations

Workflows in Collaborative Settings

• Permit overseas collaborators full access to their colleague’s library resources for the duration of a project, where infrastructure supports it. Public health scholars conducting research with an international component report having to use workarounds to communicate with their collaborators in less-developed areas, as well as that they do not have access to the same information or resources.
• Develop a standard set of research workflow tools and infrastructure that scholars collaborating across institutions can reference to share, manage, and store data while respecting IRB requirements. Public health scholars often collect sensitive data and have varying requirements around how to organize it and who can access it, leading to confusion over the appropriate tools to use in collaborations.

Information Discovery and Access

• Provide training around discovering information across a variety of platforms and optimizing word searches. Public health scholars and their students do not have specific discovery platforms tailored directly to them because of the interdisciplinary nature of their research. They would benefit from a greater understanding of the platforms that are available to them and the functions that each one provides. They also require assistance to optimize their word searches.
• Aggregate gray literature to facilitate the ease of its discovery and preservation. Public health scholars utilize gray literature to a large extent, but there is no systematic way of searching for it, nor is it systematically preserved.
• Increase access to publications by scholars outside the U.S. and to emerging journals. Public health scholars have relatively few difficulties in accessing information because of open access mandates for federally-funded studies, but struggle to access articles written by scholars outside the U.S. who do not have these grant requirements. This challenge can be compounded when libraries do not subscribe to journals in highly specific fields or to emerging journals.
• Continue preserving older peer-reviewed articles. Public health scholars reported challenges in finding and accessing older articles despite their continued validity to research. Institutions are in the process of digitizing older journals and should continue to do so.

Working with Data

• Help public health scholars identify individuals with the skills to conduct quantitative analyses and strengthen their capacity to do this work themselves. Public health scholars find it challenging to keep up with increasingly complex statistical methods and rely on others for their skills. This can affect the research they are able to produce if they are unable to find the right person to work with their data.
• Assist scholars in interpreting quantitative results. Scholars find it difficult to interpret translate results from complex statistical analyses to translate them to their audience when writing articles.
• Expand the number of IRB-approved cloud-based data management and storage platforms. Depending on the institution, scholars can be limited in the cloud-based platforms (e.g. Box, Dropbox, Google Drive) they are able to use in keeping with IRB requirements.
• Provide training to scholars and students on tools for managing data. Scholars tend to return to tools and methods that they are familiar with, even if there are more sophisticated options available, or will turn to others like data managers or students to manage their data for them; however, scholars often complained that students do not arrive to their programs with strong data management skills and require more training.
• Develop an online platform or software to assist scholars in de-identifying data. Public health scholars were reluctant to share their data because they struggle with the time and effort required to de-identify their human-subject data. Many are required to make their data available to the public to fulfill funding mandates, however, and would benefit from tools that facilitate this process.

Outputs, Audience, and Impact

• Advocate for and create mechanisms for recognizing and rewarding non-peer-reviewed outputs. Public health scholars produce a large number of outputs beyond peer-reviewed publications to fulfill the terms of their grants or to have an impact on public health outside of academia. Some websites or institution-specific systems have been established to reward scholars for these other outputs, but they are few and far between.
• Encourage the sharing of preprints through platforms like ArXiv and provide guidance on what is permitted by journals. Public health scholars often felt that they were not able to keep up with the direction of research in their fields because of slow turnaround in publishing, but were reluctant to engage in preprint culture to a greater extent because they were uncertain what journals would allow and wanted to limit how widely they could be shared.
• Support scholars as they navigate through the open access publishing landscape and continue providing open access publishing funds. Public health scholars were cautious of publishing in open access journals because they had trouble discerning among real open access journals and predatory sites. When they do publish in open access journals, several mentioned that they cover the costs using their institutional library’s funds.
• Develop services and training to help scholars promote their work more widely. Public health scholars felt pressured to market themselves to a greater extent, especially with the advent of social media and professional networking sites, but were not especially proactive about doing so and struggled with how to present themselves.

Appendix 1: Research Teams & Local Reports

Johns Hopkins University

• Team members: Sue Woodson, Claire Twose, Jaime Blanck
• Report URL: http://jhir.library.jhu.edu/handle/1774.2/44566

University of Alabama –Birmingham

• Team members: Kay Hogan Smith
• Report URL: http://guides.library.uab.edu/c.php?g=63707&p=409083

University of Arizona

• Team members: Annabelle Nuñez

University of Illinois – Chicago

• Team members: Rosie Hanneke, Jeanne Link
• Report URL: https://uofi.box.com/s/4lbw7stshdr1nsvaoznox1u54fw5n60v

University of Iowa

• Team members: Janna Lawrence, Chris Childs
• Report URL: http://ir.uiowa.edu/lib_pubs/211/

University of Minnesota – Twin Cities:

• Team members: Shanda Hunt, Caitlin Bakker
• Report URL: https://conservancy.umn.edu/handle/11299/189193

University of Washington

• Team members: Ann Gleason, Sarah Safranek
• Report URL: http://hdl.handle.net/1773/40387

Appendix 2: Semi-Structured Interview Guide

Research focus

Describe your current research focus/projects

How is your research situated within the field of Public Health? [Probe for which sub-discipline(s) their work aligns with and whether they engage in inter-disciplinary work within Public Health and/or with other fields]

Research methods

What research methods do you currently use to conduct your research? [Probe for whether these methods are typical for Public Health scholars)

Do you collaborate with others as part of your research? [If yes, probe for what these collaborations entail, who typically works on them, what the division of work is and how information pertaining to the project’s research is created and stored]

Does your research elicit data? If so:

What kinds of data does your research typically elicit?

How do you incorporate this data in your final research outputs? [Prompt for whether they use data visualization tools]

How do you manage and store this data for your ongoing use?

Beyond the data your research produces, what kinds of information do you rely on to do your research? [Probe for secondary data, primary and secondary sources]

How do you locate this information? [If not explicitly stated, probe for where they locate the information]

How do you manage and store this information for your ongoing use?

Do you experience any challenges working with this kind of information?

Think back to a past or ongoing research project where you faced challenges in the process of conducting the research.

Describe those challenges.

What could have been done to mitigate these challenges?

Are there any other challenges you regularly experience when conducting your research?

How do you keep up with trends in your field more broadly?

Dissemination Practices

Where do you typically publish your research in terms of the kinds of publications and disciplines?

Do you disseminate your research beyond scholarly publications? [If so, probe for where they publish and why they publish in these venues]

How do your publishing practices relate to those typical to your discipline?

Have you ever made your research data, materials or publications available through open access? (e.g. through an institutional repository, open access journal or journal option)

If so, where and what has been your motivations for pursuing open dissemination channels? (i.e. required, for sharing, investment in open access principles)

If no, why not?

Future and State of the Field

What future challenges and opportunities do you see for the broader field of Public Health?

If I gave you a magic wand that could help you with your research and publication process – what would you ask it to do?

Follow-up

Is there anything else about your experiences as a scholar of Public Health and/or the Public Health discipline that you think it is important for me to know that was not covered in the previous questions?

Endnotes

1. Jennifer Rutner and Roger Schonfeld, “Supporting the Changing Research Practices of Historians,” Ithaka S+R, last Modified 7 December 2012, https://doi.org/10.18665/sr.22532; Matthew Long and Roger Schonfeld, “Supporting the Changing Research Practices of Chemists,” Ithaka S+R, last Modified 25 February 2013, https://doi.org/10.18665/sr.22561; Roger Schonfeld and Matthew Long, “Supporting the Changing Research Practices of Art Historians,” Ithaka S+R, last Modified 30 April 2014, https://doi.org/10.18665/sr.22833; Danielle Cooper, Roger C. Schonfeld, Richard Adams, Matthew Baker, Nisa Bakkalbasi, John G. Bales, Rebekah Bedard, et al, “Supporting the Changing Research Practices of Religious Studies Scholars,” Ithaka S+R, last Modified 8 February 2017, https://doi.org/10.18665/sr.294119; Danielle Cooper, Sarah Bankston, Marianne S. Bracke, Beth Callahan, Hui-Fen Chang, Leslie M. Delserone, Florian Diekmann, et al, “Supporting the Changing Research Practices of Agriculture Scholars,” Ithaka S+R, last Modified 7 June 2017, https://doi.org/10.18665/sr.303663. ↑
2. Public health scholars’ use of these platforms and library resources are similar to those of agriculture scholars, who likewise rely on a variety of resources to broaden their coverage of published literature and who do not necessarily go through their library as a first point of information discovery. This latter finding is also consistent with the most recent Ithaka S+R Faculty Survey. See Christine Wolff-Eisenberg, Alisa B. Rod, and Roger C. Schonfeld, “Ithaka S+R US Faculty Survey 2015,” Ithaka S+R, last modified 4 April 2016, https://doi.org/10.18665/sr.277685. ↑