Reproducibility—the ability to duplicate a study’s results using the same materials and methods as the original investigator—is central to the scientific method.1 Study reproducibility establishes confidence in the efficacy of therapies, while results that contradict original findings may lead to overturning previous standards. Herrera-Perez et al. recently evaluated 396 medical reversals in which suboptimal clinical practices were overturned when randomized controlled trials yielded results contrary to current practices.2 Given the evolving nature of evidence-based patient care, studies must be conducted in a way that fosters reproducibility and transparency. Further, materials, protocols, analysis scripts, and patient data must be made available to enable verification.

Efforts supporting reproducibility are becoming more widespread owing to the open science movement. In 2013, the Center for Open Science was established to “increase the openness, integrity, and reproducibility of scientific research”.3 The center sponsored two large-scale reproducibility efforts: a series of 100 replication attempts in psychology and a series of 50 landmark cancer biology study replication attempts. In the first, investigators successfully reproduced only 39% of the original study findings.4 In the second, efforts were halted after only 18 replications because of lack of information and materials from authors, insufficient funding, and insufficient time to perform all the experiments.5 The center also created the Open Science Framework, a repository in which authors may deposit study protocols, participant data, analysis scripts, and other materials needed for study reproduction. More recently, the center created Transparency and Openness Promotion Guidelines, which include eight transparency standards and provides guidance for funders and journals, and initiated the use of badges for journals that adopt reproducible practices.

Current estimates of study reproducibility are alarming. In the biomedical sciences, reproducibility rates may be as low as 25%.6 One survey of 1576 scientists found that 90% of respondents believed science was experiencing a reproducibility crisis; 70% reported not being able to reproduce another investigator’s findings, and more than half reported an inability to reproduce their own findings.7 The picture is even less clear in the clinical sciences. Ioannidis found that of 49 highly cited original research publications, seven were refuted by newer studies, and seven suggested higher efficacy than follow-up results; only 22 were successfully replicated.8 The National Institutes of Health and the National Science Foundation have responded to this crisis by taking measures to ensure that studies funded by tax dollars are more reproducible. However, little is known about the extent to which reproducibility practices are used in clinical research.

In this study, we evaluated reproducible and transparent research practices in the pulmonology literature.11 Our goals were (i) to determine areas of strength and weakness in current use of reproducible and transparent research practices and (ii) to establish a baseline for subsequent investigations of the pulmonology literature.

This observational study employed a cross-sectional design. We used the methodology of Hardwicke et al.,11 with modifications. In reporting this study, we follow the guidelines for meta-epidemiological methodology research9 and the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA).10 This study did not satisfy the regulatory definition for human subjects research as specified in the Code of Federal Regulations and therefore was not subject to institutional review board oversight. We have listed our protocol, materials, and data on Open Science Framework (https://osf.io/n4yh5/).

This study was funded through the 2019 Presidential Research Fellowship Mentor – Mentee Program at Oklahoma State University Center for Health Sciences. The funding source had no role in the study design, collection, analysis, interpretation of the data, writing of the manuscript, or decision to submit for publication.

In this cross-sectional review of pulmonology publications, a substantial majority failed to provide materials, participant data, or analysis scripts. Many were not preregistered and few had an available protocol. Reproducibility has been viewed as an increasingly troublesome area of study methodology.13 Recent attempts at reproducing preclinical14,15 and clinical studies have found that only 25%–61% of studies may be successfully reproduced.6,16 Within the field of critical care medicine, a recent publication found that only 42% of randomized trials contained a reproduction attempt with half of those reporting inconsistent results compared to the original.17 Many factors contribute to limited study reproducibility, including poor (or limited) reporting of study methodology, prevalence of exaggerated statements, and limited training on experimental design in higher education.18 In an effort to limit printed pages and increase readability, journals may request that authors abridge methods sections.19 Here, we briefly comment on selected indicators to present a balanced view of the perspectives of those in favor of reproducibility and transparency and those who resist enacting such changes.

First, data sharing allows for the independent verification of study results or reuse of that data for subsequent analyses. Two sets of principles exist. The first, known as FAIR, outlines mechanisms for findability, accessibility, interoperability, and reusability. FAIR principles are intended to apply to study data as well as the algorithms, tools, and workflows that led to the data. FAIR advocates that data be accessible to the right people, in the right way, and at the right time.20 A second set of principles relate to making data available to the public for access, use, and share without licenses, copyrights, or patents.21 While we advocate for data sharing, we recognize that it is a complex issue. First, the process for making data available for others’ use requires skills. Further, the process, which includes the construction of data dictionaries and data curation, is time consuming. Furthermore, concerns exist with regard to unrestricted access to data facilitating a culture of “research parasites,” a term coined by Drazen and Longo22 that suggests that secondary researchers might exploit primary research data for publication. Drazen and Longo also cautioned that secondary authors might not understand the decisions made when defining parameters of the original investigations. Finally, the sensitive nature of some data causes concern among researchers.

Second, preregistering a study requires authors to provide their preliminary protocol, materials, and analysis plan in a publicly available website. The most common websites used by authors are ClinicalTrials.gov and the International Clinical Trial Registry Platform hosted by the World Health Organization. These registries improve the reliability and transparency of published findings by preventing selective reporting of results, preventing unnecessary duplication of studies, and providing relevant material to patients that may enroll in such trials.23 The Food and Drug Administration (FDA) Amendments Act and the International Committee of Medical Journal Editors (ICMJE) have both required registration of clinical trials prior to initiation of a study.24,25 Selective reporting bias, which includes demoting primary endpoints, omitting endpoints, or upgrading secondary endpoints in favor of statistical significance, may be especially pervasive and problematic. Numerous studies across several fields of medicine have evaluated the extent and magnitude of the problem.26–28 The consequences of selective reporting bias and manipulation of endpoints may compromise clinical decision making. Another issue—p-hacking—occurs when researchers repeatedly analyze study data until they achieve statistically significant results. Preregistration of protocols and statistical analysis plans can be used to fact check published papers to ensure that any alterations made in the interim were made for good reason.

Third, transparency related to study funding and financial conflicts of interest should be emphasized. In a previous study, we found that one-third of the authors of pivotal oncology trials underlying FDA drug approvals failed to adequately disclose personal payments from the drug sponsor.29 Recent news accounts of a prominent breast cancer researcher who failed to disclose financial relationships with pharmaceutical companies in dozens of publications has heightened awareness of the pervasiveness of this issue.30 The ICMJE considers willful nondisclosure of financial interests to be a form of research misconduct.31 It is critical that the public be able to adequately evaluate financial relationships of the authors of the published studies in order to evaluate the likelihood of biased results and conclusions.

Several changes are needed to establish a culture of reproducibility and transparency. First, increased awareness of and training about these issues are needed. The National Institutes of Health has funded researchers to produce training and materials, which are available on the Rigor and Reproducibility Initiative website,32 but more remains to be done. Strong mentorship is necessary to encourage trainees to adopt and incorporate reproducible research practices. Research on mentorship programs has found that trainees who have mentors report greater satisfaction with time allocation at work and increased academic self-efficacy compared with trainees without a mentor.33 Conversely, poor mentorship can reinforce poor research practices among junior researchers, such as altering data to produce positive results or changing how results are reported.34 Other research stakeholders must be involved as well. Although many journals recommend the use of reporting guidelines for various study designs, such as CONSORT and PRISMA, evidence suggests that these guidelines are not followed by authors or enforced by journals.35 When journals enforce adherence to reporting guidelines, the completeness of reporting is improved.36 Detractors of reporting guidelines are concerned that certain checklists (CONSORT, STROBE, STARD) will be used to judge research quality rather than improve writing clarity, that editors and peer reviewers will fail to enforce these guidelines, and that insufficient research exists to evaluate the outcomes from applying these guidelines.37 We analyzed COPD, NPJ Primary Care Respiratory Medicine, and Journal of Thoracic Oncology from our sample as the top three journals for containing reproducibility indicators in their publications. These journals have explicit instructions for authors to provide things such as materials/protocols such that independent researchers may recreate the study or raw data to confirm calculations.38–40 Although reproducibility may be an emerging topic, these recommendations appear to be encouraging authors to include more thorough and complete research.

Our study has both strengths and limitations. We randomly sampled a large number of pulmonology journals containing various types of publications to generalize our findings across the specialty. Our study design also used rigorous training sessions and a standardized protocol to increase the reliability of our results. In particular, our data extraction process, which involved blinded and duplicate extraction by two investigators, is the gold standard systematic review methodology and is recommended by the Cochrane Collaboration.41 We have made all study materials available for public review to enhance the reproducibility of this study. Regarding limitations, our inclusion criteria for journals (i.e., published in English and MEDLINE indexed) potentially removed journals that contained more lax recommendations regarding indicators of reproducibility and transparency. Furthermore, although we obtained a random sample of publications for analysis, our sample may not have been representative of all pulmonology publications. Our results should be interpreted in light of these strengths and limitations.

In conclusion, our study of the pulmonology literature found that reproducible and transparent research practices are not being incorporated into research. Sharing of study artifacts, in particular, needs improvement. The pulmonology research community should seek to establish norms of reproducible and transparent research practices.

DJT, MV: Substantial contributions to the conception and design of the work. CAS, JN, DJT: Acquisition, analysis, and interpretation of data for the work. CAS, JN, DJT, TEH, JP, KC, MV: Drafted the work and revised it critically for important intellectual content. MV: Final approval of the version submitted for publication. CAS: Accountability for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

The authors have no conflicts of interest to declare.