This observational, cross-sectional study used the methodology proposed by Hardwicke et al [3], with modifications. We reported this study in accordance with the guidelines for meta-epidemiological methodology research [19] and, when pertinent, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [20]. Our study did not use any human subjects or patient data and, as such, was not required to be approved by an institutional review board prior to initiation. We have used The Open Science Framework to host our protocol, materials, training video, and study data in a publically available database (https://osf.io/n4yh5/). This study was part of a comprehensive investigation on reproducibility across multiple clinical specialties.

Journal and publication selection

On June 25, 2019, one investigator (D.T.) searched the National Library of Medicine (NLM) catalog for all journals using the subject terms tag “Neurology [ST].” The inclusion criteria required that all journals publish English, full-text manuscripts and be indexed in the MEDLINE database. The final list of included journals was created by extracting the electronic international standard serial number (ISSN) or the linking ISSN, if necessary. PubMed was searched with the list of journal ISSNs on June 25, 2019, to identify all publications. We then limited our publication sample to those between January 1, 2014, and December 31, 2018. Four hundred publications within the time period were randomly sampled for data extraction. The rest were available, but not needed (https://osf.io/wvkgc/).

To estimate the required sample size for our study, we used Open Epi 3.0 (openepi.com). We selected data availability as our primary outcome based on its importance for study [3]. Our estimated parameters included a population size of 223,932 publications; a hypothesized % frequency of 18.5% for the data availability factor in the population (which was based upon data obtained by Hardwicke et al.); a confidence limit of 5%; and a design factor of 1, which is used in random sampling. Based upon these considerations, a 95% confidence level would require a sample size of 232. From our previous studies [21, 22], we estimated that approximately 40% of studies would be excluded following screening. Thus, a random sample of 400 publications with a hypothesized attrition rate of 40% would yield a final, minimum sample of 240 for analysis. Previous investigations, upon which this study is based, have included random samples of 250 publications in the social sciences and 150 publications in the biomedical sciences. Thus, our sample size exceeds those used in previous investigations.

Extraction training

Prior to data extraction, two investigators (S.R. and J.P.) completed in-person training designed and led by another investigator (D.T.). The training sessions included reviewing the protocol, study design, data extraction form, and likely locations of necessary information within example publications. The two authors being trained received two sample publications to extract data from. This example data extraction was performed in the same duplicate and blinded fashion used for data acquisition for this study. The two investigators then met to reconcile any discrepancies. After the two sample publications were completed, the investigators extracted data and reconciled differences from the first 10 of the included 400 neurology publications. This process insured interrater reliability prior to analyzing the remaining 390 publications. A final reconciliation meeting was conducted, with a third investigator (D.T.) available for disputes but not needed.

Data extraction

After completing the training, the same two investigators extracted the data from the included list of randomly sampled publications between June 3, 2019, and June 10, 2019, using a pilot-tested Google form. This Google form was based on the one used by Hardwicke et al., but including modifications [3]. We specified the 5-year impact factor and that for the most recent year as opposed to the impact factor of a specific year. The available types of study designs were expanded to include case series, cohort studies, secondary analyses, chart reviews, and cross-sectional analyses. Last, we specified funding sources, such as hospital, private/industry, non-profit, university, or mixed, instead of restricting the criteria to public or private.

Assessment of reproducibility and transparency characteristics

This study used the methodology by Hardwicke et al. [3] for analyses of transparency and reproducibility of research, with modifications. Full publications were examined for funding disclosures, conflicts of interest, available materials, data, protocols, and analysis scripts. Publications were coded to fit two criteria: those with and those without empirical data. Publications without empirical data (e.g., editorials, reviews, news, simulations, or commentaries without reanalysis) were analyzed for conflict of interest statements, open access, and funding. Given that protocols, data sets, and reproducibility were not relevant, these were omitted. Case studies and case series were listed as empirical studies; however, questions pertaining to the availability of materials, data, protocol, and registration were excluded due to previous study recommendations [18]. Data extraction criteria for each study design are outlined in Table 1.

Table 1 Reproducibility-related characteristics. Variable numbers (N) are dependent upon study design. Full detailed protocol pertaining to our measured variables is available online (https://osf.io/x24n3/) Full size table

Publication citations included in research synthesis and replication

For both empirical and nonempirical studies, we measured the impact factor of each journal by searching for the publication title on the Web of Science (https://webofknowledge.com). For empirical studies, we used the Web of Science to determine whether our sample of studies was cited in either a meta-analysis, systematic review, or a replication study. The Web of Science provided access to studies that cited the queried publication and provided the title, abstract, and link to the full-text article. This permitted the evaluation of the inclusion of the queried article in data synthesis. Extraction was performed by both investigators in a duplicate, blinded fashion.

Assessment of open access

Important core components of publications necessary for reproducibility are only available within the full text of a manuscript. To determine the public’s access to each publication’s full text, we systematically searched the Open Access Button (https://openaccessbutton.org), Google, and PubMed. First, we searched the title and DOI using the Open Access Button to determine if the publication was available for public access. If this search returned no results or had an error, then we searched the publication title on Google or PubMed and reviewed the journal website to determine if the publication was available without a paywall.

Statistical analysis

Microsoft Excel was used to report statistics for each category of our analysis. In particular, we used Excel functions to calculate our study characteristics, results, and 95% confidence intervals.