Our index showed serious deficiencies and variability in the availability of HPV vaccine studies and data. For example, only half of the completed studies listed on ClinicalTrials.gov posted their results. The clinical study reports we obtained via our index process confirmed that the amount of information and data are vastly greater than that in journal publications. For example, the journal publication for one GlaxoSmithKline Cervarix study (HPV-008) is 14 pages long [28] while its publicly available corresponding clinical study report is more than 7000 pages long [29], even though it is a shortened interim report.

Identification of some studies involved a considerable amount of work and a fifth of the index could not be cross-verified. We indexed studies that we would not have been able to index if we only relied on the journal publication databases. For example, one Cervarix randomized clinical trial (HPV-002) was only listed on GlaxoSmithKline’s trial register. The index that GlaxoSmithKline provided us with contained three Cervarix meta-analyses that were only identifiable by their GSK ID (i.e., 205206, 207644, and 205639), and the index did not include three randomized clinical trials (HPV-009 and HPV-016 and the prematurely terminated HPV-078) that were listed on ClinicalTrials.gov. One Cervarix trial (HPV-049) and three Gardasil studies (V501–001, V501–002, and V501–004) were only identified via regulatory registers or correspondence. One Gardasil follow-up study (for V501–005) only had a journal publication listed in PubMed without any manufacturer-specific ID or registration. Four non-industry studies were published in journal publications but were not registered in any of the 45 trial registers. Three non-industry studies were only listed on regional trial registers (Australia: https://www.anzctr.org.au/; Germany: https://www.drks.de/; and India: http://ctri.nic.in/) (see Additional files 2 and 3).

We also indexed more studies than those listed in the holdings of major regulators. For example, EMA conducted a review (of the relation between HPV vaccination and two syndromes: postural orthostatic tachycardia syndrome, POTS and complex regional pain syndrome, CRPS) [30] that EMA believed was based on the manufacturers’ complete HPV vaccine study programmes (“GSK [GlaxoSmithKline] has conducted a review of all available data from clinical studies…with Cervarix”; and “The MAH [Market Authorisation Holder, i.e., Merck Sharp and Dohme] has reviewed data from all clinical studies of the qHPV vaccine [Gardasil]” (30,31)). However, when we compared the manufacturers’ study programmes (submitted to EMA, see Additional file 1) with our index (see Table 1 and Additional file 3), we found that only half (38/79, 48%) of the manufacturers’ randomized clinical trials and follow-ups of Cervarix and Gardasil completed before the submission dates in July 2015 were included (EMA’s review did not assess Gardasil 9) [30, 31]. Similarly, the FDA’s Drug Approval Packages (DAPs) only mentioned half (32/60, 53%) of the randomized clinical trials and follow-ups that were completed before the vaccines’ date of the Drug Approval Packages (Cervarix: 17/36, 47%; Gardasil: 6/11, 54%; and Gardasil 9: 9/13, 69%). We find this very disturbing.

To our knowledge, our study is the first with the aim of indexing a complete study programme. We do not know if the considerable reporting bias we found is generalizable to all drugs and vaccines, but similar industry examples exist for oseltamivir [15], rofecoxib [32], and rosiglitazone where 83% (35/42) of the study programme was unpublished [33, 34]. Indexing is important when there is high risk of reporting bias, which often is the case for industry funded drug trials. Our approach should therefore be considered for systematic reviews of drugs and vaccines. Steps 2 and 4 contributed quantitatively the most to the identification and cross-verification of studies—in particular, searches on ClinicalTrials.gov and the HPV vaccine manufacturers’ trial registers. Searches of regulatory registers and journal publication databases contributed to a lesser extent. Steps 1, 3, 5 and 6 contributed mainly to the verification of some studies (see Additional files 1 and 3).

Our six-step process is reproducible, the step sequence is interchangeable and most steps could be performed simultaneously. For example, we started by corresponding with EMA, since we are familiar with EMA’s handling of study programmes and clinical study reports [27]. This correspondence helped us get started (but EMA response times may prove very slow and EMA often denies data requests (18)). The index took approximately 3 months to assemble. Researchers may save time if they perform the steps simultaneously and focus on steps 2 and 6. For example, researchers could start requesting the drug manufacturers’ study programmes and subsequently make an independent index and compare the two. However, correspondence with manufacturers may prove challenging and slow. Only one of the four HPV vaccine manufacturers (GlaxoSmithKline) provided us with study programme information, which we received 9 months after our initial request. Merck Sharp and Dohme responded to our enquiry, but did not provide study programme information. Shanghai Zerun Biotechnology and Xiamen Innovax Biotech did not respond to our inquiries (Additional files 1, 2, 3, and 5).

Compared to industry studies, non-industry funded studies were registered less often (for example, on ClinicalTrials.gov) and posted less study results. Non-industry researchers are not legally required to register their studies, adhere to industry reporting guidelines (the International Conference on Harmonization of Technical Requirements for Registration of Pharmaceuticals for Human Use, ICH: http://www.ich.org/) or produce clinical study reports unless their results are used to support a drug’s marketing authorization application. This involves a high risk of reporting bias. Therefore, we recommend that non-industry funders require researchers to register their studies and commit to reporting guidelines similar to the ICH.

Finally, although study programme and clinical study report access from the industry and regulators have improved since 2010 [17], access is often slow and inefficient [18]. In May 2014, one of us (TJ) requested the HPV vaccine clinical study reports from EMA. The request was initially declined by EMA because, “disclosure would undermine the protection of commercial interests”. TJ successfully appealed, but EMA has only released 18 incomplete clinical study reports more than 3 years after the initial request (as of 1 July 2017), which is only half of the clinical study reports included in the EMA review (18/38) [30] and a fifth of our indexed randomized clinical industry trials (18/96).