We aimed to determine the disagreement in primary cervical screening between four human papillomavirus assays: Hybrid Capture 2, cobas, CLART, and APTIMA. Material from 5,064 SurePath samples of women participating in routine cervical screening in Copenhagen, Denmark, was tested with the four assays. Positive agreement between the assays was measured as the conditional probability that the results of all compared assays were positive given that at least one assay returned a positive result. Of all 5,064 samples, 1,679 (33.2%) tested positive on at least one of the assays. Among these, 41% tested positive on all four. Agreement was lower in women aged ≥30 years (30%, vs. 49% at <30 years), in primary screening samples (29%, vs. 38% in follow-up samples), and in women with concurrent normal cytology (22%, vs. 68% with abnormal cytology). Among primary screening samples from women aged 30–65 years (n = 2,881), 23% tested positive on at least one assay, and 42 to 58% of these showed positive agreement on any compared pair of the assays. While 4% of primary screening samples showed abnormal cytology, 6 to 10% were discordant on any pair of assays. A literature review corroborated our findings of considerable disagreement between human papillomavirus assays. This suggested that the extent of disagreement in primary screening is neither population- nor storage media-specific, leaving assay design differences as the most probable cause. The substantially different selection of women testing positive on the various human papillomavirus assays represents an unexpected challenge for the choice of an assay in primary cervical screening, and for follow up of in particular HPV positive/cytology normal women.

Competing interests: All authors have attended meetings with manufacturers of HPV assays. Matejka Rebolj and her employer received honoraria from Qiagen for lectures on her behalf. Sarah Preisler received honoraria from Hologic/Gen-Probe for lectures. Ditte Møller Ejegod received honoraria from Genomica and Qiagen for lectures, and is project manager on a study funded by BD Diagnostics. Carsten Rygaard has served as an unpaid advisor to Roche. Elsebeth Lynge has served as an unpaid advisor to Hologic/Gen-Probe and Norchip. Jesper Bonde used to serve as a paid advisor to Roche, is a paid advisor to Genomica, and received honoraria from Hologic/Gen-Probe, Roche, Qiagen, Genomica, and BD Diagnostics for lectures. He is principal investigator on a study funded by BD Diagnostics. Hvidovre Hospital holds a recompense agreement with Genomica on a KRAS/BRAF diagnostic system. None of the authors was compensated for their work on this project, holds stock, or received bonuses from any of the manufacturers. According to the contract between the manufacturers, Hvidovre University Hospital, and University of Copenhagen, all four manufacturers had the right to comment on a draft version of this manuscript, but had no editorial rights. This does not alter the authors’ adherence to all the PLOS ONE policies on sharing data and materials.

Funding: Genomica, the manufacturer of the CLART HPV2 Assay; Hologic/Gen-Probe, the manufacturer of the APTIMA HPV Assay; Roche, the manufacturer of the cobas HPV Test; and Qiagen, the manufacturer of the Hybrid Capture 2 Assay, provided tests, instrumentation, and limited co-funding for laboratory material. According to the contract between the manufacturers, Hvidovre University Hospital, and University of Copenhagen, all four manufacturers had the right to comment on a draft version of this manuscript, but had no editorial rights. MR and SP were funded by the Danish Strategic Research Council (grant number: 10-092793; URL: http://fivu.dk/en/research-and-innovation/councils-and-commissions/the-danishcouncil-for-strategic-research ). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. The researchers worked independently of the funders.

The proposed management strategies for women testing positive for human papillomavirus have so far been based on evidence from a small number of assays. Until now, the newly commercially available assays have most often been compared against Hybrid Capture 2 in women with recent cytological abnormalities [4] – [18] . In routine screening, however, women with cytological abnormalities constitute a selected population, whereas a majority of positive human papillomavirus samples are from women without cytological abnormalities. Hence, studies of women with cytological abnormalities cannot capture the diversity of outcomes of human papilomavirus testing in primary screening. Furthermore, the few primary cervical screening studies comparing various assays used relatively crude outcome measures, e.g. kappa coefficients, and suggested good overall agreement [15] , [19] – [24] . However, to determine whether the management strategies for women with positive tests are applicable to other assays, more detailed analyses of outcomes from the various assays are needed. A first step is simply to know whether the same women test positive on different human papillomavirus assays.

Screening for human papillomavirus has better sensitivity for high-grade cervical intraepithelial neoplasia and provides protection against cervical cancer for a longer time than cytology screening [1] – [3] . This was demonstrated in studies using predominantly the Digene Hybrid Capture 2® HPV Test (Qiagen, Gaithersburg, MD), and GP5+/6+ polymerase chain reaction assays. However, several human papillomavirus assays have since become commercially available, and well-documented comparative studies are needed for laboratories to select the most appropriate assay for primary screening.

A positive human papillomavirus test was defined according to the manufacturers’ recommendations (Hybrid Capture 2: relative light unit per cut off value ≥1; cobas channels 16, 18, and other high risk genotypes: critical threshold values ≤40.5, ≤40.0, and ≤40.0, respectively; APTIMA: signal to cut off value ≥0.5). CLART was considered positive if at least one of the 13 human papillomavirus genotypes classified as high risk by the International Agency for Research on Cancer, including genotype 68, was detected [26] . Kappa coefficients were calculated as a standard measure of agreement for each pair of assays; their 95% confidence intervals were calculated by analysing 1,000 bootstrap replications (IBM® SPSS® Statistics, Version 20). The frequencies of positive concordant (positive on assay A/positive on assay B), and of discordant (positive/negative, negative/positive) samples were calculated separately. The sum of the proportions of discordant samples equalled [100% - proportion of overall agreement]. Positive agreement was calculated as the conditional probability that all compared assays were positive (concordant positive samples) given that at least one assay returned a positive result (concordant positive+any discordant samples), and was reported as a proportion. Its 95% confidence interval was calculated assuming binomial distribution of the studied events.

As described above, all women were previously screened with liquid-based cytology, and those with atypical squamous cells of undetermined significance at age 30 years or above were triaged using the Hybrid Capture 2 assay. The screening history of women from 1 January 2000 onwards was retrieved from the Danish Pathology Data Bank. Following Danish recommendations for follow-up of cervical abnormalities, Horizon samples with an earlier diagnosis of cervical cancer, a diagnosis of cervical intraepithelial neoplasia in up to three years earlier, with atypical squamous cells of undetermined significance in the previous 15 months, with more severe cytological abnormalities or a positive human papillomavirus test in the past 12 months were considered follow-up samples. Samples with no recent abnormality were considered primary samples; reflecting routine practice, these included screening samples and a small proportion of samples taken by indication.

1 ml of the diluted sample was aliquoted into an APTIMA Specimen Transfer Tube containing 2.9 ml of buffered solution (Hologic/Gen-Probe). Samples were treated with proteinase K prior to testing, using the Pace 2 Fast Expression Kit containing 1 ml diluent and lyophilized reagent (all from Hologic/Gen-Probe). 100 µl of the reconstituted proteinase K was added to each Specimen Transfer Tube and incubated at 65°C for two hours. The treated specimen tube was stored at 2 to 8°C until testing. Testing was performed on the PANTHER platform.

1 ml of the diluted SurePath sample was spun down (five minutes, 14,000 revolutions per minute), with supernatant removed and cell pellet re-suspended in a mix of 180 µl phosphate buffered saline (10x conc. pH 7.4, Pharmacy product) and 20 µl Proteinase K (recombinant, PCR Grade, Roche Diagnostics, Rotkreuz, Switzerland). Samples were then vortexed and incubated for one hour at 56°C and one hour at 90°C. Human papillomavirus DNA was purified using MagNa Pure LC 96 and MagNA Pure LC 32 instruments (Roche Diagnostics) with MagNA Pure LC Total Nucleic Acid Isolation Kit (Roche Diagnostics). Polymerase chain reaction amplification was performed using CLART® HPV2 Amplification kit (Genomica). 5 µl of purified DNA were used for the polymerase chain reaction amplification. Prior to visualisation, the polymerase chain reaction products were denatured at 95°C for 10 minutes. Visualisation was performed using 10 µl of the denatured polymerase chain reaction products on the CLART microarray. Hybridisation between the amplicons and their specific probes on the microarray resulted in formation of an insoluble precipitate of peroxidase when adding a Streptavidin conjugate that binds to the biotin labelled polymerase chain reaction products. The precipitate was analyzed automatically on the Clinical Array Reader (Genomica).

1 ml of the diluted material was aliquoted into a 13 ml round bottom test tube (Sarstedt, cat. no NC9018280), stored at 2 to 8°C until testing. No pre-treatment of SurePath samples was required. Extraction of DNA was undertaken on cobas x480, and amplification and detection of high risk human papillomavirus DNA on cobas z480 analyzer. Fluorescent TaqMan® probes were used for detection of the amplicons during polymerase chain reaction cycles. Amplification and detection of the 330 bp β-globin was used as an internal control of the testing processes.

On the post-quot material from the cytology procedure, DNA was either denatured prior to testing by pre-treating manually according to the manufacturer’s CE-IVD protocol, or DNA was isolated and purified using the DSP AXpH DNA kit on QIASymphony SP (Qiagen, Hilden, Germany). As part of the cytology processing, post-quot material was diluted approximately 1∶1 in SurePath. Testing was undertaken on automated Rapid Capture® System (Qiagen, Gaithersburg, MD, USA). A minority of samples used for routine Hybrid Capture 2 triage of women aged ≥30 years with atypical squamous cells of undetermined significance were denatured and tested manually.

From the 12,138 routine samples processed during the collection period, 6,258 (52%) were selected for Horizon. For 1,194 (19%) samples, complete human papillomavirus testing could not be undertaken: 1,165 samples were tested only with Hybrid Capture 2 owing to lack of residual material for the other three assays, whereas 29 samples could not be systematically tested on all four assays owing to human error. Consequently, 5,064 (81%) samples with cytology and complete results on the four human papillomavirus assays and cytology were included in the analysis ( Table 2 ). A single sample was available from 5,005 (99%) women, whereas 59 samples (1%) were from the remaining 29 women.

Upon arrival at the laboratory, consecutive samples were collected in racks of 48. They were collected from 10 June to 25 August 2011, equally from Monday to Friday. Approximately 2 ml of residual material were collected after completion of routine SurePath liquid based cytology and Hybrid Capture 2 triage of women aged ≥30 years with atypical squamous cells of undetermined significance. Samples were collected from the first four racks or fewer processed on the collection days. This method mimicked a collection of unselected consecutive samples, assuming that the time of sample arrival in the laboratory was not associated with its characteristics. Samples were diluted with 2 ml of SurePath to obtain enough volume for all four assays. Based on capacity and processing considerations, the target number of samples was set to 5,000.

The Department of Pathology at Hvidovre University Hospital in Copenhagen, accredited by the Joint Commission International, handles all cervical cytology from central Copenhagen. Copenhagen has been covered by an organized cervical screening program since the 1960s. Currently, women aged 23 to 49 years are invited for screening every three years, and women aged 50 to 65 years are invited every five years; in recent years, 76% of women had cytology in the recommended interval [25] .

Virtually all kappa coefficients for pairwise agreement were ≥0.60, suggesting good overall agreement between the four assays ( Table 4 ). Yet, only 52% of primary samples from women aged 30 to 65 years testing positive on either Hybrid Capture 2 or cobas were positive on both. When comparing Hybrid Capture 2 with CLART and APTIMA, these figures were 50% and 58%, respectively. In total, 8.7% of these primary samples were discordant on Hybrid Capture 2 and cobas, 9.2% on Hybrid Capture 2 and CLART, and 5.7% on Hybrid Capture 2 and APTIMA. Discordant samples between cobas and CLART constituted 8.5% of primary samples from women aged 30 to 65 years, 9.7% between cobas and APTIMA, and 10.3% between CLART and APTIMA. Cytology was abnormal in 4.4% of the same primary samples ( Table 5 ).

Overall, 1,679 (33.2%) out of 5,064 samples were positive on at least one of the four human papillomavirus assays ( Table 3 ). Of these 1,679 samples, 681 (41%) were positive on all four, 260 (15%) on three, 268 (16%) on two, and 470 (28%) on a single human papillomavirus assay. Positive agreement between the assays was lower for women aged 30 to 65 compared to women aged 23 to 29 years. Among women aged 30 to 65 years, positive agreement was higher for follow-up than for primary samples; disagreement among primary samples in this age group is presented in more detail on Figure 1 . Among the latter samples, positive agreement was substantially higher in women with atypical squamous cells of undetermined significance or worse compared to women with normal cytology. These patterns remained when the comparison was limited to the three DNA assays.

Among the 5,064 samples included in the analysis, 4,790 (94.6%) were from women targeted by the Danish cervical screening program, aged 23 to 65 years ( Table 2 ). Cytology was abnormal in 371 (7.3%) of the 5,064 samples, cobas was positive in 1,356 (26.8%), CLART in 1,273 (25.1%), Hybrid Capture 2 in 1,035 (20.4%), and APTIMA in 846 (16.7%) samples. These proportions were higher for follow-up than for primary samples for all four assays.

Discussion

Principal Findings Horizon is among the largest studies to compare several human papillomavirus assays in primary screening. Although we found kappa coefficients suggesting a good level of agreement among pairs of the four commercially available human papillomavirus assays, our analysis of positive samples demonstrated substantial disagreement between the assays, particularly in primary screening samples from women aged 30 to 65 years. For all pairwise assay comparisons, there were roughly as many discordant as concordant positive samples. While 4% of these samples showed abnormal cytology, 6 to 10% were positive on one but negative on the other human papillomavirus assay. Our analysis indicates that to fully elucidate the extent of disagreement between human papillomavirus assays, it is necessary to compare them on positive samples. The reason is that, even in a screening population with a high background risk of cervical cancer, a majority of samples test negative, and consequently discordant samples may have little impact on traditional measures such as the kappa coefficient. Similar limitations may apply to the relative sensitivity, relative specificity, and non-inferiority [19] of one assay against another. Our approach relies on the same principle as the calculation of the proportion of overall agreement, which is another commonly reported measure. Unfortunately, it has been rarely used on primary screening data [21], [22], [27], and in those studies attention has not been drawn to the implications of this type of analysis for the management of women with positive human papillomavirus tests. While APTIMA detects E6/E7 mRNA from human papillomavirus infections, the other three assays detect viral DNA. Some disagreement in comparisons between the three DNA assays and APTIMA is therefore not surprising, yet the DNA assays showed more inter-assay disagreement than expected. Possible explanations for this finding include, firstly, that cobas and CLART, but also APTIMA, were run on a fixed volume input from the residual sample material. In contrast, Hybrid Capture 2 was run after re-suspension of the pelleted processed cytology material. Theoretically, the CE-IVD post-cytology processing protocol for Hybrid Capture 2 might have removed some free viral particles prior to human papillomavirus analysis and, consequently, the assay may have returned a lower proportion of samples with a positive human papillomavirus outcome. Whether the clinical performance of the assays was affected will be determined when histological outcomes become available. Secondly, the designs of the assays differ. While Hybrid Capture 2 relies on signal amplification from RNA probes to the entire human papillomavirus genome, cobas and CLART are DNA polymerase chain reaction amplification assays targeting L1 sequences of human papillomavirus genotypes. Thirdly, CLART (by our definition) and Hybrid Capture 2 were designed to detect 13 genotypes, and cobas to detect the same plus genotype 66. Samples positive only for genotype 66 though explain few discordant samples, with 11/190 (9%) of cobas positive/Hybrid Capture 2 negative, and 20/127 (16%) cobas positive/CLART negative primary samples from women aged 30 to 65 years showing infections with genotype 66 and none of the 13 high risk genotypes. Fourthly, assay cross reactivity to low risk genotypes, which could increase the positivity rate, might vary between the DNA assays. These cross reactivity profiles and their significance for discordant samples will be evaluated in a separate report. Fifthly, assay specific calibration of primers/probes for individual human papillomavirus genotypes might result in different analytical sensitivities for detection of infections. Consistent with this, the higher average human papillomavirus viral loads in younger women [28] and in women with dysplasia [29] might increase the positive agreement between assays, as indeed suggested by the patterns observed in our data (Table 3). A possible explanation is that human papillomavirus positive samples in young women or in women with cytological abnormalities have on average higher viral loads than the minimum detectable amount needed to return a positive result even by the comparably least analytically sensitive assays. By the same token, samples with a relatively low viral load might be those that are more susceptible to the set cut-off on any assay before these assays return a positive result. Therefore, a relatively high frequency of agreement between assays in young women and in women with abnormalities observed in our study could be a consequence of the fact that in these women samples with infections but a relatively low viral load are limited in number. In unselected screening samples, on the other hand, viral loads will be much more heterogeneous, representing everything from recent transient infections to high level persistent infections, allowing samples with lower viral loads having a more prominent role in determining the frequency of disagreement between the assays. Within the WHO proficiency testing panel, CLART was evaluated against known genome equivalents of genotype specific L1 plasmids. It detected the 13 high risk human papillomavirus genotype plasmids in copy number 50 to 500 genome equivalents per genotype [30]. For assays like Hybrid Capture 2 and cobas, a similar analysis would not be possible as they return a combined outcome for the targeted genotypes (cobas though also separately for genotypes 16 and 18). To increase the transparency of assay calibration for the targeted human papillomavirus genotypes, a call for international standards of calibration could be suggested.

Strengths and Weaknesses of the Study All samples were evaluated in the same laboratory by the same staff, trained and certified by the assays’ manufacturers, using testing protocols agreed upon prior to the study, and instrumentation and software as supplied by the manufacturers. Unlike in previous studies, samples were collected and stored in SurePath, and experts have called for an evaluation of new human papillomavirus assays using media other than PreservCyt [31]. Previously, 11,617 (primary) SurePath samples from the same area evaluated in the same laboratory were tested with Hybrid Capture 2 [32]. The median age of the women in that study was 36.4 years, and 6% had atypical squamous cells of undetermined significance or worse. The proportion of women aged 25 to 64 years testing positive on Hybrid Capture 2 was ∼17%, similar to the 16% in Horizon in the same subset of samples. Horizon results are therefore in good agreement with the earlier results from the same population. Although 19% of the collected samples had to be discarded, a selection bias is unlikely as excluded samples were similar to included samples. There were no significant differences between the 5,064 included and the 1,194 excluded samples in terms of women’s age, cytology, and Hybrid Capture 2 outcomes, but follow-up samples were slightly more prevalent among the included than the excluded samples (Table 2). Following the Danish routine recommendations, women with abnormal cytology were referred for colposcopy or for repeated testing. In addition, we are currently inviting women with positive human papillomavirus tests and normal cytology for repeated testing (about one year after the baseline test). False positive rates, clinical sensitivity, and clinical specificity of the assays will be reported upon completion of this histological follow-up. The lack of histological verification might appear as a weakness of the current report. However, this is not the case as screening programs have to implement follow-up procedures for all women with positive tests.