Our study shows that the number of protocols published in ‘BMC Systematic Reviews’ has increased tremendously since the journal’s launch, from n = 21 in 2012 to n = 145 in 2017 (overall n = 544). The proportion of protocols requiring a major revision was 37.9% and has also increased with time. Three quarters of reviewer comments suggested more transparency. Most protocols were also registered with PROSPERO (89.2%), an increasing proportion before the manuscript had been submitted (reaching 94.0% in 2017).

While the time from acceptance to publication was constantly about two weeks over the years, the time from submission to publication almost doubled and went from just under three months in 2012 to over five months in 2017. This is how long it takes for a primary study from first submission to acceptance [20]. Since it takes about 1.3 years on average to conduct and publish an entire SR on medical interventions [21], the time it takes to publish a SR protocol seems to be disproportionately high.

From an author’s perspective, long processing times might have far-reaching implications. Although authors should wait until their protocol is accepted for publication, they might choose to continue working on their SR in the meanwhile. This holds the risk that valuable comments from the peer review might not lead to changes in the SR’s methodology, which might be critical unless the part requiring a revision has not been commenced or finished by then. For example, a revision of the eligibility criteria might result in a new search strategy.

Waiting for the protocol to be accepted causes a delay in the conduct of the SR and consequently in the dissemination of new knowledge [14]. Moreover, it is likely to complicate budget and project planning for authors of SRs for several reasons. People who have worked on the protocol might change or the project may simply run out of money due to the long time that publication of the protocol takes. This in turn might entail that the corresponding SR will never be published [22]. In another study, we found that about one-third of SRs remains unpublished 3–5 years after the protocol has been available [9], thus it remains unclear whether submitted but unpublished protocols exist.

The most likely reason for the increased duration from submission to publication is the increase in the proportion of protocols that required major revisions. As we have shown, protocols requiring a major revision after the first round of peer review took about a month longer (median 29 days) to get published than those requiring a minor revision. But why did more protocols require a major revision?

One explanation might be that the quality of initially submitted manuscripts for SR protocols became slightly worse over time, perhaps due to a change in the authors who submit SR protocols. One can assume that publishing protocols has become more popular over the years, while in the beginning only groups which are experienced in performing SRs submitted protocols. This is supported by the fact that the geographical scope in PROSPERO [10], as well as in our study (Additional file 2) has changed with time. Other aspects, such as the number of reviewers/revisions or the time from acceptance to publication, did not follow a clear trend over time in our study, although we found a large increase in the number of reviewers between 2016 and 2017. This suggests that the peer review policy may have been changed during recent years.

Another explanation why more and more protocols required a major revision might be the introduction of PRISMA-P in 2015. Nowadays, it is required by ‘BMC Systematic Reviews’ that the authors follow the checklist and submit it as an additional file [18]. Otherwise their submission will be returned as incomplete, which of course delays the peer review process. Furthermore, peer reviewers may have become more critical as it is likely that the 17-item checklist goes beyond what they had previously looked for. This would also explain why most reviewer comments suggest more transparency.

Because the time from submission to first peer review has also steadily increased from 2015 onwards, another explanation might be the increasing difficulties in finding appropriate peer reviewers given the increase in scientific publications in general [23]. However, it is important to note that there is not only a high burden on peer reviewers, but also on all editors (in-chief, associate, handling) [24].

About 9 out of 10 protocols were also registered in PROSPERO. It is known, that SRs with published protocols are more often registered with PROSPERO than SRs without [9]. However, an interesting finding was that the majority (87.4%) of SRs was registered with PROSPERO before the protocol had been submitted. Paradoxically, PROSPERO urges caution not to register too early and that the review protocol should be complete before submitting the registration request [25]. But this is just the case after peer review is completed. PRISMA-P recommends that, if the SR is registered in a publicly accessible registry, the name of the registry and registration number should be included in the protocol. Although these statements are not contradicting, they do not provide clear instructions for authors of SR protocols.

One major argument for publishing a protocol of a SR is to receive input regarding the review scope and review methodology from independent peer reviewers, who themselves are experienced researchers. However, in a recent study we found that SRs with published protocols had older searches compared to SRs without a published protocol; in 52.2% of the SRs, the final search had already been performed before submitting the protocol for publication [9]. This fits to the finding of this study, that authors only implemented protocol amendments regarding the search strategy in 56.3%, while overall protocol amendments were implemented in 73.7%.

Naturally, authors play an important role in the peer review process, too. We found that most of the reviewer’s comments on methodological issues suggested more transparency, not amendments to the protocol. The former could be avoided to some extent if authors paid more attention to this matter in the preparation of their manuscript, and described and justified their methods more carefully in the initial submission.

One could argue that for protocols, which did not require any amendments, the registration in PROSPERO would have been sufficient. That is because all potential advantages of publishing SR protocols, apart from peer review, could also be achieved through registering the SR with PROSPERO (provided that information is presented in the same amount of detail as they would have been in a protocol). Furthermore, a recent study found that registered reviews were of higher quality than non-registered reviews [26]. However, in practice it would turn out to be very difficult to identify protocols that will benefit most from a peer review a priori. In a random sample of 300 reviews published in 2014, only 4% mentioned that the SR has been registered [27]. However, another study found that, in a random sample of 150 SRs published in 2015, 19% all reviews actually were registered [28].

An advantage of PROSPERO is that the registration and status of the SR can be easily updated and details or the reference for the published review can be added. If the SR has never been completed, another possibility is to switch its status to ‘abandoned’ including details on reasons for that, although this is seldom done [29]. Since any previous versions of an entry are kept, amendments can be made transparent. This is not feasible with published protocols and authors therefore need to report all protocol amendments in the actual review. Despite these useful features, we feel that PROSPERO could be further optimised; the structure of PROSPERO records should be aligned with the PRIMSA-P checklist to facilitate registration for authors of SR protocols.

Strengths and limitations

This was a study of all SR protocols published in ‘BMC Systematic Reviews’ since it was launched in 2012 up to and including 2017. Despite the large number of analysed protocols (n = 544), the journal only served an example. However, it is the open peer review journal, in which most SR protocols are published (39.7%), followed by BMJ Open (23.4%) [9]. It is also important to notice that our data only come from published protocols and the associated reviewer reports and author comments. Thus, it remains unclear how protocols have been dealt with that did not get published.

Although most endpoints of this study were based on data like dates and frequencies, we had to make judgements regarding some endpoints. To be as objective as possible, we did not check back with any peer reviewers, protocol authors or the editors, which means that our judgements might vary from the original ones. To increase internal validity, the reviewer reports for each protocol were assessed by two researchers independently regarding their extent of revision, though.

We used PRISMA-P as a simplified method of content analysis and we might have misclassified some comments. Moreover, the 10%-sample of protocols might be too small for meaningful analysis. However, we had the opportunity to include a total of 450 reviewer comments. Lastly, we did not analyse whether changes in PROSPERO were made after the protocol had been finally approved. Status updates are not often done in PROSPERO [9], though, so their extent should be small.