Skip to main content

Maintaining relevance in HIV systematic reviews: an evaluation of Cochrane reviews

Abstract

Background

Research turnover in the HIV field is rapid, and as a result, maintaining high-quality, up-to-date, and relevant systematic reviews is a challenge. One approach is to frequently update published reviews.

Methods

We evaluated the methods and relevance of all HIV systematic reviews and protocols published in the Cochrane Library over a 16-year period (2000ā€“2016) to determine the need to update published reviews or complete of reviews in progress.

Results

Of 148 published reviews and protocols, 129 (87%) were identified as not for updating or progression to publication, mostly due to research questions which were either entirely outdated or addressed questions in an outdated manner (Nā€‰=ā€‰89; 60%); this was anticipated for older reviews, but was found also to be the case for recent publications. Some research questions were also inadequately conceptualized, particularly when complex pragmatic trials or behavioral interventions were included.

Conclusions

We suggest that authors clearly characterize interventions and synthesis approaches in their review protocols. In research fields, such as HIV, where questions change frequently, systematic reviews and protocols should be regularly re-evaluated to ensure relevance to current questions. This process of re-evaluation should be incorporated into the methods of living systematic reviews.

Peer Review reports

Background

The HIV field has high research outputs, and the cycle of questions, trials, and guideline modification is rapid. This raises the question of how to keep HIV systematic reviews up to date, relevant, and derived from the highest methodological standards [1, 2]. One approach is to regularly update reviews to incorporate new findings; and indeed, some questions endure and should be updated; and for others however, the review question may have been answered, changed substantially, or become irrelevant, and blindly updating such reviews can result in wasted effort and resources. To provide insight into how review questions change over time, we formally appraised the relevance, methods, and need to update a suite of HIV systematic reviews and protocols published in the Cochrane Library between 2000 and 2016.

Methods

Between 1 May 2016 and 30 May 2018, we critically appraised all published Cochrane HIV reviews and protocols based on the Cochrane Updating Classification System [3, 4]. The Updating Classification System acts as a guide to readers to assess whether a review is up to date, needs updating, or might need updating in the future [3, 4]. We modified the tool for assessing protocols. Reviews and protocols were appraised by evaluating: (a) the currency and priority of the review question (through discussion with experts in the field and exploring the literature), (b) the methods, by using an abbreviated version of the MECIR standards [2], and (c) whether there were new studies to be included in the review, or whether another high-quality review had already been published on the topic outside of Cochrane. Based on the outcome of this assessment, we determined whether reviews should be updated, and whether protocols should continue to full reviews. We categorized them as follows: (1) continue with review update or allow protocol to progress to full review and (2) not for update or protocol progression due to (a) methodological concerns, (b) outdated research question, or (c) review already up to date. At least two people read and appraised each review independently, and the classification was agreed in team meetings chaired by experienced systematic review authors. The tool is available in the AdditionalĀ fileĀ 1.

Results

We identified and appraised 109 Cochrane HIV reviews and 39 HIV protocols (148 articles in total), published on the Cochrane Database of Systematic Reviews between 2000 and 2016. Review and protocol topics included questions about HIV treatment interventions (63%), HIV prevention interventions (32%), health system interventions (2%), and interventions to improve HIV diagnosis (3%). The majority of reviews (Nā€‰=ā€‰98; 66%) were published between 2010 and 2016, with a smaller proportion published between 2000 and 2009 (Nā€‰=ā€‰50; 34%). We concluded that 19 (13%) of all reviews and protocols were eligible for an update or progression to full review, and the remaining 129 (87%) were assessed as ā€œnot for updating or progressionā€.

Of the 109 published systematic reviews, 95 were assessed as ā€œnot for updatingā€ (87%). Of these, 62% had outdated research questions (Fig.Ā 1) where the intervention was superseded by a superior intervention or by another more up-to-date review, or where questions were already answered, and there was little to no ongoing research in the topic area. For example, a review of the ā€œEffectiveness of first-line tenofovir, emtricitabine and efavirenz for patients with HIVā€ would not require updating as this question has been answered. Another example is the use of HSV-2 suppressive therapy for HIV prevention: this is no longer considered an option for HIV prevention; although there may be some additional studies for inclusion, updating such a review would be of little value to consumers or policy makers. We discontinued 11 reviews due to methodological errors (37%): this was the result of inadequately conceptualized research questions (the population, interventions, comparison, or outcomes were not sufficiently characterized or relevant). Sometimes, this led to ā€œemptyā€ reviews (with no included studies after searches were conducted). Another methodological problem was review questions which were not unique to HIV populations, for example, reviews evaluating drug effects in HIV subpopulations with no clear rationale for why these agents would have different outcomes in HIV-positive people.

Fig. 1
figure 1

Outcomes of review and protocol appraisals

Thirty-four of 39 protocols were stopped from progressing to full reviews (87%) (Fig.Ā 1). The two main reasons for discontinuation also included outdated research questions. This included protocols where the topic was no longer relevant to policy makers or consumers, or where an important HIV research area was evaluated but the protocol review question did not reflect current practices or thinking. The remaining protocols were discontinued due to methodological limitations, as a result of inadequately conceptualized research questions. Specifically, protocols encompassed a very broad set of treatment strategies that were difficult to combine in one review, or authors did not consider how to meaningfully combine complex HIV interventions. This was common in protocols which included behavioral HIV prevention interventions and pragmatic implementation trials. Thirty protocols were published more than 2ā€‰years previously (interquartile range 3ā€“9ā€‰years).

Discussion

This appraisal of Cochrane systematic reviews and protocols demonstrates that topics in the HIV field become outdated quickly, and this was by far the commonest reason for discontinuation of both reviews and protocols, including those from more recent years. In addition, changes in reporting standards over time resulted in outdated methodology. Protocols which progressed slowly became irrelevant before completion, and inadequately conceptualized research questions resulted in disorganized or empty reviews which, at times, did not address issues specifically relevant to HIV populations, or sufficiently characterize interventions and key HIV outcomes.

Evaluating the relevance of systematic reviews prior to updating is essential in all fields but particularly important where there is rapid development of new innovations and care strategies, as seen in HIV research. Where there are delays, protocols should be re-evaluated and adapted to ensure that final reviews answer current questions. With the recent drive to automate and create ā€œlivingā€ systematic reviews with frequent updates [5, 6], it is essential that methods of maintaining currency are incorporated into these evidence synthesis approaches as questions may change or become irrelevant [7, 8].

We recommend that authors planning to conduct HIV systematic reviews reflect on how their review can contribute, by thoroughly exploring the current evidence, determining if primary studies exist on the topic of interest, identifying other systematic reviews already conducted, and clearly defining their research question and what their review may add. In addition to presenting the PICO (population, intervention, comparison, and outcomes) elements, authors should fully characterize the interventions of interest, limit the number and types of interventions, and determine how to synthesize results from studies with behavioral or complex interventions. Several available tools can be drawn on and adapted to aid the design of conceptual frameworks and characterization of interventions and complexity: this can help structure review questions to produce meaningful results [7,8,9,10,11]. Review team effort and Cochrane editorial procedures need to ensure timely completion of reviews to avoid them being outdated before completion.

In summary, HIV research questions and the systematic reviews summarizing the relevant research evidence become outdated quickly, and rapid conduct and publication is critical; clearly defined research questions need to be regularly re-evaluated alongside changes in the field to maintain relevance, particularly in the era of living systematic reviews.

Abbreviations

HIV:

Human immunodeficiency virus

HSV-2:

Herpes simplex virus 2

IQR:

Interquartile range

MECIR:

Methodological Expectations of Cochrane Intervention Reviews

References

  1. Barroso J, Sandelowski M, Voils CI. Research results have expiration dates: ensuring timely systematic reviews. J Eval Clin Pract. 2006; https://0-doi-org.brum.beds.ac.uk/10.1111/j.1365-2753.2006.00729.x.

  2. Higgins J, Lasserson T, Chandler J, Tovey D, Churchill R. Standards for the conduct and reporting of new Cochrane Intervention Reviews, reporting of protocols and the planning, conduct and reporting of updates. 2018. Version 1.07. Accessed at: https://community.cochrane.org/mecir-manual.

    Google ScholarĀ 

  3. Garner P, Hopewell S, Chandler J, et al. When and how to update systematic reviews: consensus and checklist. BMJ. 2016;354:i3507 https://0-doi-org.brum.beds.ac.uk/10.1136/bmj.i3507.

    ArticleĀ  Google ScholarĀ 

  4. Cochrane Editorial Unit. Updating Classification System: guide to applying to Cochrane reviews. 2016. Accessed at: https://community.cochrane.org/editorial-and-publishing-policy-resource/cochrane-review-management/updating-classification-system-cochrane-reviews.

  5. Elliott JH, Synnot A, Turner T, et al. Living systematic review: 1. Introductionā€”the why, what, when, and how. J Clin Epidemiol. 2017;91:23ā€“30 https://0-doi-org.brum.beds.ac.uk/10.1016/j.jclinepi.2017.08.010.

    ArticleĀ  Google ScholarĀ 

  6. Oā€™Connor AM, Tsafnat G, Gilbert SB, Thayer KA, Wolfe MS. Moving toward the automation of the systematic review process: a summary of discussions at the second meeting of International Collaboration for the Automation of Systematic Reviews (ICASR). Syst Rev. 2018;7(1):3 https://0-doi-org.brum.beds.ac.uk/10.1186/s13643-017-0667-4.

    ArticleĀ  Google ScholarĀ 

  7. Hickey MD, Odeny TA, Petersen M, et al. Specification of implementation interventions to address the cascade of HIV care and treatment in resource-limited settings: a systematic review. Implement Sci. 2017;12(1):102 https://0-doi-org.brum.beds.ac.uk/10.1186/s13012-017-0630-8.

    ArticleĀ  Google ScholarĀ 

  8. Rohwer A, Pfadenhauer L, Burns J, et al. Series: clinical epidemiology in South Africa. Paper 3: logic models help make sense of complexity in systematic reviews and health technology assessments. J Clin Epidemiol. 2017;83:37ā€“47. https://0-doi-org.brum.beds.ac.uk/10.1016/j.jclinepi.2016.06.012.

    ArticleĀ  PubMedĀ  Google ScholarĀ 

  9. Guise JM, Chang C, Butler M, Viswanathan M, Tugwell P. AHRQ series on complex intervention systematic reviewsā€”paper 1: an introduction to a series of articles that provide guidance and tools for reviews of complex interventions. J Clin Epidemiol. 2017;90:6ā€“10. https://0-doi-org.brum.beds.ac.uk/10.1016/j.jclinepi.2017.06.011.

    ArticleĀ  PubMedĀ  Google ScholarĀ 

  10. Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8(1):1ā€“11 https://0-doi-org.brum.beds.ac.uk/10.1186/1748-5908-8-139.

    ArticleĀ  Google ScholarĀ 

  11. Lewin S, Hendry M, Chandler J, et al. Assessing the complexity of interventions within systematic reviews: development, content and use of a new tool (iCAT-SR). BMC Med Res Methodol. 2017;17(1):1ā€“13 https://0-doi-org.brum.beds.ac.uk/10.1186/s12874-017-0349-x.

    ArticleĀ  Google ScholarĀ 

Download references

Acknowledgements

We would like to thank Taryn Young and Dierdre Walsh for facilitating this work.

Funding

The work was funded by Cochrane CRG Support Programme in a grant given to the Cochrane Infectious Diseases Group. The work of the Cochrane Infectious Diseases Group is further funded by UK AID through the Research, Evidence and Development Initiative (READ-It, project ID 300342ā€“104). The views expressed do not necessarily reflect the UK governmentā€™s official policies.

Availability of data and materials

Data are available on request from Dierdre Walsh from the Cochrane Infectious Diseases Group, Liverpool School of Tropical Medicine, Liverpool, United Kingdom: Deirdre.Walshe@lstmed.ac.uk

Author information

Authors and Affiliations

Authors

Contributions

All authors contributed to the conduct of the review and protocol appraisal and participated in the drafting and reviewing this manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Ingrid Eshun-Wilson.

Ethics declarations

Authorsā€™ information

None.

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Publisherā€™s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional files

Additional file 1:

Modified tool for assessing reviews and protocols. (DOCX 18 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Eshun-Wilson, I., Jaffer, S., Smith, R. et al. Maintaining relevance in HIV systematic reviews: an evaluation of Cochrane reviews. Syst Rev 8, 46 (2019). https://0-doi-org.brum.beds.ac.uk/10.1186/s13643-019-0960-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://0-doi-org.brum.beds.ac.uk/10.1186/s13643-019-0960-5