Table 3 Guidance summary
From: Integration of existing systematic reviews into new reviews: identification of guidance needs
| Â | AHRQ Evidence-based Practice Center Program (EPC program) | Cochrane collaboration | Danish Centre for Health Technology Assessment (DACEHTA) | |
|---|---|---|---|---|
Locating | Two strategies are recommended for identifying existing systematic reviews for a CER. The first strategy is to perform a targeted search of a higher yield database, which includes output from the Evidence-based Practice Center program, MEDLINE’s Top 120 Index Medicus Journals, Health Technology Assessments, Cochrane Database of Systematic Reviews and Database of Abstracts and Reviews of Effects. The second strategy is to identify systematic reviews during a broad de novo literature search. | Systematic reviews can be located through CDSR, DARE and HTA database. MEDLINE and EMBASE can also be used to search for systematic reviews. In MEDLINE, most review articles can be found under the publication Term ‘Meta-analysis’ and in EMBASE, the thesaurus term ‘Systematic Review’ can be used. Specific search strategies can be used to identify systematic reviews in MEDLINE and EMBASE. Additionally, systematic reviews can be identified through search services such as Turning Research into Practice (TRIP). | Secondary studies (for example, systematic reviews, HTA reports, and clinical guidelines) should be located to determine if key questions have already been answered. Secondary studies can be identified through several databases (for example, The HTA Database, Cochrane Database of Systematic Reviews, Database of Abstracts of Reviews of Effects, Guidelines International Network, National Guidelines Clearinghouse, Health Evidence Network, National Electronic Library for Health: Guidelines Finder, and Turning Research Into Practice). | |
In an Overview, primarily only Cochrane Intervention reviews should be included, but other reviews may be included occasionally | ||||
Assessing Relevance | An existing systematic review should be used with the intent to answer parts or all of specific key questions. PICOTS-SD must be considered for relevance of existing systematic reviews. Reviews that are partially relevant may be useful for background or checking references. An initial screening for relevance should be performed, considering the timeliness of the review’s literature search. It is recommended to bridge any search date that ended more than one year from the time the systematic review is identified. If a review is outdated but still desired to be used, an update of the search should be done. | In an Overview, included reviews should be assessed using specific criteria. Considerations include whether a review is up-to-date and if there are specific limitations for the objectives of the Overview. | All evidence should be assessed for relevance to the topic. Identified articles should be compared to the focused question to determine if the article may answer the focus question. The literature can be divided into two groups (secondary studies and primary studies). If a large amount of evidence has been identified, it can be subdivided into groups based on presumed quality. The hierarchy of evidence is: 1) meta-analyses and systematic reviews (among others Cochrane reviews); 2) randomized controlled trials (RCTs); 3) non-randomized controlled trials; 4) cohort studies; 5) case-control studies; 6) descriptive studies, limited series; and 7) position papers, non-systematic reviews, leading articles, expert opinions. | |
In the second stage of screening, the review’s PICOTS-SD elements should be compared to those in the new review protocol for relevance. If these elements are poorly reported, the review should not consider including the existing review. | ||||
Assessing Review Quality | Only reviews of high quality should be included in a CER. Two independent reviewers should assess for quality and methods for resolving discrepancies should be reported. A quality rating instrument should be used to addresses all aspects of the review that will be incorporated into the CER. Both the methods used to minimize bias and the reporting should be assessed. QUOROM (PRISMA) is a checklist that can be used to assess the reporting of systematic reviews. As a common starting point, the AMSTAR tool should be used to assess the quality of reviews. Reproducibility and application of inclusion and exclusion criteria should be confirmed. As some limitation to AMSTAR exists, it is recommended to describe implications of potential methodological flaws instead of relying on numerical scores. | Generally, selection criteria for a Cochrane Overview limits included reviews to Cochrane reviews. Non-Cochrane systematic reviews may be included if there are good quality reviews for which a Cochrane review is not available. | No guidance | |
Determining use: Scanning References | The list of included articles from an existing review can be used in a CER if methods for identifying articles are of adequate quality. | Existing systematic reviews can be used as sources of relevant studies. References lists of systematic reviews can be searched to identify relevant articles. This should be done as an adjunct to other search methods as bias may be present in what studies were included in existing reviews. | No guidance | |
Determining use: Use Search | Part or all of the search strategy may be used from an existing review if it is consistent with EPC program methods for finding evidence. A search strategy from an existing review can be used, followed by de novo analysis and synthesis of data. | Existing reviews may be a useful source information about search strategies | No guidance | |
Determining use: Risk of Bias Assessment | In order to use the risk of bias assessment from a systematic review, the methods used must be consistent with the EPC program methods guide. These methods include selection of design specific criteria for risk of bias assessment and use of appropriate tools. | In an Overview, an assessment of the quality of evidence should be done. If no assessment was done in an included systematic review, authors should perform the assessment. If a quality assessment was done in an included systematic review, authors should assess the judgments and ensure consistency between included reviews. | A quality assessment tool should be used to uniformly assess the quality of identified articles. Check list tools developed by different national centers (for example, SIGN, NICE, GRADE and Centre for Evidence-based Medicine, Oxford) can be used to assess quality. | |
Determining use: Data Abstraction | The data extraction tables may be used from an existing review, if they are deemed to be of adequate quality. However, if results of individual trials are not reported, the use of summary findings from an existing review may compromise transparency for a CER and this is not recommended. | In an update, data from new studies should be abstracted and included, if applicable. | No guidance | |
In an Overview, if necessary, authors may seek additional data or information from the authors of primary studies from included systematic reviews. | ||||
Use Synthesis | If an existing review is very similar to a CER in research questions and is high quality, the entirety or portions of the existing review may be incorporated. Summarized evidence for specific populations or interventions may be included in a CER. If summarized evidence is to be included, the existing review must have methods consistent with EPC program methods for finding evidence, assessing quality, grading the strength of evidence and other principles including conflicts of interest. Summarized evidence can also be incorporated with a de novo sensitivity analysis. | In an update, data collected from new studies should be included and a new meta-analysis should be done. | No guidance | |
If multiple high quality reviews are found, a single review can be chosen, which is most relevant and least biased, or a meta-review can be performed. | In an Overview, authors should reply on previous analyses when possible. If there are differences between reviews (for example, different populations or subgroups are analyzed), data may need to be reanalyzed. | |||
If more than one high quality review is found with discordant findings, it may be an indication to start a de novo review on that key question. | Â | |||
Report Methods/Results | It is recommended to provide a summary table to show where existing review(s) were used to replace de novo processes. Summary tables of existing systematic reviews should be included to compare the reviews and should address any overlap (or lack thereof) in the primary research included in reviews. | In an update, revision to text of the existing review will depend on the influence of the new data and results. If there is no change in the results, little revision to the text is required. However, some updates may require a change to the conclusion of a review which will require much modification of the text. It should be noted in the Abstract and Background that this is an update. A ‘What’s new’ table should be completed and changes should be made to ensure no dates or other information is out of date. | No guidance | |
The discussion section should include a justification for using an existing systematic review and address any limitations. It is also important to compare findings from the CER with the findings from existing reviews. | ||||
In an update, it is important to show explicitly what has changed from the previous report. The desired depth of information varies between users of reports. Review updates may be effectively presented in an executive summary with tables and figures, identifying and modifications followed by a full report for users who require further depth of information. | ||||
| Â | European Collaboration Health Technology Assessment (ECHTA) | Institute for Quality and Efficiency in Health Care (IQWiG) | National Institute for Health and Clinical Excellence (NICE) | York Centre for Reviews and Dissemination: Systematic Reviews (CRD) |
Locating | In order to determine if key questions have already been answered, a search for previous HTA reports should be conducted. The search for previous reports should be systematic and well documented. | Different databases should be considered in locating systematic reviews that are used for primary literature. Databases that exclusively or mostly hold systematic reviews should be searched as well as biomedical databases (for example, MEDLINE and EMBASE), which also hold systematic reviews. | Core and subject databases, including MEDLINE, EMBASE, Cochrane Database of Systematic Reviews, Database of Abstracts of Reviews of Effects, Cochrane Central Register of Controlled Trials, and Health Technology Assessment (HTA) database, should be searched for every review. For questions on effectiveness, a search should be done for systematic reviews, followed by randomized controlled trials, then cohort or case–control studies. Search filters are available to assist in identifying studies, including a search filter for systematic reviews. | To check if key questions have been answered by existing or ongoing reviews, a search for systematic reviews should be conducted. The Database of Abstracts of Reviews of Effects (DARE), and the Cochrane Database of Systematic Reviews (CDSR) should be searched. Additionally, NICE and NIHR HTA program Websites can be searched along with the Campbell Library of Systematic Reviews and the Evidence for Policy and Practice Information Centre’s Database of Systematic and Non Systematic Reviews of Public Health Information (DoPHER). Guideline groups including NGC and SIGN can be searched for guidelines based on systematic review evidence. MEDLINE and other databases can be searched for previous reviews. |
Evidence scanning should be done continuously to identify systematic reviews that concern published or developing information products. Two people should regularly screen (CDSR, DARE, INAHTA, MORE and PubMed). Identified reviews that concern a product of the Institute can influence the updating process, including triggering an update or modifying the updating plan. | ||||
Assessing Relevance | All identified evidence should be assessed with pre-defined inclusion and exclusion criteria. Selection criteria should be developed from background information, research questions and the availability of evidence and should be defined prospectively to avoid bias in selection of evidence. Inclusion and exclusion criteria should cover patient characteristics, condition characteristics, technology aspects, methodological issues, outcomes measured, publication type. | No guidance | No guidance | No guidance |
Assessing Review Quality | No guidance | For systematic reviews to be used in a benefit assessment, they must be assessed for sufficient quality. They must ‘show only a minimum risk of bias; present the evidence base in a complete, transparent and reproducible manner; and thus allow clear conclusions to be drawn’. The searches conducted in the systematic reviews must not contradict the methodology of the Institute. Quality assessment should be done with Oxman and Guyatt’s quality index for systematic reviews or AMSTAR. Sponsors and authors’ conflicts of interests should be documented and discussed for systematic reviews. | Guidelines may contain reviews of evidence that are applicable to questions formulated by the guideline development group. These may be used as evidence if: ‘they are assessed using the appropriate methodology checklist from this manual and are judged to be of high quality, they are accompanied by an evidence statement and evidence table(s), the evidence is updated according to the methodology for the exceptional update of NICE clinical guidelines’. | Identified reviews should be assessed for quality. Quality reviews should have a well-defined question, comprehensive search, clear and appropriate selection or studies, unbiased processes for assessing study quality and extracting and synthesizing data. Checklists can be used to help in assessing quality of systematic reviews (for example, Oxman and Guyatt, Guidelines for reading literature reviews). |
If more than one systematic review of adequate quality is found to address a particular subject, additional quality assessment should be done. Items to compare include: content of the review, search strategy and date, sensitivity analysis, how bias is assessed and dealt with, and updating provisions. | ||||
Determining use: Scanning References | No guidance | No guidance | No guidance | Other sources of literature include references lists of systematic reviews. References lists of existing reviews can be scanned to identify additional studies. |
Determining use: Search | No guidance | No guidance | No guidance | No guidance |
Determining use: Risk of Bias Assessment | All evidence should be critically assessed for quality. Checklists can be used for appraisal of medical literature. All sources of information should be appraised for validity. No guidelines exist for assessing quality of sources of information other than medical literature, and this is a gap that future guidance needs to address. | No guidance | No guidance | No guidance |
Determining use: Data Abstraction | No guidance | No guidance | If using reviews of evidence published in other guidelines, the guideline development group should create new evidence summaries or statements. The original evidence tables should be referenced with a direct link to the source if possible or a reference to the published document. Verbatim quotes of recommendations from other guidelines should not be used, unless the recommendations come from NHS policy or legislation. | No guidance |
Determining use: Synthesis | No guidance | No guidance | No guidance | No guidance |
Report Methods/ Results | No guidance | Results of systematic reviews should be summarized in tables if possible. If discordant results on the same outcome are found, possible explanations should be given. If it appears that a new benefit assessment based on primary studies would produce different results, a new assessment should be done. | Original evidence tables from published guidelines should be referenced with a direct link to the source if possible or a reference to the published document. Verbatim quotes of recommendations from other guidelines should not be used, unless the recommendations come from NHS policy or legislation. | No guidance |