Table 5 Assessment of the certainty of the evidence arising from the overview

      1.1.1 Assess the certainty of the evidence using a method developed for use in overviews

Wagner 2012 [80]

▪ Wagner 2012 [80] report an approach to assigning levels of evidence in an overview based on the number and quality of included SRs (primary studies were not considered).

      1.1.2 Assess the certainty of the evidence using an ad hoc method developed for a specific overview

Bolland 2014 [5]; Cooper 2012 [6]; Crick 2015 [48]; Hartling 2012 [53]; Pollock 2015 [31]; Ryan 2009 [25]; Thomson 2010 [26]; Wagner 2012 [80]

▪ Pollock 2015 [31] adapted GRADE methods for their overview, incorporating an additional domain to account for potential bias arising from the methods used in included SRs. Decision rules were used to ensure consistent grading of domains deemed important to their overview question; these did not specifically address considerations unique to overviews

      1.1.3 Report assessments of certainty of the evidence from the included SRs, using the approaches specified for data extraction to deal with missing data, flawed or discordant assessments (e.g. where two SRs use different methods to assess certainty of the evidence or report discordant assessments using the same method) (see ‘Data extraction’ table in [10]).

Becker 2008 [4]; Cooper 2012 [6]; Hartling 2012 [53]; Hartling 2014 [55]; JBI 2014 [39, 59]; Kramer 2009 [61]; Pieper 2014c [66]; Robinson 2015 [24, 69,70,71,72]; Ryan 2009 [25]; Silva 2014 [75]

▪ Report assessments of the certainty of the evidence for each comparison and outcome directly from the included SRs, irrespective of the method used, noting missing data and discrepancies (Hartling 2012 [53]; JBI 2014 [39]; Robinson 2015 [24, 69,70,71,72])

▪ Report the certainty of the evidence data from the Cochrane review with the most comprehensive assessment

      1.1.4 Report assessments of certainty of the evidence from the included SRs after performing quality checks on a sample of assessments to verify that the assessment method has been applied appropriately and consistently across SRs

Becker 2008 [4]; Robinson 2015 [24, 69,70,71,72]; Thomson 2010 [26]

▪ Report the certainty of the evidence assessments after retrieving primary study data from the included trials and independently check 10% of primary study data

▪ Report the certainty of the evidence assessments after cross-checking the assessments across overlapping SRs (Becker 2008 [4]; and quoted in Thomson 2010 [26])

      1.1.5 (Re)-assess the certainty of the evidence using an existing method developed for SRs of primary studies without adapting the method for overviews

Crick 2015 [48]; Foisy 2014 [51]; JBI 2014 [39]; Hartling 2012 [53]; Robinson 2015 [24, 69,70,71,72]; Ryan 2009 [25]; Thomson 2010 [26]

▪ Use GRADE [30] for assessing the certainty of the evidence without modifying the domains or decision rules used to assess the certainty of the evidence in a SR of primary studies (Hartling 2012 [53]; JBI 2014 [39, 59]; Robinson 2015 [24, 69,70,71,72]). May be done for missing assessments, if there are missing studies from an assessment, if there are concerns about reported assessment(s), or if there are differences between the overview and SR questions that necessitate re-assessment (e.g. different population).

▪ For new primary studies or those not integrated into the assessment reported in SRs, re-assess the certainty of evidence (Hartling 2012 [53]; JBI 2014 [39, 59]; Robinson 2015 [24, 69,70,71,72])

▪ When two different tools are used (e.g. GRADE [30] and AHRQ [33], then re-assess certainty of the evidence for each comparison and outcome by standardising the assessments based on similar domains

      1.1.6 Do not report or assess the certainty of the evidence

Inferred

      2.1.1 Independent assessment by 2 or more authors

Baker 2014 [43]; Becker 2008 [4]; Cooper 2012 [6]; Li 2012 [62]; JBI 2014 [39, 59]; Ryan 2009 [25]

      2.1.2 One author assesses

Inferred

      2.1.4 One assesses, 2nd confirms

Cooper 2012 [6]

      2.1.5 One assesses, 2nd confirms if the first author is unsure

Cooper 2012 [6]

  2.2 Determine if authors (co-)authored one or several of the SRs included in the overview, and if yes, plan safeguards to avoid bias in certainty of the evidence assessment

Büchter 2011 [45, 65]

▪ Overview authors do not assess the certainty of the evidence from their co-authored SRs