Table 4 Challenges encountered when including overlapping systematic reviews in overviews

 Some non-Cochrane SRs were quite broad and examined all relevant interventions for the condition of interest (3/7 overviews affected).

Overview authors may need to closely examine the results of the SRs to identify all intervention comparisons included within these SRs.

One SR in the acute otitis media overview examined “the comparative effectiveness of [all] different treatment options for treating uncomplicated acute otitis media” and contributed outcome data to ten relevant intervention comparisons.

 SRs sometimes referenced only one of multiple related publications. Across SRs, this made multiple related publications hard to identify since they looked like independent publications (2/7 overviews affected).

When looking for overlap among primary studies included in SRs, overview authors should look closely for multiple publications of the same primary study.

In the eczema overview, one Cochrane SR and one non-Cochrane SR included primary study. Examining the “list of excluded references” in the Cochrane SR revealed that these references were duplicate publications of the same primary study.

 Some overlapping primary studies included in non-Cochrane SRs were identified by, but excluded from, the Cochrane SRs for being outside the scope or for having methodological deficiencies (6/7 overviews affected).

If researchers agree with the inclusion decisions made in the Cochrane SRs, it may not be appropriate to include this subset of excluded primary studies in the overview.

One non-Cochrane SR in the acute asthma overview included a primary study that was excluded by a similar Cochrane SR due to methodological deficiencies (cross-over design inappropriate for acute asthma).

 Input from a clinical expert was often required to determine whether the Cochrane SRs comprehensively examined all relevant intervention comparisons (6/7 overviews affected).

Clinical experts should be prepared and able to help make methodological decisions related to inclusion of SRs in overviews.

A clinical expert determined that the Cochrane SRs identified for the croup and acute otitis media overviews likely were comprehensive, and were not comprehensive, respectively.

 Multiple Cochrane SRs sometimes contributed outcome data to the same comparison (i.e., Cochrane SRs sometimes overlapped) (2/7 overviews affected).

Including Cochrane SRs in overviews may not always eliminate issues related to overlapping SRs. Additional decision rules may be needed to address this situation.

Two Cochrane SRs on epinephrine for treatment of bronchiolitis, and glucocorticoids for treatment of bronchiolitis, each included outcome data for the comparison “epinephrine and glucocorticoid vs. placebo.”

 Not all groups of overlapping SRs included a Cochrane SR (2/7 overviews affected).

Additional decision rules may be needed to capture data from groups of overlapping SRs that do not include a Cochrane SR.

In the eczema overview, two overlapping non-Cochrane SRs (but no Cochrane SRs) provided outcome data on “pet exposure vs. no pet exposure at home.”

 Overlapping SRs were sometimes “tied” for most recent year of publication or for highest quality (3/7 overviews affected).

Additional decision rules may be needed to differentiate between SRs with similar publication dates or quality scores.

Because the two most recent overlapping SRs in the acute otitis media overview were both published in 2010, we instead examined the search dates to determine which to include in the overview.

 Search dates were not comprehensively or consistently reported in all SRs (6/7 overviews affected).

Using search dates to choose between overlapping SRs published in the same year may not always be possible or straightforward.

When looking at search dates of SRs, the following issues were encountered: exact search dates not reported; month of search not reported; and different search dates reported for different databases.

 Conducting quality assessments could be challenging and time-intensive (7/7 overviews affected).

Using methodological quality as an inclusion criterion to choose between groups of overlapping SRs may be more complex than using other decision rules.

For all overview topics, we found that conducting quality assessments was always more time and resource intensive than simply assessing the Cochrane status or year of publication of the SRs.

 Overlapping SRs sometimes presented the same or similar outcome data in different ways (6/7 overviews affected).

Researchers must decide how best to extract data from overlapping SRs when the same or similar outcome data are reported differently across SRs.

Different SRs sometimes reported different numerators or denominators for the same outcomes, used different types of analysis or statistical methods to analyze the same outcomes, or measured similar outcomes using different definitions, instruments, scales, or time points.

 Overlapping SRs sometimes had discordant results for the same outcome (5/7 overviews affected).

Including outcome data from all overlapping SRs may not result in a coherent overall analysis within a single outcome. Reconciling the discordance may require in-depth exploration of the methods and results of the different SRs.

Two SRs in the gastroenteritis overview examined “length of hospital stay” for “oral rehydration therapy vs. intravenous therapy.” One SR contained meta-analyzed data that significantly favored the intervention. The other SR contained narrative data that was not significant.

 When including non-Cochrane SRs, data extraction was sometimes difficult due to deficiencies in conduct and reporting of SRs (6/7 overviews affected).

Non-Cochrane SRs with gross deficiencies in conduct and/or reporting may be difficult to include in overviews.

For Cochrane SR, study-level outcome data were always available in well-reported narrative summaries or meta-analyses, but this was not always the case for non-Cochrane SRs.