Table 1 Proposed steps to develop QA tools with examples drawn from existing tool development
From: A proposed framework for developing quality assessment tools
Stage | QUADAS-2[8] | ROBIS [9] | PROBAST [10] | ROBINS-I [11] | RoB 2.0 [12] |
|---|---|---|---|---|---|
Stage 1: initial steps | |||||
1.1 Identify need | Update for QUADAS | New tool specifically for RoB in SR | New tool specifically for prediction models | New tool specifically for NRS of interventions | Update to Cochrane RoB tool |
1.2 Obtain funding | Part of MRC methodology programme grant | MRC methodology project grant | No external funding | MIF grant from Cochrane and MRC methodology project grant | MRC hubs project grant |
1.3 Assemble team | Larger group (n = 18) | Steering group (n = 10) + larger group (n = 19) | Larger group (n = 40) | Domain-based working groups and others (n = 35) | Domain-based working groups and others (n = 38) |
1.4 Manage project | Steering group (n = 9) | Project group (n = 5) | Steering group (n = 9) | Steering group (n = 5) | Steering group (n = 8) |
1.5 Define scope | Diagnostic test accuracy | Systematic reviews | Prediction modelling (prognostic and diagnostic) | Non-randomised studies of interventions | Randomised controlled trials |
Bias + applicability Domain based + signalling questions Signalling questions phrased so yes indicates low risk of bias | Bias + relevance Domain based + signalling questions Signalling questions phrased so yes indicates low risk of bias | Bias + applicability Domain based + signalling questions Signalling questions phrased so yes indicates low risk of bias | Bias Domain based + signalling questions | Bias Domain based + signalling questions + algorithm for domain level ratings | |
Stage 2: tool development | |||||
2.1. Generate items | Starting point original QUADAS tool [7] Evidence review of sources of bias and variation | Review of existing QA tools + how QA handled in existing overviews Classification of MECIR guidance to identify sources of bias | Key publications on prognostic modelling studies | Survey of Cochrane review groups and face-to-face meeting. Domain-based working groups | Starting point draft tool based on original Cochrane RoB [23] tool. Evidence review of sources of bias. |
2.2. Agree items | Face-to-face meeting | Face-to-face meeting | Web-based survey | Face-to-face meeting + domain-based working group | Face-to-face meeting + domain-based working group |
2.3. Produce first draft | Steering group | Steering group | Steering group | Domain-based working groups + steering group | Domain-based working groups + steering group |
2.4. Pilot and refine | Web-based survey + piloting in reviews | Web-based survey + piloting in reviews and by steering group members on key reviews | Web-based survey + piloting in reviews | Domain-based working groups, web survey, cognitive interviews, piloting by working groups and external reviewers on key papers, event to pilot version 1.0 | Domain-based working groups + 3 day piloting event + additional piloting of refined version |
Stage 3: Dissemination | |||||
3.1 Publication | Single publication + guidance document as web supplement | Single publication + guidance document as web supplement | Primary publication + E&E document (in process) | Single publication + guidance document on website | Not yet published, guidance document on website |
3.2 Website | Not yet available | ||||
3.3 Uptake | Recommended by multiple organisations including Cochrane, NICE and AHRQ, | Recommended by Cochrane, NICE and Estonia Health Insurance Fund | Recommended by Cochrane Prognosis Group | Not yet, only recently available | Not yet, only recently available |
3.4 Translations | Italian and Japanese versions | Estonian and Portuguese versions in preparation | None to date | None to date | None to date |