Skip to main content

Table 1 Average measured loss for each criterion, measured for all tests where the criteria triggered

From: Measuring the impact of screening automation on meta-analyses of diagnostic test accuracy

  

Triggered

Effort

Recall

L2 loss

Sensitivity loss

Specificity loss

Criterion type

Criterion

MA

SR

abs

perc

  

mean

lb

ub

mean

lb

ub

Ranked

Retrospective

Recall (95%)

41

1

5494.122

0.489

0.960

0.001

0.001

0.001

0.001

0.000

0.000

0.000

 

Knee/elbow

314

30

120.752

0.047

0.299

0.056

0.040

0.174

0.086

0.029

0.221

0.038

 

Loss/effort (0.02/1000)

314

30

259.570

0.104

0.527

0.023

0.018

0.140

0.053

0.010

0.170

0.026

 

Loss/effort (0.015/1000)

314

30

271.589

0.108

0.539

0.022

0.018

0.136

0.052

0.009

0.168

0.025

 

Loss/effort (0.01/1000)

314

30

297.064

0.113

0.546

0.021

0.016

0.136

0.049

0.009

0.166

0.023

Prospective

Found/effort (1/500)

277

30

811.220

0.268

0.770

0.012

0.008

0.011

0.014

0.006

0.020

0.003

 

Found/effort (1/1000)

254

22

1338.465

0.386

0.783

0.007

0.005

0.006

0.005

0.003

0.007

0.002

 

Found/effort (1/2000)

174

17

2330.448

0.424

0.815

0.002

0.002

0.002

0.002

0.001

0.002

0.000

 

Relevant found (n= 20)

41

1

162.732

0.050

0.565

0.023

0.020

0.026

0.022

0.007

0.014

0.008

 

Relevant found (n= 15)

57

3

140.000

0.039

0.534

0.029

0.024

0.029

0.026

0.010

0.020

0.011

 

Relevant found (n= 10)

96

4

109.042

0.044

0.529

0.031

0.024

0.039

0.028

0.013

0.027

0.014

 

Displacement MA2 (0.005)

35

0

344.714

0.086

0.667

0.014

0.012

0.015

0.012

0.005

0.011

0.005

 

Displacement MA2 (0.010)

57

2

280.000

0.065

0.578

0.018

0.016

0.021

0.018

0.005

0.013

0.008

 

Displacement MA2 (0.015)

74

3

205.189

0.059

0.538

0.024

0.019

0.030

0.021

0.008

0.018

0.012

 

Displacement MA2 (0.020)

91

4

124.099

0.044

0.511

0.029

0.024

0.034

0.025

0.010

0.022

0.013

 

Displacement LOOCV (0.005)

42

0

131.667

0.050

0.538

0.021

0.019

0.030

0.016

0.006

0.017

0.008

 

Displacement LOOCV (0.010)

90

3

192.356

0.054

0.461

0.030

0.026

0.038

0.025

0.010

0.021

0.013

 

Displacement LOOCV (0.015)

130

5

161.923

0.063

0.427

0.035

0.028

0.042

0.030

0.013

0.029

0.015

 

Displacement LOOCV (0.020)

152

5

172.947

0.053

0.370

0.042

0.034

0.054

0.034

0.016

0.033

0.019

Randomized

Retrospective

Recall (95%)

36

1

6956.583

0.961

0.959

0.002

0.002

0.002

0.002

0.000

0.001

0.000

 

Knee/elbow

314

30

1658.322

0.299

0.203

0.064

0.042

0.208

0.103

0.039

0.228

0.051

 

Loss/effort (0.02/1000)

309

27

1315.816

0.351

0.264

0.047

0.036

0.269

0.101

0.022

0.333

0.049

 

Loss/effort (0.015/1000)

309

27

1429.359

0.364

0.285

0.043

0.033

0.249

0.097

0.020

0.309

0.046

 

Loss/effort (0.01/1000)

309

27

1552.871

0.390

0.313

0.040

0.030

0.237

0.093

0.018

0.299

0.043

Prospective

Found/effort (1/500)

218

16

1854.106

0.374

0.169

0.112

0.073

0.329

0.143

0.063

0.418

0.061

 

Found/effort (1/1000)

151

10

2942.583

0.465

0.237

0.097

0.062

0.268

0.113

0.055

0.338

0.045

 

Found/effort (1/2000)

68

6

5097.559

0.493

0.272

0.082

0.052

0.239

0.083

0.042

0.294

0.022

 

Relevant found (n= 20)

41

1

5106.000

0.574

0.565

0.022

0.018

0.023

0.021

0.008

0.013

0.010

 

Relevant found (n= 15)

57

3

4288.070

0.550

0.534

0.024

0.020

0.026

0.025

0.010

0.015

0.013

 

Relevant found (n= 10)

93

4

3275.194

0.537

0.517

0.032

0.024

0.039

0.028

0.015

0.025

0.016

 

Displacement MA2 (0.005)

33

0

6478.970

0.663

0.644

0.016

0.014

0.017

0.014

0.006

0.011

0.009

 

Displacement MA2 (0.010)

54

2

4395.889

0.537

0.519

0.023

0.019

0.027

0.021

0.009

0.016

0.012

 

Displacement MA2 (0.015)

74

3

4103.730

0.533

0.505

0.025

0.021

0.030

0.023

0.010

0.017

0.011

 

Displacement MA2 (0.020)

87

4

3478.276

0.509

0.485

0.026

0.022

0.034

0.024

0.010

0.020

0.013

 

Displacement LOOCV (0.005)

40

1

5525.525

0.557

0.537

0.017

0.014

0.020

0.014

0.008

0.014

0.007

 

Displacement LOOCV (0.010)

87

3

3928.483

0.498

0.454

0.027

0.022

0.037

0.022

0.009

0.022

0.011

 

Displacement LOOCV (0.015)

130

4

2902.831

0.475

0.415

0.038

0.031

0.048

0.030

0.014

0.032

0.015

 

Displacement LOOCV (0.020)

154

5

2808.455

0.453

0.396

0.041

0.035

0.051

0.034

0.014

0.035

0.016

  1. Triggered signifies the number of meta-analyses (MA, maximum 400) for which the criterion triggered and the number of systematic reviews (SR, maximum 48) where the criterion triggered for all meta-analyses. Effort signifies the absolute and relative number of references needed to be screened before triggering the stopping criteria. Recall signifies the percentage of relevant studies identified when the stopping criterion triggered. The loss in sensitivity and specificity are measured as the difference to the final estimates at the criterion threshold. We also include the the difference between the measured lower and upper bounds of the 95% confidence intervals and their final estimated values (lb, ub)