Skip to main content

Table 1 Items, response choices, and inter-rater reliability estimates for the Evidence Project risk of bias tool

From: The Evidence Project risk of bias tool: assessing study rigor for both randomized and non-randomized intervention studies

Risk of bias tool domains

Items

Response choices

Kappa (κ)1

Dichotomous

Categorical

 Study design

1: Cohort

Yes, No

0.48

 

2: Control or comparison group

Yes, No

0.80

 

3: Pre/post intervention data

Yes, No

0.74

 

 Participant representativeness

4: Random assignment of participants to the intervention

Yes, No, NA

0.78

0.56

5: Random selection of participants for assessment

Yes, No

0.41

 

6: Follow-up rate of 80% or more

Yes, No, NA, NR

0.67

0.55

 Equivalence of comparison groups

7: Comparison groups equivalent on sociodemographics

Yes, No, NA, NR

0.65

0.56

8: Comparison groups equivalent at baseline on outcome measures

Yes, No, NA, NR

0.59

0.50

Median kappa score (κ) across individual items 1–8

 

0.66

 

Weighted kappa (κw) of the total count of items (sum of individual item dichotomous responses)

 

0.66

 
  1. 1Kappa estimates are reported for dichotomous (Yes, No) and categorical ratings when appropriate. Categorical response sets further classify binary No ratings as: No (reported), NA (not applicable), NR (not reported). Agreement was categorized as poor (0.00), slight (0.01–0.20), fair (0.21–0.40), moderate (0.41–0.60), substantial (0.61–0.80), or almost perfect (0.81–1.00)