Table 3 Comments on signalling questions in the ROBINS-E risk of bias tool that were difficult to assess and often irrelevant to a particular study

Domain 1: Confounding

Signalling question 1.3: time-varying confounding: Were exposure discontinuations or switches likely to be related to factors that are prognostic for the outcome?

Cohort studies can continue over decades so changes in exposure may be related to a wide variety of factors. For example, in studies assessing dietary exposures, it is impossible to distinguish whether someone has made a change in their diet due to a diagnosis or onset of a symptom rather than personal choice or social reasoning (e.g. veganism).

1.4: Baseline confounding: Did the authors use an appropriate analysis method that adjusted for all the critically important confounding areas?

Most of the studies we coded had many relevant confounders, and it was rare that all confounders were controlled in every study, so we modified this question by developing decision rules around the number of confounders that were taken into account.

We also determined if the study avoided adjusting for post-exposure variables. For example, in a study assessing cardiovascular disease (CVD) as an outcome, it is inappropriate to adjust for new incidence of hypertension that has occurred during the exposure period. Hypertension is not a confounder because it is on the causal pathway to CVD.

Domain 2: Bias in selection of participants

2.3 and 2.3: Were the post-exposure variables that influenced selection associated with exposure? Were the post-exposure variables that influenced eligibility selection influenced by the outcome or a cause of the outcome?

Since cohort studies are often assembled based on exposure levels, it is rare for selection to be unrelated to exposure. In exposure studies, participants are almost always selected into the study based on characteristics that are assessed after the start of exposure. For example, in a study assessing the association of an exposure with cardiovascular disease, subjects may be excluded if baseline surveys or clinical records determine they have diabetes, hypertension or metabolic syndrome, characteristics which may be associated with exposure or outcome.

Domains 3 and 4: Exposures

2.4 Do start of follow-up and start of exposure coincide for most participants?

3.2 Did entry into the cohort begin with start of the exposure?

For many types of exposures, such as dietary exposures or various types of pollution, exposure can begin in infancy, long before entry into a cohort. Unlike interventions, exposures are not initiated by the investigators, so exposure and follow-up will rarely coincide.

4.1 Is there concern that changes in exposure status occurred among participants?

4.2 Did many participants switch to other exposures?

In exposure studies, there is always a concern that changes in exposure status occurred among participants. It is rare that exposure measurements are made continuously over long periods of exposure. Techniques are used that are likely to correct for this issue, such as multiple assessments of exposure (e.g. every 2 years) and person-years adjustment. ROBINS-E terms such as ‘intended’ exposure, ‘initiating and adhering to an exposure’ and ‘switching’ exposures are applicable to randomised trials, but do not apply to exposure studies where exposure is not controlled by the investigators.

Domain 5–7: Bias due to missing data, bias in measurement of outcomes and bias in the selection of reported results

Most of the questions related to these domains were applicable to observational studies.

Signalling questions related to selective reporting of results (domain 7) ask whether particular outcomes are reported from multiple outcome measures, particular analyses are reported from multiple analyses, and whether data are reported for only a subset of participants. We were unable to answer these questions unless the protocol for the study was available and published protocols are rare for observational studies. Therefore, we most frequently coded this domain as ‘not enough information’.