Risk of Bias Instrument for Non-Randomized Studies (NRS) of Exposures

Download RoB Instrument for NRS of Exposures

The Risk of Bias Instrument for Non-randomized Studies of Exposures (v_2017Dec)

Stage I: At the review level

Specify the research question

Participants
Experimental exposure
Control exposure

List the confounding domains relevant to all or most studies

List the possible co-exposures that could differ between exposure groups and could have an impact on study outcomes

List the criteria used to determine the accuracy of exposure measurement

Factors to consider when evaluating health outcome assessment

Stage II: For each study

The protocol-specified target experiment fully applies

Participants
Experimental exposure
Control exposure

Specify the outcome

Specify which outcome is being assessed for risk of bias (typically from among those earmarked for the Summary of
Findings table). Specify whether this is a proposed benefit or harm of exposure.

Is your aim for this study…?

?
to assess the effect of initiating intervention (as in an intention-to-treat analysis)

?
to assess the effect of initiating and adhering to intervention (as in a per-protocol analysis)

? other (specify)

Specify the numerical result being assessed

In case of multiple alternative analyses being presented, specify the numeric result (e.g. RR = 1.52 (95% CI 0.83
to 2.77) and/or a reference (e.g. to a table, figure or paragraph) that uniquely defines the result being assessed.

Preliminary consideration of confounders

Complete a row for each important confounding area (i) listed in the review protocol; and (ii) relevant to the setting
of this particular study, or which the study authors identified as potentially important.

“

Important” confounding areas are those for which, in the context of this study, adjustment is expected
to lead to a clinically important change in the estimated effect of the exposure. “Validity”
refers to whether the confounding variable or variables fully measure the area, while “reliability”
refers to the precision of the measurement (more measurement error means less reliability).

I. Confounding areas listed in the review protocol
Confounding area	Measured variable(s)	Is there evidence that controlling for this variable was unnecessary?*	Is the confounding area measured validly and reliably by this variable (or these variables)?	OPTIONAL: Is adjusting for this variable (alone) expected to move the effect estimate up or down?
			Yes / No / No information	Favor intervention / Favor control / No information
			Yes / No / No information

I. Additional confounding areas relevant to the setting of this particular study, or which the study authors identified as important
Confounding area	Measured variable(s)	Is there evidence that controlling for this variable was unnecessary?*	Is the confounding area measured validly and reliably by this variable (or these variables)?	OPTIONAL: Is adjusting for this variable (alone) expected to move the effect estimate up or down?
			Yes / No / No information	Favor intervention / Favor control / No information
			Yes / No / No information

* In the context of a particular study, variables can be demonstrated not to be confounders and so not included in
the analysis: (a) if they are not predictive of the outcome; (b) if they are not predictive of exposure; or (c)
because adjustment makes no or minimal difference to the estimated effect of the primary parameter. Note that
“no statistically significant association” is not the same as “not predictive”.

Preliminary consideration of criteria used to determine the accuracy of measurement of exposure and outcome

Complete a row for each measure listed in the study for the (i) exposure and (ii) outcome. Of the measures listed
in the protocol, consider the sensitivity, specificity, and confidence in the methods used in the study.

I. Exposure measurement method listed in the study
Method of measurement	Measured exposure	Is the exposure measured validly and reliably by this method (or these methods)?
		Yes / No / No information

II. Outcome measurement method listed in the study
Method of measurement	Measured outcome	Is the outcome measured validly and reliably by this method (or these methods)?
		Yes / No / No information

Preliminary consideration of co-exposures

Complete a row for each important co-intervention (i) listed in the review protocol; and (ii) relevant to the setting
of this particular study, or which the study authors identified as important.

“

Important” co-interventions are those for which, in the context of this study, adjustment is expected
to lead to a clinically important change in the estimated effect of the intervention.

I. Co-exposures listed in the review protocol
Co-exposure	Is there evidence that controlling for this co-exposure was unnecessary (e.g., because it was not administered)?	Is presence of this co-exposure likely to favor outcomes in the experimental or the control group
		Favor experimental / Favor comparator / No information
		Favor experimental / Favor comparator / No information
		Favor experimental / Favor comparator / No information

II. Additional co-exposures relevant to the setting of this particular study, or which the study authors identified as important
Co-exposure	Is there evidence that controlling for this co-exposure was unnecessary (e.g., because it was not administered)?	Is presence of this co-exposure likely to favor outcomes in the experimental or the control group
		Favor experimental / Favor comparator / No information
		Favor experimental / Favor comparator / No information
		Favor experimental / Favor comparator / No information

Stage III: For each study: risk of bias assessment

Risk of bias assessment (cohort-type studies)

Bias due to confounding	1.1 Is there potential for confounding of the effect of exposure in this study? If N or PN to 1.1: the study can be considered to be at low risk of bias due to confounding and no further signaling questions need be considered	Y / PY / PN / N	[Description]
		If Y/PY to 1.1, answer 1.2 and 1.3 to determine whether there is a need to assess time-varying confounding:
		1.2. If Y or PY to 1.1 : Was the analysis based on splitting follow up time according to exposure received? If N or PN to 1.2 , answer questions 1.4 to 1.6, which relate to baseline confounding	NA / Y / PY / PN / N / NI	[Description]
		1.3. If Y or PY to 1.2 : Were exposure discontinuations or switches likely to be related to factors that are prognostic for the outcome?	NA / Y / PY / PN / N / NI	[Description]
		If N or PN to 1.3 , answer questions 1.4 to 1.6, which relate to baseline confounding
		1.4. Did the authors use an appropriate analysis method that adjusted for all the critically important confounding areas?	NA / Y / PY / PN / N / NI	[Description]
		1.5. If Y or PY to 1.4 : Were confounding areas that were adjusted for measured validly and reliably by the variables available in this study?	NA / Y / PY / PN / N / NI	[Description]
		1.6. Did the authors avoid adjusting for post-exposure variables?	NA / Y / PY / PN / N / NI	[Description]
		If Y or PY to 1.3 , answer questions 1.7 and 1.8, which relate to time-varying confounding
	1.7. Did the authors use an appropriate analysis method that adjusted for all the critically important confounding areas and for time-varying confounding?	NA / Y / PY / PN / N / NI	[Description]
	1.8. If Y or PY to 1.7: Were confounding areas that were adjusted for measured validly and reliably by the variables available in this study?	NA / Y / PY / PN / N / NI	[Description]
	Risk of bias judgement	Low / Moderate / Serious / Critical / NI	[Support for judgement]
	Optional: What is the predicted direction of bias due to confounding?	Favors experimental / Favors comparator / Unpredictable	[Rationale]
Bias in selection of participants into the study	2.1. Was selection of participants into the study (or into the analysis) based on variables measured after the start of the exposure? If N or PN to 2.1 go to 2.4	Y / PY / PN / N / NI	[Description]
	2.2. If Y/PY to 2.1: Were the post-exposure variables that influenced selection associated with exposure?	Y / PY / PN / N / NI	[Description]
	2.3. If Y/PY to 2.2: Were the post-exposure variables that influenced eligibility selection influenced by the outcome or a cause of the outcome?	NA / Y / PY / PN / N / NI	[Description]
	2.4 Do start of follow-up and start of exposure coincide for most participants?	NA / Y / PY / PN / N / NI	[Description]
	2.5 If Y/PY to 2.2 and 2.3, or N/PN to 2.4: Were adjustment techniques used that are likely to correct for the presence of selection biases?	NA / Y / PY / PN / N / NI	[Description]
	Risk of bias judgement	Low / Moderate / Serious / Critical / NI	[Support for judgement]
	Optional: What is the predicted direction of bias due to selection of participants into the study?	Favors experimental / Favors comparator / Towards null /Away from null / Unpredictable	[Rationale]
Bias in classification of exposures	3.1 Is exposure status well defined?	Y / PY / PN / N / NI	[Description]
	3.2 Did entry into the study begin with start of the exposure?	Y / PY / PN / N / NI	[Description]
	3.3 Was information used to define exposure status recorded prior to outcome assessment?	Y / PY / PN / N / NI	[Description]
	3.4 Could classification of exposure status have been affected by knowledge of the outcome or risk of the outcome?	Y / PY / PN / N / NI	[Description]
	3.5 Were exposure assessment methods robust (including methods used to input data)?	Y / PY / PN / N / NI	[Description]
	Risk of bias judgement	Low / Moderate / Serious / Critical / NI	[Support for judgement]
	Optional: What is the predicted direction of bias due to measurement of outcomes or exposures?	Favors experimental / Favors comparator / Towards null /Away from null / Unpredictable	[Rationale]
Bias due to departures from intended	4.1. Is there concern that changes in exposure status occurred among participants? If your aim for this study is to assess the effect of initiating and adhering to an exposure (as in a per-protocol analysis), answer questions 4.2 and 4.3, otherwise continue to 4.4 if Y or PY to 4.1.	Y / PY / PN / N / NI	[Description]
		4.2. Did many participants switch to other exposures?	Y / PY / PN / N / NI	[Description]
		4.3. Were the critical co-exposures balanced across exposure groups?	Y / PY / PN / N / NI	[Description]
		4.4. If NY/PN PY to 4.1, or Y/PY to 4.2, or 4.3: Were adjustment techniques used that are likely to correct for these issues?	NA / Y / PY / PN / N / NI	[Description]
		Risk of bias judgement	Low / Moderate / Serious / Critical / NI	[Support for judgement]
	Optional: What is the predicted direction of bias due to departures from the intended exposures?	Favors experimental / Favors comparator / Towards null /Away from null / Unpredictable	[Rationale]
Bias due to missing data	5.1 Were there missing outcome data?	Y / PY / PN / N / NI	[Description]
		5.2 Were participants excluded due to missing data on exposure status?	Y / PY / PN / N / NI	[Description]
		5.3 Were participants excluded due to missing data on other variables needed for the analysis?	Y / PY / PN / N / NI	[Description]
		5.4 If Y/PY to 5.1, 5.2 or 5.3: Are the proportion of participants and reasons for missing data similar across exposures?	NA / Y / PY / PN / N / NI	[Description]
		5.5 If Y/PY to 5.1, 5.2 or 5.3: Were appropriate statistical methods used to account for missing data?	NA / Y / PY / PN / N / NI	[Description]
		Risk of bias judgement	Low / Moderate / Serious / Critical / NI	[Support for judgement]
		Optional: What is the predicted direction of bias due to missing data?	Favors experimental / Favors comparator / Towards null /Away from null / Unpredictable	[Rationale]
Bias in measurement of outcomes	6.1 Could the outcome measure have been influenced by knowledge of the exposure received?	Y / PY / PN / N / NI	[Description]
	6.2 Was the outcome measure sensitive?	Y / PY / PN / N / NI	[Description]
	6.3 Were outcome assessors unaware of the exposure received by study participants?	Y / PY / PN / N / NI	[Description]
	6.4 Were the methods of outcome assessment comparable across exposure groups?	Y / PY / PN / N / NI	[Description]
	6.5 Were any systematic errors in measurement of the outcome unrelated to exposure received?	Y / PY / PN / N / NI	[Description]
	Risk of bias judgement	Low / Moderate / Serious / Critical / NI	[Support for judgement]
	Optional: What is the predicted direction of bias due to measurement of outcomes?	Favors experimental / Favors comparator / Towards null /Away from null / Unpredictable	[Rationale]
Bias in selection of the reported result	Is the reported effect estimate likely to be selected, on the basis of the results, from…?
	7.1. … multiple outcome measurements within the outcome domain?	Y / PY / PN / N / NI	[Description]
	7.2 … multiple analyses of the exposure-outcome relationship?	Y / PY / PN / N / NI	[Description]
	7.3 … different subgroups ?	Y / PY / PN / N / NI	[Description]
	Risk of bias judgement	Low / Moderate / Serious / Critical / NI	[Support for judgement]
	Optional: What is the predicted direction of bias due to selection of the reported result?	Favors experimental / Favors comparator / Towards null /Away from null / Unpredictable	[Rationale]
Overall bias	Risk of bias judgement	Low / Moderate / Serious / Critical / NI	[Support for judgement]
Overall bias		Optional: What is the overall predicted direction of bias for this outcome?	Favors experimental / Favors comparator / Towards null /Away from null / Unpredictable	[Rationale]

RL Morgan, K Thayer, G Guyatt, R Blain, S Eftim, P Ross, N Santesso, AC Holloway, HJ Schünemann.
Reliability and validity assessment of a risk of bias instrument for non-randomized studies of exposures.
Presentation at Global Evidence Summit, 2017.