Critical Appraisal of Evidence - Harm Scenario

Are the results of this study valid?

Evidence about harm can come from a number of different study types. Ideally we'd like to see a high quality systematic review of randomised trials but these aren't easy to find because RCTs aren't always feasible for issues of harm. As a result, we usually find evidence about harm in cohort studies (groups of patients who are and aren't exposed to the treatment are followed up for the outcome of interest) and case-control studies (patients with the outcome of interest are matched with patients without the outcome and investigators look retrospectively to determine exposure). Case-control studies are useful when the outcome of interest is rare or when the required follow-up is long. The strength of inference that can be drawn from a case-control study is limited because they are more susceptible to bias.

Returning to our clinical scenario from the question formulation tutorial:

You see a 50 year old man who asks for a repeat prescription of sotalol which he has been taking for extrasystoles for several years. He has a remote history of an MI. You haven't seen him previously and are concerned about the proarrhythmic properties of sotalol given what is known about other antiarrhythmics.

Searching the literature we found an RCT from the Lancet (1996;348:7-12).

How do we critically appraise this harm paper? We'll start off by considering validity first and the following list outlines the questions that we need to consider when deciding if a harm paper is valid.

Were there clearly defined groups of patients, similar in all important ways other than exposure to the treatment or other cause?

Consider the following table:

	Adverse Event		Totals
	Present (Case)	Absent (Control)
Exposure to treatment (RCT or cohort)	a	b	a + b
No exposure to treatment (RCT or cohort)	c	d	c + d
Totals	a + c	b + d	a + b + c + d

This first question is easy to answer if we've been able to find an RCT during our search. Randomisation should make the 2 groups of patients similar for all causes of the outcome that we are interested in. In an RCT, patients in the experimental treatment group would be in cells a or b in the table above and patients in the control group would be in cells c or d.

Returning to our clinical scenario, we have been fortunate in our search and have managed to find an RCT and are satisfied that patients are similar in all important ways other than exposure to sotalol.

However, there's not always an RCT available to answer our questions and indeed more frequently we find cohort or case control studies to answer our questions about harm and etiology. In a cohort study, 2 groups of patients are followed - one group with the exposure to the treatment (a+b in the table) and one group without the exposure (c+d) - for the development of the outcome of interest (either a or c ). Because the decision about who receives treatment is not randomised, exposed patients may differ from nonexposed patients for important determinants of the outcome (these determinants are called confounders). Investigators should document characteristics of patients and either show that they are similar or adjust for the confounders that they identify. This is limited by the fact that investigators can only adjust for confounders that are known and that have been measured.

In case control studies, people with the outcome of interest (cases = a+c) are identified along with those without it (controls = b+d). The proportion of each group who were exposed to the putative agent is assessed. Case control studies are susceptible to more bias than cohort studies because confounders that are transient or that lead to early death won't get measured. We also need to ensure when reading a case control study, that people in the control group had the same opportunity for exposure as people in the case group. For example, if we found a case control study looking at the association between sotalol and sudden cardiac death and its investigators assembled people with sudden cardiac death as the cases but excluded patients with atrial fibrillation from the control group, we'd be concerned that the association found between sotalol and sudden cardiac death could be spurious.

Were treatments/exposures and clinical outcomes measured in the same ways in both groups? (Was the assessment of outcomes either objective or blinded to exposure?)

The application of explicit criteria for the outcomes of interest, a discussion of how they were applied and evidence that they were applied without knowledge of which group the patient was in is important. Blinding is crucial if any judgment is required to assess the outcome (in RCTs and cohorts studies) or the exposure (in case control studies). For example, an unblinded investigator may search more aggressively for outcomes in people with exposure to the putative agent. Similarly, people with the adverse outcome may be more likely to have brooded about their situation and may have greater incentive to recall possible exposure. Therefore we would want patients and interviewers to be blind to the study hypothesis.

In the RCT that we retrieved, the outcome was death and was the same for both groups.

Was the follow-up of the study patients sufficiently long (for the outcome to occur and complete)?

If follow-up is short, it may be that too few study patients will have the outcome of interest, thus providing little information of use to a patient. For example, if investigators were looking at the association between cancer and a particular agent and the follow-up time was 1 month, this would be too short for the investigators to see a clinically important effect.

The more people who are unavailable for follow-up, the less accurate the estimate of the risk of the outcome is. Losses may occur because patients are too ill (or too well) to be followed or may have died, and the failure to document these losses threatens the validity of the study.

The RCT that we found was stopped early because an increased risk of death was noted.

Do the results of the harm study fulfil some of the diagnostic tests for causation?

Is it clear that the exposure preceded the onset of the outcome?

We'd want to make sure that the exposure occurred before the outcome and that it wasn't just a marker that the outcome was already underway. With an RCT, the exposure clearly precedes the outcome as with the trial that we found. If it's a case control study, this question becomes more difficult to answer, and more important to ascertain.

Is there a dose-response gradient?

With larger doses of the agent, was there an increased risk of the outcome event? In the study we retrieved, this wasn't tested since the investigators looked at one dose of sotalol.

Is there positive evidence from a dechallenge-rechallenge study?

This occurs when the outcome event disappears (or decreases in intensity) when the putative agent is withdrawn and reappears when it is reinstituted. This couldn't be done in the RCT we found because the outcome was death.

Is the association consistent from study to study?

Or, is this the only study where the association has been identified? We would be happy to see that several studies have looked at this question and have come to the same conclusion (or even better, if there was a systematic review of the topic). Only 1 RCT has had sufficient power to look at the use of sotalol and the risk of death.

Does the association make biological sense?

If the association between outcome and exposure makes biological sense, a causal relationship is more plausible. The results of the sotalol RCT are consistent with findings from studies that have looked at other antiarrythmics (e.g. CAST).

If the study fails any of the above criteria, we need to decide if the flaw is significant and threatens the validity of the study. If this is the case, we'll need to look for another study. Returning to our clinical scenario, the paper we found satisfies all of the above criteria and we will proceed to assessing it for importance.

What is the magnitude and precision of the association between the exposure and the outcome?

Let's begin by drawing a 2x2 table using the data from the RCT that we found.

	Adverse Event		Totals
	Present (Case)	Absent (Control)
Experimental group (d-solatol)	78 a	1471 b	1549 a + b
Control group (placebo)	48 c	1524 d	1572 c + d
Totals	a + c 126	b + d 2995	a + b + c + d 6242

For RCTs and cohort studies, we look at the risk of the event in the treatment group relative to the risk of the event in the untreated patient. This 'relative risk' is calculated as:

RR = [ a/(a+b) ] / [ c/(c+d) ]

Using the values in the table, the relative risk for death in patients receiving d-sotalol is:

RR = [ 78/1549 ] / [ 48/1572 ]
RR = 1.65

Case control studies sample outcomes, not exposure and therefore we can't calculate the relative risk. Instead, the strength of association is estimated indirectly using the odds ratio = ad/bc.

How big should the relative risk (RR) or odds ratio (OR) be for us to be impressed by it? OR and RR > 1 indicate that there is an increased risk of the adverse outcome with the exposure. Because cohort studies and case control studies are susceptible to many biases, we need to ensure that the OR/RR is greater than that which could occur from bias alone. We also need to look at the confidence interval around the OR and RR to see how precise the estimate is.

A more clinically useful measure than the OR and RR is the number of patients that we'd need to treat with the putative agent in order to cause 1 additional harmful event (number needed to harm or NNH). Using the OR, the NNH can be calculated as:

NNH = [ PEER (OR-1) + 1 ] / [ PEER (OR-1) x (1-PEER) ]

Where PEER = the patient's expected event rate

Alternatively, we can refer to the tables below for this information. We can see from these tables that for different PEER, the same OR can generate very different NNHs.

When OR < 1:

Adapted from John Geddes, 1999

		For Odds Ratios LESS than 1
		0.9	0.8	0.7	0.6	0.5	0.4	0.3
Patient Expected Event Rate (PEER)	0.05	209	104	69	52	41	34	29
	0.10	110	54	36	27	21	18	15
	0.20	61	30	20	14	11	10	8
	0.30	46	22	14	10	8	7	5
	0.40	40	19	12	9	7	6	4
	0.50	38	18	11	8	6	5	4
	0.70	44	20	13	9	6	5	4
	0.90	101	46	27	18	12	9	4

When OR > 1:

Adapted from John Geddes, 1999

		For Odds Ratios GREATER than 1
		1.1	1.25	1.5	1.75	2	2.25	2.5
Patient Expected Event Rate (PEER)	0.05	212	86	44	30	23	18	16
	0.10	113	46	24	16	13	10	9
	0.20	64	27	14	10	8	7	6
	0.30	50	21	11	8	7	6	5
	0.40	44	19	10	8	6	5	5
	0.50	42	18	10	8	6	5	4
	0.70	51	23	13	10	9	8	7
	0.90	121	55	33	25	22	19	18

We can also convert the RR to an NNT/NNH using the following equations:

For RR < 1

NNT = 1/(1-RR) x PEER

For RR > 1

NNT (or NNH) = 1/(RR-1) x PEER

Using the PEER (3.1%) from the study we found and the RR (1.65) that we calculated, the NNH for death from d-sotalol in the study is:

NNH = 1/(1.65-1) x 0.031
NNH = 50

Therefore we would need to treat 50 people with d-sotalol to cause 1 additional death. We can also calculate the confidence interval around this estimate using the inverse of the confidence interval for the absolute risk increase.