Critical Appraisal of Evidence - Therapy (Systematic Review) Scenario
Are the results of this study valid?
Anytime we do a search, finding a systematic review should be our first goal. (Note: the critical appraisal of systematic reviews requires the skills for appraising individual studies and therefore you might want to review the section on critically appraising single therapy trials before tackling this one).
A systematic review is a summary of the medical literature that uses explicit methods to systematically search, critically appraise and synthesise the literature on a specific topic. Let's start by returning to our clinical scenario from the question formulation tutorial:
You admit a 75 year old man with a stroke (left sided weakness) who is having trouble ambulating, feeding, bathing and dressing himself. He has hypertension but it is well controlled with a diuretic. He is otherwise well and now that he is medically stable you decide after discussion with him to transfer him to a stroke unit. His family asks to see you because they are concerned about this transfer. They live very close to the acute care hospital and wonder why he can't stay on the general medical ward where he currently is. You arrange to meet with him and his family to discuss their concerns. In the meantime, you decide to review the evidence for the use of stroke units.
Our search of the literature to answer this question retrieved an article from the The Cochrane Library (Stroke Trialists' Collaboration. Cochrane Library, Update Software, 1999, Issue 4).
How can we critically appraise this paper? We'll start off by considering validity and the following list outlines the questions that we need to consider when deciding if a systematic review is valid.
Is this a systematic review of randomised trials?
In our discussion of single therapy trials, we reviewed how randomised trials are susceptible to less bias than nonrandomised trials. Systematic reviews of RCTs provide the highest level of evidence but systematic reviews of trials that are of lower quality can compound the problems of the individually misleading trials and produce a lower quality of evidence.
The review we found is a Cochrane review and follows the methodology outlined by the Methods Group. It includes randomised and quasi-randomised trials comparing inpatient stroke unit care with conventional care.
Does it include a methods section that describes finding and including all the relevant trials and assessing their individual validity?
We need to review the study to determine how the authors identified the studies and if this process was sufficiently complete. For example, we'd like to see that they contacted authors looking for nonpublished studies, checked bibliographies of all the articles retrieved, handsearched journals and included articles from all languages and from drug company databases. Negative studies are less likely to get published and therefore if these are not searched for, it could lead to a false positive conclusion.
We'd also want to see that the study includes information on how individual studies were appraised for validity. We'd also be reassured if multiple reviews were done and if there was good agreement amongst the reviewers.
In the Cochrane review we identified, the authors searched the Cochrane trials registry, the reference lists of all articles and contacted researchers to identify published and nonpublished studies. Two reviewers independently assessed the quality of the trials and determined their eligibility for inclusion in the review. Although a formal scoring system was not used to record methodological quality, the authors did record the method of allocation concealment, the duration of follow-up, whether or not an intention to treat analysis was done and whether theirs was a blinded assessment of outcomes.
Were the results consistent from study to study?
Did all the studies show results in the same direction? The results shouldn't be exactly the same for all studies but we should be concerned if some studies showed a significant beneficial effect and others significantly excluded any benefit or showed a significant harmful effect. The investigators should test to determine if this lack of consistency (or heterogeneity) was statistically significant. If indeed it was significant, we would like to see that they have explored the reasons why this may have been observed.
The authors of the Cochrane review tested for heterogeneity and it was not significant.
Were individual patient data used in the analysis?
This is a finer point to consider when assessing validity of a systematic review. We'd feel more assured if this was done, especially with regard to reports of subgroups.
If the review fails any of the above criteria, we need to decide if the flaw is significant and threatens the validity of the study. If this is the case, we'll need to look for another study to answer our question, Returning to our clinical scenario, the review we found satisfies all of the above criteria and we will proceed to assessing it for importance.
Are the results of this study important?
What is the magnitude and precision of the treatment effect?
The results of systematic reviews are commonly presented as odds ratios (OR) or relative risks (RR). The ratio of the odds of having the outcome of interest in the experimental group relative to the odds of that same event occurring in the control group is termed the odds ratio. The ratio of the risk of the outcome of interest in the experimental group relative to the risk of the same outcome in the control group is called the relative risk.
Like the relative risk reduction, neither the odds ratio nor the relative risks preserves the baseline risk of the outcome event, and, they are difficult to translate at the point of care. This can be overcome by converting them to a number needed to treat (NNT).
To calculate the NNT/NNH for any OR or RR we can use the following equations:
- For RR < 1:
- NNT = 1/(1-RR) x PEER
- For RR > 1:
- NNT = 1/(RR-1) x PEER
- For OR < 1:
- NNT = 1-[PEER x (1-OR)]/(1-PEER) x (PEER) x (1-OR)
- For OR > 1:
- NNT = 1 + [PEER x (OR-1)]/(1-PEER) x (PEER) x (OR -1)
Alternatively, (and more simply) we could use the tables below to obtain the NNT for specific OR and PEERs.
When OR < 1:
| For Odds Ratios LESS than 1 | ||||||||
|---|---|---|---|---|---|---|---|---|
| 0.9 | 0.8 | 0.7 | 0.6 | 0.5 | 0.4 | 0.3 | ||
| Patient Expected Event Rate (PEER) | 0.05 | 209 | 104 | 69 | 52 | 41 | 34 | 29 |
| 0.10 | 110 | 54 | 36 | 27 | 21 | 18 | 15 | |
| 0.20 | 61 | 30 | 20 | 14 | 11 | 10 | 8 | |
| 0.30 | 46 | 22 | 14 | 10 | 8 | 7 | 5 | |
| 0.40 | 40 | 19 | 12 | 9 | 7 | 6 | 4 | |
| 0.50 | 38 | 18 | 11 | 8 | 6 | 5 | 4 | |
| 0.70 | 44 | 20 | 13 | 9 | 6 | 5 | 4 | |
| 0.90 | 101 | 46 | 27 | 18 | 12 | 9 | 4 | |
When OR > 1:
| For Odds Ratios GREATER than 1 | ||||||||
|---|---|---|---|---|---|---|---|---|
| 1.1 | 1.25 | 1.5 | 1.75 | 2 | 2.25 | 2.5 | ||
| Patient Expected Event Rate (PEER) | 0.05 | 212 | 86 | 44 | 30 | 23 | 18 | 16 |
| 0.10 | 113 | 46 | 24 | 16 | 13 | 10 | 9 | |
| 0.20 | 64 | 27 | 14 | 10 | 8 | 7 | 6 | |
| 0.30 | 50 | 21 | 11 | 8 | 7 | 6 | 5 | |
| 0.40 | 44 | 19 | 10 | 8 | 6 | 5 | 5 | |
| 0.50 | 42 | 18 | 10 | 8 | 6 | 5 | 4 | |
| 0.70 | 51 | 23 | 13 | 10 | 9 | 8 | 7 | |
| 0.90 | 121 | 55 | 33 | 25 | 22 | 19 | 18 | |
Returning to our clinical scenario, we are interested in the outcome death and dependency. The OR for these events is 0.75 (95 % CI 0.65 to 0.87). From the review, we've obtained a PEER for death and dependency of 68% and using the table above, we can estimate that the NNT is approximately 20.
