Prognosis Clinical scenario

Man with a history of a stroke who is concerned about his risk of seizure

Are the results of this study valid?

Information about prognosis can come from a variety of study types. Cohort studies (investigators follow 1 or more groups of individuals over time and monitor for the occurrence of the outcome of interest) are the best source of evidence about prognosis. Randomised control trials can also provide information about prognosis although trial participants may not be representative of the population with the disorder. Case-control studies (investigators retrospectively determine prognostic factors by defining the exposure of cases who have already experienced the outcome of interest and of controls who haven't) are useful when the outcome of interest is rare or when the required follow-up is long. The strength of inference that can be drawn from a case-control study is limited because they are more susceptible to bias.

Returning to our clinical scenario from the question formulation tutorial:

You see a 70 year old man in your outpatient clinic 3 months after he was discharged from your service with an ischemic stroke. He is in sinus rhythm, has mild residual left-sided weakness but is otherwise well. His only medication is ASA and he has no allergies. He recently saw an article on the BMJ website describing the risk of seizure after a stroke and is concerned that this will happen to him.

In the tutorial on clinical questions, we formulated the following question: In a 70 year old man does a history of stroke increase his risk for seizure?

Our search of the literature to answer this question retrieved an article from the BMJ (1997;315:1582-7).

How do we critically appraise this prognosis paper? We'll start by considering validity first and the following list outlines the questions that we need to consider when deciding if a prognosis paper is valid.

Was a defined, representative sample of patients assembled at a common (usually early) point in the course of their disease?
We hope to find that the individuals included in the study are representative of the underlying population (and reflect the spectrum of illness). But, from what point in the target disorder should patients be followed? Above, we state 'usually early' implying an inception cohort (a group of people who are assembled at an early point in their disease), but clinicians may want information about prognosis in later stages of a target disorder. Thus, a study that assembled patients at a later point in the disease may provide useful information. However, if observations are made at different points in the course of disease for various people in the cohort, the relative timing of outcome events would be difficult to interpret. Thus, the ideal cohort is one in which participants are all at a similar stage in the course of the same disease.
Returning to the paper we found, the study included patients who were entered after their first stroke. Further details on entry procedures aren't included in the study.
Was patient follow-up sufficiently long and complete?
Ideally, we'd like to see a follow-up period for a study that lasts until every patient recovers or has one of the other outcomes of interest, or until the elapsed time of observation is of clinical interest to clinicians or patients. If follow-up is short, it may be that too few study patients will have the outcome of interest, thus providing little information of use to a patient.

The more patients who are unavailable for follow-up, the less accurate the estimate of the risk of the outcome. Losses may occur because patients are too ill (or too well) to be followed or may have died, and the failure to document these losses threatens the validity of the study. Sometimes, however, losses to follow-up are unavoidable and unrelated to prognosis. Although an analysis showing that the baseline demographics of these patients are similar to those followed up provides some reassurance that certain types of participants were not selectively lost, such an analysis is limited by those characteristics that were measured at baseline. Investigators cannot control for unmeasured traits that may be important prognostically, and that may have been more or less prevalent in the lost participants than in the followed-up participants. most evidence-based journals of secondary publication (like ACP Journal Club and Evidence Based Medicine) require at least 80% follow-up for a prognosis study to be considered valid.
In the study we retrieved, follow-up was sufficiently complete and patients were followed from 2 to 6.5 years.
Were objective outcome criteria applied in a "blind" fashion?
We need to assess whether and how explicit criteria for each outcome of interest were applied and if there is evidence that they were applied without knowledge of the prognostic factors under consideration. Blinding is crucial if any judgement is required to assess the outcome because unblinded investigators may search more aggressively for outcomes in people with the characteristic(s) felt to be of prognostic importance than in other individuals. Blinding may be unnecessary if the assessments are preplanned for all patients and/or are unequivocal, such as total mortality. However, judging the underlying cause of death is difficult and requires blinding to the presence of the risk factor to ensure that it is unbiased.
In the study we identified, patients were asked at follow-up if they had a seizure and if they said "yes", a study neurologist subsequently assessed them. It is unclear if the study neurologist was "blind".
If subgroups with different prognoses are identified:
1. Was there adjustment for important prognostic factors?
2. Was there validation in an independent group of "test-set" patients?
We often want to know if patients with certain characteristics will have a different prognosis. For example, are patients with an intracranial hemorrhage at increased risk of seizure? Demographic, disease-specific or comorbid variables that are associated with the outcome of interest are called prognostic factors. They need not be causal but must be strongly enough associated with the development of an outcome to predict its occurrence.

The identification of a prognostic factor for the first time could be the result of a chance difference in its distribution between patients with different prognoses. Therefore, the initial patient group in which the variable was identified as a prognostic factor may be considered to be a training set or a hypothesis generation set. Indeed, if investigators were to search for multiple potential prognostic factors in the same data set, a few would likely emerge on the basis of chance alone. Ideally, therefore, data from a second independent patient group, or a "test set" would be required to confirm the importance of a prognostic factor. Although this degree of evidence has often not been collected in the past, an increasing number of reports are describing a second, independent study validating the predictive power of prognostic factors. If a second, independent study validates these prognostic factors, it can be called a clinical prediction guide.
In the study we found, the investigators looked at patients with different stroke types and identified that patients in these groups had different risks of seizures. This was not tested in an independent group of patients to see if it holds true.

If the study fails any of the above criteria, we need to consider if the flaw is significant and threatens the validity of the study. If this is the case, we'll need to look for another study. Returning to our clinical scenario, the paper we found satisfies all of the above criteria and we will proceed to assessing it for importance.

Are the results of this study important?

How likely are the outcomes over time?

Typically, results of prognosis studies are reported in one of three ways: as a percentage of the outcome of interest at a particular point in time (e.g. 1 year survival rates), as median time to the outcome (e.g. the length of follow-up by which 50% of patients have died) or as event curves (e.g. survival curves) that illustrate, at each point in time, the proportion of the original study sample who have not yet had a specified outcome.

From the study we found, the risk of seizure after any type of stroke is 5.7% at 1 year.

How precise is this prognostic estimate?

The precision of the estimate is best reflected by its 95% confidence interval; the range of values within which we can be 95% sure that the population value lies. The narrower the confidence interval, the more precise is the estimate. If survival over time is the outcome of interest, earlier follow-up periods usually include results from more patients than later periods, so that survival curves are more precise (i.e. have narrower confidence intervals) earlier in follow-up.

To calculate the 95% confidence interval for the study we identified, we can use the following equation:

95% Confidence Interval = p +/- 1.96 × SE

where:

Standard Error(SE) = sqrt(p(1-p)/n)

And 'p' is a proportion of people with the outcome of interest and 'n' is the sample size.

From the study, n = 675 and p = 0.057

SE = sqrt(0.057(1-0.057)/675) = 0.009

Therefore the 95% CI is:

0.057 +/- 1.96 × 0.009 = 3.9% to 7.5%

Where to go from here?

Now that we've decided our article is both valid and important, we need to decide if we can apply it to our patient.

Other options:

Do you want to consider the validity of a prognosis article?
Do you want to see a 'CAT' for this paper?
Do you want to learn about critically appraising:
- Diagnosis articles
- Therapy articles - single trials
- Systematic reviews of therapy articles
- Harm articles
Do you want some practice critically appraising prognosis articles from other clinical specialties?
Do you want more reading about critically appraising prognosis articles?

Prognosis articles from other clinical specialties

Child Health Are children with middle ear disease at increased risk of behavioural problems? Arch Dis Child 1999;80:28-35.
Critical Care Medicine In patients without motor response three days after hypoxic stroke, what is the probability of a good neurologic recovery (cerebral performance category[CPC] 1 or 2)? Edgren E, Hedstrand U, Sutton-Tyrell K, Safar P, and BRCTI study group. Assessment of neurological prognosis in comatose survivors of cardiac arrest. Lancet 1994;343:1055-59.
Gastroenterology and Hepatology In a patient with newly diagnosed Crohn's disease what are the chances of dying prematurely? Loftus EV, Silverstein MD, Sandborn WJ, et al. Crohn's Disease in Olmstead County Minnesota, 1940-1993: Incidence, Prevalence, and Survival. Gastroenterology 1998;114:1161-1168.
General Practice What is the risk of recurrence of ureteric calculus after a single episode in a middle aged male? Kamihira O, et al. Long-term stone recurrence rate after extracorporeal shock wave lithotripsy. J Urol 1996 Oct;156(4):1267-71.
General Surgery In patients admitted as an emergency with complications of diverticular disease, what is the risk of a second emergency admission for further complications? Farmakis N et al. The 5-year history of complicated diverticular disease. Br J Surg 1994;81:733-5.
Geriatric Medicine In patients with a history of stroke, what is the risk of seizure within the first year? Burn J, Dennis M, Bamford J et al. Epileptic seizures after a first stroke: the Oxfordshire community stroke project. BMJ 1997;315:1582-7.
Mental Health In a patient with first episode functional psychosis, what is the likelihood of remaining free from relapse in the long term? Wiersma, D, Nienhuis, FJ, Slooff, C.J. and Giel R. Natural course of schizophrenic disorders: a 15-year follow-up of a Dutch incidence cohort. Schizophr.Bull. 24 (1):75-85, 1998.
Neonatal Medicine In a preterm infant who is expected to survive with posthemorrhage ventriculomegaly, what is the risk of future neurological disability? Aziz K., Vickar D.B., Sauve R.S., et al. Province-Based Study of Neurologic Disability of Children Weighing 500 Through 1249 Grams at Birth in Relation to Neonatal Cerebral Ultrasound Findings. Pediatrics 1995; 95: 837-844.
Nursing In healthy women who have had a miscarriage, what is the usual grieving process and are any factors associated with longer than normal grieving? Janssen HG, Cuisinier MC, de Graauw KP, Hoogduin KA. A prospective study of risk factors predicting grief intensity following pregnancy loss. Arch Gen Psychiatry. 1997;54:56-61.
Physiotherapy Practice In people who have had an open surgical repair to the anterior cruciate ligament, what are the chances of returning to strenuous sport? Andersson C, Gillquist J. Treatment of acute isolated and combined ruptures of the anterior cruciate ligament. Am J Sport Med 1992;20:7-12.