Blood, Vol. 94 No. 5 (September 1), 1999:
pp. 1515-1516
REVIEW ARTICLE
The Evidence-Based Analysis of Treatment for Chronic Myeloid
Leukemia: An Introduction to Its Methods and Clinical Implications
By
James N. George,
Steven H. Woolf, and
Gary E. Raskob
From the Departments of Medicine and Biostatistics and Epidemiology,
University of Oklahoma Health Sciences Center, Oklahoma City, OK; and
the Department of Family Practice, Medical College of Virginia, Fairfax
Family Practice Center, Fairfax, VA.
 |
ARTICLE |
IN THIS ISSUE OF BLOOD, a panel
sponsored by the American Society of Hematology (ASH) publishes its
analysis of data on selected modalities used for treatment of the
chronic phase of chronic myeloid leukemia (CML).1 The text
is long, and the data presented are complex and arrayed in elaborate
tables. The strengths and weaknesses of the scientific evidence are
discussed in detail, and the recommendations are carefully worded. All
of these features make it a complex document for the reader. However, each of these features is critical to the goal of this project: to
achieve an objective assessment of the effectiveness and risks of the
available treatments.
Systematic reviews featuring this comprehensive approach to evaluate
the literature have become a cornerstone of "evidence-based medicine."2,3 The benefit of systematic,
evidence-based reviews is that they inform practitioners about the
quality of evidence supporting what we do in medicine. In the case of
CML, as is often the case, widely held beliefs about the effectiveness,
superiority, and safety of treatments are challenged when exposed to
critical appraisal of the evidence. Clinicians can continue to use
these treatments, but they do so with a better understanding of the scientific basis. Systematic reviews aid in practice guideline development by critically appraising and summarizing the results of
primary research.4 Identifying gaps in the evidence is also useful to researchers, and to funding agencies, to help set priorities for future research.
Evidence-based practice guidelines have some limitations. Good evidence
is lacking for much of what is done in medicine, yet clinicians need
practical advice on how to manage patients even when high-quality
evidence is lacking. Evidence-based reviews are relatively expensive,
take months to years to complete, and may become outdated when new
evidence becomes available. Perhaps of greatest concern to
practitioners, the conclusions of evidence-based reviews and guidelines
can be harmful if they are cited by health plans or government agencies
as grounds for withholding coverage and referrals, or if they are used
unfairly for judging the quality of care or for malpractice litigation.
Some physicians believe that these risks argue against participating in
guideline development, while others contend it is more dangerous to
allow forces outside the specialty to write and impose guidelines.
These issues took center stage in the work of the CML panel. The report
it produced1 is the product of 3 years of intensive effort
by an international panel of experts, which conducted a systematic
search of the literature, using standardized methods to judge the
quality of over 200 studies. The data from this analysis are contained
in seven tables. Table 1 is a statement of predefined rules that guided
the recommendations. Tables 2 through 7 contain a summary of the
published data. Readers should appreciate both the scientific
limitations of the literature, which includes uncontrolled observational studies, as well as the absence of key data in many of
the cited publications. For example, many studies do not report fundamental data on their patient population, such as age, Philadelphia chromosome status, and duration of follow-up. Some data on long-term survival are based on observations on very few patients. These features
help to explain the extreme variation between studies in the reported
survival benefits and the incidence of toxicities and complications of
interferon treatment and allogeneic bone marrow transplantation (BMT).
The methodologic limitations in the evidence had different meaning for
panel members, some of whom were the lead authors of the studies under
review. Proponents of particular treatments, who believed strongly in
their value, were less concerned about these limitations or the absence
of evidence from randomized controlled trials than were panel members
who served as "methodologists." When evidence was lacking, some
panel members felt a duty to offer clinicians firm recommendations
anyway, even if based on opinion, whereas others believed that it was
preferable to avoid such recommendations and to state only that there
was insufficient evidence. Some panel members believed that the
evidence should drive the recommendations, while others preferred to
adjust the guidelines to reflect current practices and opinion. Much of
the 3 years involved in the production of the CML guideline was spent
arbitrating these disagreements.
Some of these difficulties reflected the composition of the panel,
which consisted largely of CML investigators. Some guideline panels of
other specialty societies and government agencies intentionally limit
the participation of renowned experts and of authors of the studies
under review to avoid these difficulties and to allow reviewers to
approach the evidence free of preconceived viewpoints. Guideline panels
of the American College of Physicians, US Preventive Services Task
Force, and other groups, which review evidence for topics that cut
across multiple specialties, are composed of individuals who are
skilled in the critical appraisal of evidence but who may not even
belong to a relevant specialty. Critics of this approach worry that
unfamiliarity with the topic and lack of experience with direct care of
patients with the condition limits the capacity to produce clinically
relevant guidelines. Sensitive to this concern, Dr Richard Silver, the
Chair of the CML panel, chose leading clinical experts to serve as
members, but he also appointed a statistician and a practice guideline
methodologist. This set the stage for an interesting dilemma. Many of
the studies reviewed by the CML panel were considered excellent by the
clinical experts but of poor quality by the clinical research
methodologists. Whose views of the evidence should prevail? Experts on
CML, many of whom had conducted the studies, surely knew the data best.
On the other hand, having conducted the studies, could they objectively
review their own work? Epidemiologists and biostatisticians may be more dispassionate and are probably more qualified to evaluate the validity
of Kaplan-Meier survival curves and statistical power calculations.
But, without background in CML, can they fully interpret all the
information in the reports? Without having treated CML patients, should
they influence the recommendations?
What is the practicing hematologist to make of all of this? What is the
value of the CML review (and of other evidence-based reports) in the
real world of patient care, where decisions cannot be postponed because
of limited data? Clinical decisions are based on multiple factors, only
one of which is the evidence from clinical research. Practitioners of
evidence-based medicine emphasize three key elements of clinical
decision-making: clinical experience, research evidence, and patient
preferences.5 Thus, beyond research evidence, choices about
BMT, interferon, or other options for patients with CML are influenced
by clinical experience, including assessment of the patient's history
and risk factors, as well as by patient expectations and preferences,
and by restrictions imposed by health care systems or third-party
payers. To the extent that clinical decisions incorporate scientific
evidence, systematic reviews serve a critical purpose by casting a
spotlight on the strength and weaknesses of the evidence. Clinicians
must still consider other factors in clinical decision-making, but with
more information on the quality of the research evidence, they are better able to correctly inform their patients about the benefits and
risks of the available treatment options.
The fact that current data do not allow more definitive recommendations
is perhaps the most important finding of the CML analysis. This is an
important message for both clinicians and researchers about the
limitations of existing clinical research evidence on CML treatment.
With a few notable exceptions, the CML panel found that the studies
lacked basic documentation. Inclusion criteria were vague and applied
inconsistently. Sample sizes were generally small, providing inadequate
statistical power to evaluate differences in outcomes, and wide 95%
confidence intervals for the estimates of treatment effect. Treatment
protocols were not adhered to systematically, with large drop-out
rates. These deficiencies pose a serious problem as the health care
system applies increasing scrutiny to the quality of evidence.
Interventions that are not supported by well-designed studies are less
likely to be paid for. The findings of the CML report are a clarion
call to researchers to use rigorous study designs and to observe
greater scientific diligence in data collection and adherence to protocol.
The CML review brings into sharp focus the fact that all treatment
options for CML involve difficult tradeoffs between benefits and harms,
and that the choice of which option is "best" is influenced by
subjective value judgments. Reflecting a growing trend in practice guidelines,6 the CML report argues that the patient should have the opportunity to decide for himself or herself, based on personal preferences, which option is best. It is a recommendation that
some CML panel members, accustomed as experts to telling patients what
is best, found discomforting. Yet the era of "shared decision-making" is upon us,6,7 as empowered patients
informed by the Internet and other resources seek more knowledge about their options, and play an active role in decisions about their care.
The section of this document outlining recommendations required the
most effort and greatest diplomacy. For the reader, it represents the
best interpretation of the current research evidence. Evidence from
randomized controlled trials demonstrates that treatment with
interferon is superior to treatment with hydroxyurea, but there are
more side effects with interferon. There are no randomized controlled
clinical trials to document the comparative risk and benefit of BMT.
Observational studies suggest that BMT does increase the probability of
long-term survival, but also document a significant short-term risk of
complications or death from the procedure. These observational studies
involve significant patient selection. Therefore, at present,
definitive conclusions cannot be made about the relative benefit and
risk of BMT versus interferon in patients with CML. The trade-offs
between higher early mortality with BMT but a potential long-term
survival advantage, as well as the side-effects of interferon therapy,
underscore the importance of incorporating patient preferences into the
clinical decision about treatment options. The evidence-based analysis
by the CML panel makes a major contribution by bringing these issues
into focus. There is an urgent need for a definitive randomized
clinical trial, incorporating sufficient long-term follow-up, to
evaluate the relative benefits and risks of BMT versus the most
promising alternative treatment in patients with CML. The report of the
CML panel creates sufficient "clinical equipoise"8
to undertake such a trial.
| |
FOOTNOTES |
Address reprint requests to James N. George, MD, Chief,
Hematology-Oncology Section, The University of Oklahoma Health Sciences
Center, PO Box 26901, Oklahoma City, OK 73190; e-mail: jim-george{at}ouhsc.edu.
| |
REFERENCES |
1.
Silver RT, Woolf SH, Hehlmann R, Appelbaum FR, Anderson J, Bennett C, Goldman JM, Guilhot F, Kantarjian HM, Lichtin AE, Talpaz M, Tura S:
An evidence-based analysis of the effect of busulfan, hydroxyurea, interferon, and allogeneic bone marrow transplantation in treating the chronic phase of chronic myeloid leukemia: Developed for the American Society of Hematology.
Blood
94:1517, 1999[Abstract/Free Full Text]
2.
Cook DJ, Mulrow CD, Haynes RB:
Systematic reviews: Synthesis of best evidence for clinical practice.
Ann Intern Med
126:376, 1997[Abstract/Free Full Text]
3.
Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS:
Evidence based medicine: What it is and what it isn't.
Br Med J
312:71, 1996[Free Full Text]
4.
Cook DJ, Greengold NL, Ellrodt AG, Weingarten SR:
The relation between systematic reviews and practice guidelines.
Ann Intern Med
127:210, 1997[Abstract/Free Full Text]
5. Haynes RB, Sackett DL, Gray JMA, Cook DJ, Guyatt GH: Transferring
evidence from research into practice: 1. The role of clinical care
research evidence in clinical decisions. Ann Intern Med ACP Journal
Club Nov/Dec A-14, A-15, 1996
6.
Woolf SH:
Shared decision-making: The case for letting patients decide which choice is best.
J Fam Pract
45:205, 1997[Medline]
[Order article via Infotrieve]
7.
Laine C, Davidoff F:
Patient-centered medicine: A professional evolution.
JAMA
275:152, 1996[Abstract/Free Full Text]
8.
Passamani E:
Clinical trials
Are they ethical?
N Engl J Med
324:1589, 1991[Medline]
[Order article via Infotrieve]
TOP
ARTICLE
REFERENCES