Effect of an Educational Toolkit on Quality of Care: A Pragmatic Cluster Randomized Trial

Summary

Background:
Printed educational materials for clinician education are one of the most commonly used approaches for quality improvement. The objective of this pragmatic cluster randomized trial was to evaluate the effectiveness of an educational toolkit focusing on cardiovascular disease screening and risk reduction in people with diabetes.

Methods and Findings:
All 933,789 people aged ≥40 years with diagnosed diabetes in Ontario, Canada were studied using population-level administrative databases, with additional clinical outcome data collected from a random sample of 1,592 high risk patients. Family practices were randomly assigned to receive the educational toolkit in June 2009 (intervention group) or May 2010 (control group). The primary outcome in the administrative data study, death or non-fatal myocardial infarction, occurred in 11,736 (2.5%) patients in the intervention group and 11,536 (2.5%) in the control group (p = 0.77). The primary outcome in the clinical data study, use of a statin, occurred in 700 (88.1%) patients in the intervention group and 725 (90.1%) in the control group (p = 0.26). Pre-specified secondary outcomes, including other clinical events, processes of care, and measures of risk factor control, were also not improved by the intervention. A limitation is the high baseline rate of statin prescribing in this population.

Conclusions:
The educational toolkit did not improve quality of care or cardiovascular outcomes in a population with diabetes. Despite being relatively easy and inexpensive to implement, printed educational materials were not effective. The study highlights the need for a rigorous and scientifically based approach to the development, dissemination, and evaluation of quality improvement interventions.

Trial Registration: http://www.ClinicalTrials.gov NCT01411865 and NCT01026688

Please see later in the article for the Editors' Summary

Introduction

Diabetes is a common and serious chronic disease associated with impaired quality of life, premature mortality, and significant economic costs [1]–[3]. Patients with diabetes require complex care to manage multiple risk factors such glycemia, blood pressure, and lipids, and to screen for and treat the many complications of the disease. Clinical practice guidelines assist health care providers to deliver this complex care by synthesizing the enormous literature on diabetes management into specific recommendations for care [4]–[6]. However, gaps between guideline recommendations and actual care delivered to patients with diabetes are well documented [7]–[10]. Therefore, the implementation of evidence-based guideline recommendations needs to be improved.

Although multifaceted and intensive quality improvement interventions have been shown to improve processes of care for diabetes patients [11]–[14], they are often difficult to widely implement because of their complexity and cost. In contrast, printed educational materials for clinician education are one of the most commonly used approaches for quality improvement and for increasing guideline adherence, because they are familiar, relatively inexpensive, and easily scalable to reach large populations [15]. A systematic review of methods to improve guideline adherence found that dissemination of educational materials led to an absolute improvement of more than 8% for dichotomous outcomes [15]. Therefore, when the Canadian Diabetes Association (CDA) updated their national clinical practice guidelines for diabetes in December 2008 [5], they created a quality improvement strategy of mailed educational toolkits to family physicians targeting a sequence of key themes from the guidelines. The first theme was cardiovascular disease screening and treatment, since it is the most prevalent complication of the disease and since evidence-based preventative interventions are readily available. The objective of this study was to evaluate the effectiveness of this educational toolkit to improve management of cardiovascular risk factors and outcomes of cardiovascular disease in people with diabetes.

Methods

We conducted a pragmatic cluster randomized controlled trial. A detailed description of the intervention and allocation has been previously published (Text S1) [16].

Ethics Statement

The study was approved by the research ethics board of Sunnybrook Health Sciences Centre, Toronto, Ontario.

Intervention and Allocation

The cardiovascular disease toolkit was a collection of printed educational materials, packaged in a brightly colored box with CDA branding, sent to Canadian family physicians. The contents included an introductory letter from the chair of the practice guidelines' Dissemination and Implementation Committee; an eight-page summary of selected sections of the practice guidelines targeted towards family physicians; a four-page synopsis of the key guideline elements pertaining to cardiovascular disease risk; a small double-sided laminated card with a simplified algorithm for cardiovascular risk assessment, vascular protection strategies, and screening for cardiovascular disease; and a pad of tear-off sheets for patients with a cardiovascular risk self-assessment tool and a list of recommended risk reduction strategies [16]. The toolkit was created for the CDA by clinical experts including family physicians, endocrinologists, and other health care professionals, with guidance from clinicians with expertise in knowledge translation and implementation. The implicit theory behind its development was that the guidelines were too long and complex to be easily incorporated into clinical practice, so the toolkit aimed to simplify the information, tailor it towards clinical practice, and provide explicit actionable recommendations.

While the toolkit was sent to most family physicians in Canada, family practices in the province of Ontario were allocated 1∶1 into the intervention or control group using random number sequences generated by SAS version 9.3 (SAS Institute Inc.), stratified by the 14 health regions into which responsibility for health care delivery in Ontario is divided. Randomization at the practice level helped prevent contamination by ensuring that all patients seen at a single location were assigned to the same study arm. An independent analyst, not otherwise involved with the study, generated the randomized list and provided it to the mailing house distributing the toolkit on behalf of the CDA. The toolkit was mailed in June 2009 to all family practices assigned to the intervention arm, and in May 2010 to practices assigned to the control arm. Physicians were unaware that they were part of a randomized trial.

Data Sources and Study Participants

Two separate studies were conducted. Both studies were registered after trial initiation. In early 2009, the randomization process was conducted to assign family practices to receive the toolkit early (June 2009) or late (May 2010), in the anticipation of acquiring future funding to evaluate the effectiveness of the intervention. Such funding was received in October 2009, and the clinical data study was registered in December 2009, prior to the recruitment of family practices and patients to participate in the study. The administrative data study was registered in August 2011, linked to the earlier registration for the clinical data study. It used data that were routinely collected by the Ministry of Health for the administration of the health care system, not specifically for the purposes of research nor for this study. Analyses of the administrative data were only conducted once these data were received and linked in early 2012.

Administrative data study

A province-wide study was conducted using population-level health care administrative data from the Ontario Ministry of Health and Long-Term Care. Because of the single payer health care system in Ontario, these data cover the entire population with no loss to follow-up. These data are collected routinely for health care administration purposes, independently of this trial. Available data sources included a demographic database; records of all hospital separations and emergency department visits; physician service claims for consultations, assessments, and diagnostic and therapeutic procedures; and prescriptions filled under the provincial formulary, which provides universal drug insurance coverage for all residents aged ≥65 years. Individuals are linked between all of these databases and across time using a unique anonymous identifier.

The study included all residents of Ontario aged ≥40 years who were diagnosed with diabetes as of 1 July 2009. Individuals with diabetes were identified using the Ontario Diabetes Database, a registry of all people with diagnosed diabetes derived from the administrative databases [17]. The ODD has a sensitivity of at least 86% and specificity of 97% when compared to primary care chart abstraction [17]. Residents of long-term care facilities were excluded. Each patient was assigned to their regular family practice [18]. Individuals who could not be assigned to a family practice were excluded.

Clinical data study

Because many important measures of quality of diabetes care are not available from administrative databases, we conducted a separate study in which detailed clinical data were directly collected. We randomly selected practices from each of the intervention and control arms, and randomly selected one physician from each practice. For practical reasons, recruited practices were within 150 km of the study center (67% of all practices in Ontario were within this radius). Each selected physician was contacted, and if willing to participate in the study, we randomly selected 20 diabetic patients who had visited the physician during the study period, and who fulfilled the CDA's definition of being at “high risk for cardiovascular events” [5]: (i) men aged ≥45 years, women aged ≥50 years, or (ii) men aged <45 years and women aged <50 years with at least one of: macrovascular disease; microvascular disease; a history of premature coronary or cerebrovascular disease in a first-degree relative; LDL cholesterol >5.0 mmol/l; systolic BP >180 mmHg; or duration of diabetes >15 years with age >30 years.

Patients were selected using random number sequences generated by SAS version 9.3 (SAS Institute). Their charts were reviewed by a trained and experienced registered nurse, blinded to treatment allocation, who abstracted relevant data into a computerized data collection template. Data elements collected included demographic information, medical history, medication utilization, blood pressure and anthropometric measurements, and laboratory test results.

Outcomes

All patients were followed up for ten months, between July 2009 and April 2010 (i.e., the period when only the intervention group had received the toolkit). Outcomes were ascertained at the patient level.

Administrative data study

The primary outcome was the composite endpoint of death or non-fatal myocardial infarction. Secondary clinical event outcomes included all-cause death; myocardial infarction; myocardial infarction or unstable angina; stroke; stroke or transient ischemic attack; the composite of death, non-fatal myocardial infarction or non-fatal stroke; and the composite of death, non-fatal myocardial infarction, non-fatal stroke, unstable angina, or transient ischemic attack. Because the toolkit particularly highlighted new guideline recommendations on the use of diagnostic testing for coronary artery disease, process measures of care related to this testing were examined, included electrocardiograms, cardiac stress tests or nuclear imaging, coronary angiography, and ambulatory cardiology or internal medicine visits. For patients who were at least 65 years of age and therefore eligible for the provincially funded drug insurance program, drug prescriptions outcomes for cardiovascular risk reduction and treatment were also examined: initiation of an angiotensin-converting enzyme inhibitor or angiotensin receptor blocker; initiation of at least one anti-hypertensive from any class; initiation of medications from at least two anti-hypertensive classes; initiation of medications from at least three antihypertensive classes; initiation of a statin; initiation of any glucose-lowering drug; initiation of insulin; and initiation of nitrates. Prior prescriptions in the previous 10 months were used to separate ongoing prescriptions from initiation.

Clinical data study

In the clinical data study, outcomes were related to cardiovascular risk reduction. The primary outcome was the use (initiation or ongoing) of a statin during the observation period of July 2009 to April 2010. Secondary outcomes included: use of an angiotensin converting enzyme inhibitor or angiotensin receptor blocker; HbA1c≤7.0%; blood pressure ≤130/80; LDL cholesterol ≤2.0 mmol/l; and total- to HDL-cholesterol ratio ≤4.0. We also examined “clinical inertia” outcomes, where a measurement clearly above established treatment targets did not elicit a change in the patient's treatment at either the current or subsequent visit [19]. We examined changes in glucose-lowering treatments following an HbA1c >8.0%; changes in blood pressure-lowering treatment following a blood pressure >140/90; and changes in lipid-lowering treatment following an LDL cholesterol >3.0 mmol/l. In all cases, either pharmacological or non-pharmacological treatment changes were considered as absence of clinical inertia.

Analysis

The statistical analysis was completed on an intention to treat basis. The analysis accounted for the cluster randomized design by using logistic regression models estimated using generalized estimating equation (GEE) methods to assess the statistical significance of the intervention's effect on each outcome. Explanatory variables in all models were the randomization arm (the variable of interest) and diabetes patient volume as practice-level variables, and age, sex, diabetes duration, and previous cardiovascular disease (defined from hospitalization records in the previous 5 years in the administrative data study, and from records in the patient's chart in the clinical data study) as patient-level variables. Data were analyzed using SAS version 9.3 (SAS Institute Inc.). All tests were two-sided with p-values<0.05 denoting statistical significance. Although the administrative data study used the entire population rather than a sample and therefore p-values do not apply, the clinical data study was a sample of the population and so for consistency we have included p-values for both studies.

Power and Sample Size

Administrative data study

The cohort included in the administrative data study was the entire population aged ≥40 years with diagnosed diabetes in Ontario, which was more than 900,000 people. With this fixed sample size, the study had >95% power to detect an unadjusted absolute difference of at least 0.4% in a dichotomous primary outcome, using an α-error of 0.05. Power would be reduced after adjustment for baseline differences and for clustering, but would remain sufficient to detect even very small differences in outcomes.

Clinical data study

The sample size for the clinical data study was calculated a priori on the basis of an ability to detect an absolute 10% difference in statin prescription rates between intervention and control patients, a threshold similar to the median effect size found in a systematic review of printed educational materials [15]. To have 80% power to detect this difference with an α-error of 0.05, a sample size of 796 per group with 20 patients per practice was needed [16].

Results

We randomized 4,007 family practices in Ontario, to which 933,789 patients with diabetes were assigned in the administrative data study (Figure 1). In the clinical data study, we approached 372 intervention practices and 395 control practices to recruit 40 of each, and we examined the records of 1,592 randomly selected patients at high risk for cardiovascular disease. The pre-specified sample size was met. The baseline characteristics of patients and practices were well balanced (Table 1).

Administrative Data Study

The primary outcome, death and non-fatal myocardial infarction, occurred in 11,736 (2.5%) patients in the intervention group, and 11,536 (2.5%) patients in the control group (odds ratio 1.00, 95% confidence interval 0.96–1.03, p = 0.77). The other clinical event outcomes did not show benefit with the intervention (Table 2). Two of the processes of care secondary outcomes related to coronary artery disease screening were statistically significantly worse in the intervention group: electrocardiograms (38.8% versus 40.2%, odds ratio 0.96, 95% confidence interval 0.93–0.99, p = 0.02) and cardiac stress tests or nuclear imaging (7.8% versus 8.1%, odds ratio 0.96, 95% confidence interval 0.93–1.00, p = 0.04). The intervention did not significantly increase initiation of medications targeting cardiovascular risk factors or glycemia. Fewer patients in the intervention group were started on nitrates than in the control group (odds ratio 0.96, 95% confidence interval 0.92–1.00, p = 0.03).

Clinical Data Study

Seven hundred (88.1%) patients in the intervention group and 725 (90.1%) patients in the control group used a statin (odds ratio 0.73, 95% confidence interval 0.42–1.26, p = 0.26). Likewise, there was no difference between groups in the proportion of patients receiving an angiotensin converting enzyme inhibitor or angiotensin receptor blocker (Table 2). Fewer patients in the intervention group reached blood pressure control targets (52.8% versus 63.5%, odds ratio 0.72, 95% confidence interval 0.53–0.98, p = 0.04). There was no difference in the proportion reaching glycemic and LDL-cholesterol control targets. Clinical inertia also did not differ between groups. An exploratory analysis also adjusting for practice type (Table S1) gave virtually identical results.

Discussion

Printed educational materials in the form of a cardiovascular disease toolkit did not improve quality of care or outcomes in a population with diabetes. Some of the secondary outcomes were statistically significantly worse in the intervention group, although we cannot exclude that these represent chance findings due to multiple hypotheses testing. Thus, this quality improvement intervention not only failed to improve care, but if anything may have adversely affected some of the secondary outcomes.

The study has several strengths to highlight. It used a pragmatic randomized trial design to obtain a valid estimate of the effectiveness of the toolkit in real world clinical care, not just in an idealized efficacy trial setting. By randomizing in clusters at the practice level, the risk for contamination was minimized. The study included nearly 1 million patients—the entire population of patients with diabetes in Canada's most populous province—making it the largest ever randomized trial in diabetes. Using administrative data sources to evaluate outcomes ensured complete data collection with no loss to follow-up or missing data. The lack of detailed clinical data available in these data sources was supplemented with data collected directly from patient charts in the clinical data study. However, there are some possible limitations to note. First, the toolkit was developed without a specific quality improvement or educational theory to guide its content or delivery, which might otherwise have increased the likelihood of it leading to clinically important improvements in care [20]. Although the toolkit addressed clinician knowledge, other barriers to implementation of evidence-based care (such as time, patient adherence or comorbidity, or organization of health care delivery) were not addressed [21]. Second, the follow-up time of the study was short, which may have been insufficient time in which to see changes in clinical events. However, process measures and drug prescriptions could very easily be influenced within the short time frame of the study, and yet the intervention had no impact on these secondary outcomes as well. Third, although practices participating in the clinical data study were randomly selected, they may have been subject to volunteer bias as those who agreed to participate could be different from the large number who did not. However, it is unlikely that this would influence the validity of the study, as such bias would have affected both intervention and control practices equally. Finally, in the clinical data study, utilization of statins and of angiotensin converting enzyme inhibitors or angiotensin receptor blockers were both >85%, even in the control group. Thus, utilization may already have reached a ceiling, beyond which further improvements from an intervention would be unlikely.

The possibility that the intervention might have worsened care is unexpected. It is of course possible that the four statistically significant findings showing harm were chance findings due to multiple hypothesis testing, but notably there were no results in the opposite direction. Therefore, we feel it necessary to speculate on possible explanations for why this intervention could have worsened care, should these findings be real. Although its content was focused on cardiovascular disease, the toolkit was disseminated by the Canadian Diabetes Association and it was temporally linked with the newly released national diabetes practice guidelines. Hence, the intervention may have inadvertently focused the attention of physicians onto other aspects of diabetes care, and away from broad cardiovascular disease risk management, which might have been perceived of as a “mere” comorbidity of diabetes. Lack of attention to cardiovascular risk factor management would be detrimental to patient outcomes, especially since cardiovascular disease is the most common cause of death for diabetic patients [2],[22],[23]. These findings highlight the importance of monitoring for unintended consequences of quality improvement interventions [24].

The previous literature has demonstrated that the benefits of printed educational interventions are, at best, modest. A systematic review of methods to improve practice guideline adherence demonstrated an absolute improvement of 8% for educational materials [15]. A more recent Cochrane review found that printed educational materials led to a median absolute improvement in performance of only 2% [25]. A diabetes-specific review of quality improvement strategies to improve glycemic control showed that clinician education interventions had the lowest impact of any quality improvement approach, with a mean reduction in HbA1c of 0.16% [26]. Studies of printed materials specifically tied to clinical practice guidelines also showed modest benefits. A small English study randomized 42 family physicians to receive an algorithm for monitoring and treatment of hypertension of diabetic patients based on practice guidelines, but found no difference in blood pressure control between the intervention and control groups [27]. However, some processes of care were slightly improved: patients in the intervention group were prescribed higher doses of antihypertensive medications, and had more physician visits to monitor blood pressure. In a larger Canadian study, family physicians were randomized to receive a one-page summary of a three-year-old practice guideline on anti-anginal therapy from the local medical governing body [28]. No differences were noted in prescription of β-blockers, antiplatelet agents, or lipid-lowering drugs between groups in the 7,000 patients reviewed. We had hoped that the intervention in our study might have had greater impact because it was produced and disseminated by the CDA, an advocacy and professional organization with credibility among family physicians for diabetes messaging, and because the intervention was released in conjunction with a well-publicized update to national diabetes practice guidelines, increasing the salience of the intervention and its content. However, these differences of this intervention from those previously studied did not result in improved outcomes.

In summary, the cardiovascular disease management toolkit failed to reduce clinical events or improve quality of care for patients with diabetes. It may actually have led to worsening in some secondary outcomes, though this may represent a chance finding. Despite years of evidence that printed educational materials have, at best, only a modest impact on quality of care, they remain a very commonly used intervention for quality improvement. Therefore, even the remote potential for them to cause true harm is worrying, as the public health consequences could be significant. Quality improvement interventions are often developed by clinicians or policy makers in an unsystematic way. The results of this study highlight the need for a rigorous and scientifically based approach to the development, dissemination, and evaluation of quality improvement interventions.

Supporting Information

Attachment 1

Attachment 2

Attachment 3

Attachment 4