Generalizing findings from a randomized controlled trial to a real-world study of the iLookOut, an online education program to improve early childhood care and education providers’ knowledge and attitudes about reporting child maltreatment

Summary

In recent years, real-world studies (RWS) are gaining increasing interests, because they can generate more realistic and generalizable results than randomized controlled clinical trials (RCT). In 2017, we published a RCT in 741 early childhood care and education providers (CCPs). It is the Phase I of our iLookOut for Child Abuse project (iLookOut), an online, interactive learning module about reporting suspected child maltreatment. That study demonstrated that in a RCT setting, the iLookOut is efficient at improving CCPs’ knowledge of and attitudes towards child maltreatment reporting. However, the generalizability of that RCT’s results in a RWS setting remains unknown. To address this question, we design and conduct this large RWS in 11,065 CCPs, which is the Phase II of the iLookOut. We hypothesize replication of the earlier RCT findings, i.e., the iLookOut can improve CCPs’ knowledge of and attitudes toward child maltreatment reporting in a real world setting. In addition, this RWS also explores whether demographic factors affect CCPs’ performance. Results of this RWS confirmed the generalizability of the previous RCT’s results in a real world setting. It yielded similar effect sizes for knowledge and attitudes as were found in the earlier RCT. Cohen’s d for knowledge improvement was 0.95 in that RCT, 0.96 in this RWS; Cohen’s d for attitude improvement was 0.98 in that RCT, 0.80 in this RWS. Also, we found several significant differences in knowledge and attitude improvement with regard to age, race, education, and employment status. In conclusion, iLookOut improves knowledge and attitudes of CCPs about child maltreatment prevention and reporting in a real-world setting. The generalizability of the initial RCT findings to this RWS provides strong evidence that the iLookout will be effective in other real world settings. It can be a useful model for other interventions aimed at preventing child maltreatment.

Clinical trial registration for the original RCT: NCT02225301 (ClinicalTrials.gov Identifier)

Keywords:

teachers – Employment – Jobs – Internet – Religion – Randomized controlled trials – Child abuse – Pennsylvania

Introduction

While randomized controlled trials (RCT) have long been seen as the “gold standard” for evaluating the efficacy of interventions, there are well-known limitations to their generalizability [1]. Accordingly, there have been growing interests in real-world studies (RWS) to generate real-world evidence (RWE) that are more realistic and generalizable [2–9], and RWE is increasingly valued by regulators and payers [10]. In addition, RWE and the RCT can happily co-exist and complement each other [9].

Recently, we published data from an RCT about the online educational intervention, the iLookOut for Child Abuse (iLookOut), showing that it improved early childhood care and education providers (CCPs) knowledge and attitudes about child maltreatment and its reporting [11]. In this follow-up study, through an RWS, we evaluate whether these results are generalizable to a broad population of CCPs in a real-world setting.

There are more than 675, 000 confirmed cases of child maltreatment annually in the United States [12], but less than 1% of these are reported by CCPs (U.S. DHHS, 2017). This extremely low report rate by CCPs is alarming, given the fact that about 12 million U.S. children are served in some form of a child care setting, that children five years-old or younger account for 46% of confirmed maltreatment and more than 75% of maltreatment-related deaths (U.S. DHHS, 2017), and that the true incidence of child maltreatment is likely much higher than currently detected [13, 14]. Such underreporting suggests a need for CCPs to become better prepared to protect young children from maltreatment by improving their knowledge and attitude towards child maltreatment reporting. As has been identified by the Institute of Medicine and others, a key obstacle to improving awareness and reporting is the lack of evidence-based interventions [15–17]. In addition, the US Preventive Services Task Force (USPSTF) recently called for more evidence-based primary care interventions to prevent child maltreatment [12]. Several small studies have evaluated in-person training for CCPs [18, 19], and a brief online intervention [20, 21]. However, large studies involving scalable interventions are still lacking.

To meet this need, we created iLookout, an interactive online learning program designed specifically for CCPs (https://ilookoutproject.org/). An initial RCT using a test and re-test design with 741 participants demonstrated the feasibility of this three-hour online training, as well as its efficacy at increasing knowledge and changing attitudes about child maltreatment and its reporting [11]. Though this initial trial was promising, with large Cohen’s d effect sizes for knowledge (0.95) and attitudes (0.98), its generalizability was limited by several factors, notably the potential for selection bias. Participants were enrolled only if the director of the child care program responded to the recruitment mailing. Family- and home-based CCP programs were under-represented, as were racial and ethnic minorities. In addition, enrollment was limited to a four-week period in early summer. Also, the sample size limited the opportunity for in-depth comparisons among subgroups.

To address these limitations, the present RWS used a statewide, open-enrollment design to enlist a larger, more representative sample of CCPs. We hypothesized that iLookOut’s efficacy at increasing knowledge and attitudes would be confirmed in this real-world sample, and our exploratory aim was to evaluate the impact of key demographic characteristics.

Materials and methods

Design

The Penn State College of Medicine Institutional Review Board approved this study prior to its initiation (IRB #: 1243). This RWS employed an open enrollment, single group, pre- and post-test design. Participants completed a demographic questionnaire, as well as previously validated knowledge and attitude measures regarding child maltreatment and its reporting [11]. Given the observational feature of this RWS, we have ensured that the manuscript adheres to the appropriate Equator Network guidelines, such as the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) Statement [22].

Participants

As an open-enrollment RWS, participants were not actively recruited to this study. However, all mandated reporters in Pennsylvania (including CCPs) are required by law to complete a mandated reporter training, and iLookOut was one of more than a dozen state-approved trainings listed on Pennsylvania’s Department of Human Services website, and was available online at no charge. As such, online searches and word of mouth were the means for dissemination. Participant data reported here are from CCPs who completed iLookOut between January 2015 and March 2018. CCPs provided online informed consent prior to participating, and earned three hours of professional development credit for completing the learning program. No other incentives or remuneration were provided.

Intervention

The iLookOut online learning program uses an interactive, video-based storyline in which the learners take the role of a teacher of 4–5 year-olds at a child care facility. As key events unfold through interactions involving children, parents, and co-workers (all played by actors), the learners have to decide how to best respond. At different points, learners are posed questions. Based on their answer, they are provided didactic material to educate them about various aspects of child maltreatment. Other times, the learners must choose how to respond to events in the story. Throughout the learning program, CCPs can access multiple resource files covering definitions of maltreatment, facts about maltreatment, red flags, etc.[11].

Measures

The pre- and post-test comprise two parts. The first is a 21-item, true or false, expert-validated instrument previously described [11]. It measures individuals’ knowledge about what constitutes child maltreatment, risk factors for maltreatment, and legal requirements for reporting suspected maltreatment. Correct answer to each of the 21 true or false items is scored as 1 point, and wrong answer is scored as 0 point. Therefore, the total score of the knowledge scale ranges from 0 to 21, which higher score representing more knowledge about child maltreatment. The second part contains 13 items, rated on 7-point Likert-style scales, from a previously validated instrument [23] adapted to comport with Pennsylvania jurisdictional standards. It measures individuals’ attitude towards reporting potential child maltreatment. An individual’s attitude score is the average score of the 13 items, ranges from 1 to 7, with higher score representing more positive attitude towards reporting potential child maltreatment. The pre- and post-test question items were identical, but to minimize recall bias, their sequencing orders were changed between the pre- and the post-test.

Sample size and statistical analysis

Given the RWS nature of this study, no a priori sample size estimation was planned. However, post-hoc power analyses were implemented to check the statistical power for some important subgroup analyses [4]. We also compared participant demographics between the initial RCT and this RWS.

As with the RCT, the statistical analysis of this RWS examined iLookOut’s impact on CCPs’ knowledge and attitudes related to child maltreatment and its reporting. The two primary outcome variables were the total knowledge score and the total attitude score, both measured as “change”, i.e., total score at post-test minus at pre-test. The analysis focused on whether the present RWS confirmed the results of the initial RCT. To compare effect sizes between the RCT and the RWS, we used two measures: 1) the absolute difference, i.e., the measured change in pre- to post-test score for the RWS, minus the measured change in initial RCT; and 2) the Cohen’s d calculation [24]. In addition, we explored the impact of demographic factors on these two primary outcome variables through analysis of covariance (ANCOVA), framing demographic variables as covariates, and adjusting for pre-measurement scores. These demographic variables include age, gender, race/ethnicity, education, employment, parent/guardian status, prior trained status, work environment, years as practitioner, primary job responsibilities, and religiosity. We used the SAS software package, version 9.4, for statistical analyses, and the G*Power software package, version 3.1.9, for post-hoc power analyses.

Results

During the 38 months of the RWS reported here, 11,605 CCPs completed the iLookOut online training. Compared to those CCPs in the initial RCT, these RWS participants were more representative of the general population of CCPs in Pennsylvania, particularly for its enrollment of Blacks (20.8% vs. 8.0%) and males (10.9% vs. 2.3%). In addition, the CCPs in this RWS were younger (48.0% vs. 40.4% aged below 30), and a greater proportion worked in more urban area (36.4% vs. 22.1%). Table 1 illustrates comparisons of full demographics between these two studies.

Table 2 illustrates comparisons of the iLookOut training’s effect sizes on knowledge and attitude scores between this RWS and the RCT, demonstrating improved knowledge and attitudes about child maltreatment reporting for both studies. Pre- to post- changes in knowledge score increased by 2.80 for RWS participants, compared to 2.65 in the initial RCT, a 5.7% relative change. The Cohen’s d on the total knowledge score was 0.96 in this RWS versus 0.95 in the RCT, a 1% relative change. The pre-to post- change in attitude average score was 0.5 for RWS participants, versus 0.59 in the initial RCT, a -15.3% relative change. The Cohen’s d on the average attitude score was 0.80 in this RWS, versus 0.98 in the RCT, a relative change of -18.4%.

Table 3 summarizes the results of exploratory multivariate analyses (ANCOVA) for each of the two outcome variables (knowledge and attitude scores) with all of the demographic variables. After adjustment for pre-measurement scores and all the other demographic variables, only four demographics (age, race, education, and employment) showed impacts on either of the two outcome variables, with age and education being positively correlated with increase in knowledge scores.

Post hoc power analysis indicates that with a total sample size of 11,065, at an alpha level of 0.05, with 80% power, the ANCOVA with 11 covariates would be able to detect an effect size as small as 0.04 among six groups; and using an effect size cut-off of 0.25, then the power would approach to 99.5%.

Discussion

The results from this RWS demonstrate that in a large, representative sample of child care professionals (CCPs), the online iLookOut learning program is effective at improving knowledge and changing attitudes about child maltreatment and its reporting. These findings confirm the conclusions from the initial RCT of iLookOut, and demonstrate the feasibility of scaling this evidence-based, online mandated reporter training. This is notable insofar as more than 11,000 CCPs completed iLookOut, even when no special incentives were offered, and they reported being highly satisfied with the learning experience (paper forthcoming). No significant differences were identified with regard to CCPs’ parenting status, previous training, work environment, years as practitioner, primary job responsibility, or religiosity. However, age, race, education, and employment affected changes in knowledge or attitude scores, with older and more educated CCPs achieving increased gains in knowledge scores.

The generalizability of the initial RCT findings provides supporting evidence that the iLookout online learning program will be effective in other real world settings, and may be a useful model for other interventions aimed at preventing child maltreatment [12]. iLookOut’s general storyline and overall format are generalizable for all kinds of CCPs in all U.S. states, in part because state-specific information is housed in discrete learning modules (within the learning program) that can be readily adapted to comport with the laws and policies of different states. The efficacy of iLookOut does not appear to be affected by previous training, work environment, years as practitioner, primary job responsibility, parenting status, or religiosity. However, larger gains in knowledge were seen in CCPs who were older, more highly educated, employed seasonally, or white. More research is warranted to better understand the underpinnings of these differences, and how best to optimize gains in knowledge for all CCPs.

The statistical analyses reported here focus on effect sizes, instead of p-values, for several reasons. First, p-values are not a good measure of evidence [25]. Second, the misuse and maltreatment of p-values has led both researchers and the American Statistical Association to raise concerns about the limitations of p-value-driven conclusions [26, 27]. Third, the very large sample size (over 11,000) of this RWS could yield findings of statistical significance for even very small effect sizes that have no clinical significance [28]. Fourth, the large difference in sample size between the initial RCT and this RWS renders effect sizes a more meaningful comparison than p-values. Finally, for proposed sub-group analyses involving many demographic covariates, p-values are less likely to yield meaningful findings [29]. Accordingly, we compared effect sizes by examining the overlap of their confidence limits.

The present findings are limited by potential biases encountered in all RWS, including selection bias, information bias, and confounding [3]. Multivariate analysis (ANCOVA) was used to try to account for these factors, and the initial RCT does provide additional reassurance that the present findings are valid. However, without qualitative data, an explanatory model for the present findings will remain incomplete.

Conclusion

This real-world study of more than 11,000 early childhood professionals (CCPs) who were neither recruited nor incentivized to complete the iLookOut for Child Maltreatment confirms that iLookOut significantly improves knowledge and attitudes regarding child maltreatment and its reporting. These results provide strong evidence that interactive, online interventions for helping prevent child maltreatment are both effective and scalable. A 5-year randomized controlled trial (https://clinicaltrials.gov/ct2/show/NCT02225301?term=NCT02225301&rank=1) is currently underway to evaluate how well iLookOut helps CCPs identify and report true child maltreatment.

Supporting information

S1 Dataset [csv]