Effects of Topper Training on psychosocial problems, self-esteem, and peer victimisation in Dutch children: A randomised trial

Summary

Most interventions aimed at improving social interactions either target internalising or externalising problem behaviour in children. However, a recent review shows that a transdiagnostic approach might fit better to the diversity of problems within a group and within an individual (comorbidity). We examined the effectiveness of a transdiagnostic intervention, called Topper Training: a cognitive behavioural intervention in the peer group with parents included, that targets both internalising and externalising behaviour problems. A randomised trial with a waiting list control group was conducted, using 132 children with mild to severe psychosocial problems. Children were randomised into 77 intervention and 55 waiting list children (50% boys; age = 8–11 years). GLM repeated measures analyses yielded significant intervention effects directly after the training on parent-reported (but not teacher-reported) emotional symptoms (Cohen’s d = .70), peer relationship problems (d = .41), and impact of these problems (d = .59). Significant effects were also found for child-perceived peer victimisation (d = .62), self-esteem (d = .45) and teacher-reported conduct problems (d = .42). Parent-reported effects on emotional, conduct problems and impact of the problems and child-reported effects on self-esteem were clinically relevant. No significant effects of Topper Training were found for prosocial behaviour and bullying. Within-participant t-tests in the intervention group between post-intervention and follow-up indicated that effects extended over a six-month follow-up period. Depression decreased significantly from post-test to follow-up. In conclusion, children with mild to severe internalising and/or externalising problems can benefit from the transdiagnostic Topper Training intervention.

Keywords:

teachers – Psychological and psychosocial issues – Behavior – Schools – children – emotions – Prosocial behavior – Parenting behavior

Introduction

Children spend a lot of time interacting with other children and this is not an easy job for all of them. Children can show aggressive reactions but also depressive and withdrawn reactions to daily life challenges such as trying to belong to a group, bullying, denial or other social situations [1]. To a certain level, these challenges belong to normal development. From a biopsychosocial perspective [2], the interaction of biological, psychological and social aspects can create vulnerability for children in certain environments to develop problems. When these interaction processes continue, the first symptoms or psychosocial problems can develop. With psychosocial problems we mean emotional (or internalising), conduct (or externalising) and social peer problems, following the definition of Theunissen [3]. The prevalence of psychosocial problems in Dutch 8- to 12- year old children is 10% [4]. More specific: 8% of Dutch primary school children shows conduct problems and 12% shows emotional problems (parent report) [5]. Early conduct and emotional problems are found to be important predictors of depression, delinquency, school dropout and psychological disorders later on in life [6]. Reducing these problems at an early age with indicative preventive interventions directed at psychosocial problems may prevent escalation into severe problems that are harder to treat [7] and save society from the associated costs and risks [8].

Many of the interventions directed at social interactions target one kind of problem behaviour: either internalising or externalising problems. In a recent review, Marchette and Weisz [9] argue that there is a mismatch between this focal treatment on single problems and treatment of children in real-world clinical care. Children very frequently have comorbid problems and are thus diagnostically heterogeneous. This notion is in line with findings of Caspi et al. [10]. Their study indicated that psychiatric disorders could best be explained using one general psychopathology factor: the p factor. They argued that this p factor makes it difficult to find strongly effective treatments for individual mental disorders. Thus, working with transdiagnostic approaches may be a better idea. With a transdiagnostic approach we mean an intervention in which a guiding therapeutic strategy is universally applied across the range of presenting conditions (see also [11]). This approach has some advantages above single-diagnosis protocols. First, the single-diagnosis protocols do not provide guidance on how to address co-occurring diagnoses (e.g. [12]. Some studies have indeed shown that these protocols demonstrate poorer outcomes for the primary disorder for the individuals presenting with more than one diagnosis (e.g. [13]). Another advantage is that therapists need only to receive training in one protocol rather than costly and time-intensive training for multiple interventions [14].

In delineating a potentially effective transdiagnostic approach, it is important to delineate key intervention strategies and focus points of the intervention. Earlier studies have identified specific effective intervention strategies that seem to work in decreasing internalising and externalising problems in youth. Cognitive behavioural interventions [15, 16], parent-child training [17] and peer group interventions [18] are generally found to be effective ways of stimulating social interactions. Social competence calls upon a complex set of skills and competencies. Therefore, in the Handbook of Youth Prevention Science [19], we recommended to focus on several factors to include in preventive interventions for psychosocial problems. In sum, these are relational factors (such as involving peers, diminish reinforcement of negative behaviour, give dominant children insight into their actual popularity, train parents and teachers) and child factors (such as practice social skills, train social information processing, emotion regulation, and realistic self-esteem. In addition to focussing on these risk and protective factors, we recommended to focus on the authentic desire of children (their positive intentions) and on their feeling of responsibility for their behaviour (at a developmentally appropriate level). These last focus points are similar to the concept of autonomy in the Self-Determination Theory [20, 21]. The theory is based on the assumption that people have natural tendencies. They want to grow, to master challenges and to integrate new experiences into a coherent sense of self. This does not occur automatically. When people feel satisfied in their basic psychological needs: autonomy, competence and relatedness, they will develop and function effectively and experience wellness. Whether or not these needs are satisfied is depending on the child’s interaction with the environment. Following this theory, internalising and externalising behaviour can be understood in terms of reactions to basic needs being thwarted [20]. This implies that in trying to decrease internalising or externalising behaviour and to increase socially competent behaviour, an intervention should stimulate not only the above-mentioned developmental child and relational factors, but also children’s feelings of autonomy, competence and relatedness.

Topper training

This study examines the effectiveness of Topper Training (“Kanjertraining” in Dutch; [22]) in a mental health care setting. Topper Training is a well-known intervention in the Netherlands [23]. This training is directed at children with internalising as well as externalising behaviour and takes into account the importance of motivation, autonomy, competence and relatedness. It includes cognitive behavioural techniques and is directed at the child and its environment: classmates, school and parents. More specifically, the training is given in three settings: 1) as a universal intervention in primary and secondary schools by teachers; 2) as a curative classroom intervention in disrupted classes by psychologists; and 3) as an indicative preventive intervention in mental health care centres to children and their parents when there is a concern about the social development of the child because of mild to severe psychosocial problems. In all three settings, the intervention is transdiagnostic: targeting mild to severe internalising and externalising behaviour (problems). The program focuses on the attitudes and behaviour of children and parents and in the school settings of educators and the head of the school. Variants of the Topper Training method to create positive group climates are also widely used in sports associations, out-of-school childcare, churches and entire neighbourhoods. In this article, the intervention is studied in the context of a mental health care centre, as an indicated preventive intervention.

The training takes into account the importance of motivation and autonomy by reminding children of their positive intentions and motivation to behave prosocially and by making children aware of their ability to choose their own (autonomous) behaviour. The main method that is used to foster these skills is the use of four caps.

The white cap stands for authentic behaviour on a base of trust in oneself and in the other. Different coloured caps in combination with the white cap cover many ways in which people feel authentic and act based on trust. The black cap in combination with the white cap represents power, leadership, initiative taking and spirit. In the same way, the yellow with white cap represents modesty and being sensitive to others needs and feelings. The combination of red and white cap represents humour (with respect for all parties) and being able to relativise.

All coloured caps have their pitfall. Problematic behaviour (internalising and externalising behaviour) is seen as non-authentic behaviour because, most of the time, it is not the desire of the child to behave without the white cap [22]. The black cap without the white cap stands for aggressive and dominating behaviour; the yellow cap stands for shy, anxious and depressed behaviour; the red cap stands for annoyingly funny, careless and 'accomplice-like' behaviour. A key point is that while children may behave in accordance with the role that belongs to a certain cap, they are not identified or labelled as such. In other words: the cap refers to behaviour, not to a personal trait. Difficult social situations are acted out in role-plays, for example bullying situations that have arisen in the class. The caps can also be used outside the training sessions: children, teachers and parents can ask children “Which cap are you wearing?” so as to make children more conscious of their behaviour. Subsequently, they can ask the child whether he/she would like to put on the white cap. A more detailed description of the theoretical ground and method of Topper Training can be found in [23]. The concept of the white cap is comparable to the view of Self-Determination Theory (SDT) where people have a natural tendency to express and develop themselves. The basic needs in SDT can be linked to the method and theoretical grounds of Topper Training: autonomy (Topper Training says “be yourself, make your own choices”), feeling of competence (by practicing social skills and increasing the feeling of control over one’s life) and relatedness (exercises in interaction with others and trust in others).

Previous research on Topper training

In a quasi-experimental study, the effectiveness of Topper Training was established in a classroom context [23]. Classes (third to sixth grade; age range 8 to 13 years) designated as problematic by their teacher and/or the head of the school were trained by a psychologist. Parents and heads of schools were actively involved and the teachers were coached. The intervention consisted of an average of 15 training hours. Fourteen trained classes (n = 353) were compared to fourteen control classes (n = 343) from the same primary schools. Multilevel analyses revealed medium to large effects on classroom climate: relationship with the teacher, perceived social acceptance by classmates and disruptive behaviour according to the teacher. Cohen’s effect sizes ranged from .66 to 1.55. At the individual level, trained children showed improvements in self-reported prosocial behaviour, depressed mood and self-esteem when compared to the control children. Effect sizes ranged from .20 to .41.

In another quasi-experimental study in a mental healthcare setting [24], 185 trained children were compared to 39 waiting list control children (all between 8 and 11 years old). The training was directed at children with mild to severe psychosocial problems and their parents. After ten 90-minute sessions, the children showed significant decreases in parent-reported internalising and externalising problems, aggression, withdrawn-depressed behaviour, social problems and their problems in general. Marginally significant effects were found for attention problems, anxious-depressed problems and somatic problems. Effect sizes ranged from .26 to .46.

These studies were done under real-world conditions: participants applied for the training as usual and the training was given as usual. An advantage of this approach is that the results are easily transferrable to daily practice. This is crucial because Topper Training is already widely implemented in the Netherlands.

The present study

The quasi-experimental design of both earlier studies did not allow strong conclusions to be drawn on the causal effect of the intervention. To overcome this limitation, the aim of this study was to examine Topper Training effects with a more stringent test: a randomised trial. Earlier studies were based on child, parent or teacher reports. The present study uses multiple informants in one study: parents, teachers and children. Moreover, we added a follow-up measurement after six months. The main question is: Is Topper Training effective for 8- to 11 year olds with mild to severe problems in social interaction in a mental health care setting, and does this effect remain for half a year?

We conducted the research in a mental healthcare centre in Almere, a medium-sized city in a central region of the Netherlands. The target population in this mental healthcare centre consisted of children with mild to severe problems in social interaction. Our primary hypothesis was: Topper Training can effectively reduce emotional problems and, conduct problems. Moreover, we expected that Topper Training could increase self-worth and prosocial behaviour and could decrease peer problems, depression, bullying and victimisation and could help children to cope more adequately with the challenges or problems they faced. Therefore, we hypothesised that Topper Training would also reduce the impact that problems have on the lives of children. Moreover, we hypothesised that the effects would sustain until 6-months follow-up.

Materials and methods

Design

We used a randomised trial with two conditions (intervention group and waiting list control group), three measurement points (pre, post and six-month follow-up) and three informants (child, teacher, parents). Individual children were randomly assigned to the intervention group or to the waiting list group in a 3:2 ratio using a simple randomisation procedure (a throw of the dice, 6 was ‘throw again’). The 3:2 allocation ratio was chosen for practical reasons: in September 2010 and 2011 three groups could start and in February 2011 and 2012 only two groups could start with the training (which was the delayed intervention of the waiting list group). To recruit sufficient numbers of participants, children were recruited in two time periods, between February 2010 and August 2010 and the same period one year later. The intervention started half yearly in September 2010, February 2011, September 2011 and February 2012 so that the waiting list group received the intervention six months after the intervention group.

All parents signed a consent form to indicate that they agreed to participate in the study. The study was approved by the Ethics review board of the Faculty of social and behavioural sciences of the University of Amsterdam, The Netherlands. The trial was registered under number 2014-CDE-3827 as “Effectiveness of Topper Training”. This trial is listed on the ISRCTN registry as “Effects of Topper Training on psychosocial problems, self-esteem, and peer victimisation” with study ID ISRCTN14967790, see http://www.isrctn.com/ISRCTN14967790. We registered the trial after participant recruitment, because the training was financed as preventive and not part of clinical mental health care at that time. There are no other clinical trials at the moment for this intervention.

Participants

Children were recruited in primary schools and public health institutions in Almere in the Netherlands. Schools and institutions received posters and were informed about the possibility for children to participate in the Topper Training for free. The posters were directed at parents who were concerned about their child because of problems regarding social interaction. Examples of these problems were given, such as victimisation, low self-esteem, socially unskilful behaviour and aggressive behaviour.

Eligible participants were children who were in primary school, were aged between 8 and 11 years, experienced internalising and/or externalising problems in social interactions and were motivated to follow the training programme (as were their parents). These criteria were exactly the same as those used in the daily practice of the training. A total of 140 families were eligible for inclusion in the study (see Fig 1). Of those, 134 families (96.3%) expressed their desire to participate in the study and gave their permission. The 134 children from these families were randomly assigned to the intervention group (n = 79) and waiting list group (n = 55). Two children from the intervention group did not report any problems at the interview stage and therefore chose not to participate in the intervention. At post-intervention (T2), all of the remaining children (77 intervention and 55 waiting list children) were still participating in the study. A sensitivity analysis in Gpower indicates that with power of .95, the sample size allowed for detection of modest effect sizes of d > .31 (effect size f > .157).

The waiting list group received the intervention half a year later than the intervention group, see Fig 2. By that time, one child had decided not to participate in the intervention because the previously reported problems were no longer apparent. Five other (waiting list) children dropped out during the intervention: one child dropped out because the parents were in the process of a divorce, two children dropped out because of family problems and two children dropped out of the intervention for other, unknown reasons. All of these six children were included in the second time point before their intervention and dropped out thereafter. At the third measurement point, we were unable to contact two other children in the intervention group and one in the waiting list group.

Baseline demographic and clinical characteristics of the intervention and waiting list groups are shown in Table 1. Clinical problems that were the most reported were emotional symptoms, self-perceived victimisation and impact of the problems. According to the parents, about 10% of the children were diagnosed as having ADHD or ADD, one child had an anxiety disorder, one child had a disorder in the autistic spectrum and one child had attachment problems. The children with ADHD or ADD were prescribed medication for their condition.

The intervention and control groups did not differ in age (t(130) = 1.540, p = .126) or gender (Chi ²(1) = .779, p = .377). Mean age was 9.38 years (SD = 1.2). The percentage of boys was 50%. Level of education of the parents did not differ between the groups: the distribution of families in low, middle and high educational segments was 7%, 42% and 51% respectively (Chi ²(2) = .338, p = .845). Also, ethnic composition did not differ between the groups (Chi ²(2) = 3.349, p = .187), 78% was Dutch, 5% Western migrant and 17% non-Western migrant.

Attendance

Attendance was high for both groups. The mean attendance for the intervention group over ten group sessions was 9.4 sessions (SD = .7), with 55% of the children attending all ten sessions, 35% attending nine sessions and 10% attending eight or seven sessions. Mean attendance during the intervention period of the waiting list group was 9.5 sessions (SD = .8), with 64% of the children attending all ten sessions, 24% attending nine sessions and 12% attending eight or seven sessions. Five intervention children filled in the post-intervention measurement after nine training sessions instead of ten. This was done because these children would not be able to fill in the questionnaires directly after the last training session. To ensure a post-test measure for these children, we chose to let them fill it in directly after the ninth session.

Procedure

After recruitment, the pre-test measurement took place (T1, around June 2010 and for the second group June 2011), followed by the randomisation procedure. The intervention group then started with the intervention, followed by a post-intervention measurement (T2, December of the same year) directly after the last training session. Half a year later the follow-up measurement (T3 in May) took place. We organised a meeting for each training group to fill in all the questionnaires again. The waiting list group had to wait half a year after the first measurement and then completed the second measurement at the same time point as the intervention group. Thereafter, the waiting list group received the intervention, followed by the post-intervention measurement (T3) directly after the last training session (see Fig 2). All children had an interview that was planned after their pre-test preceding the intervention: after T1 for the intervention group and after T2 for the waiting list group. In the original study protocol, (see S2 File and S3 File), we planned to have two pre-test measurement occasions: in May and August. However, we decided to omit the August measurement. The reason was that for some of the children theses time points were too close to each other (because they registered in June or July). This made the August measurement less functional.

In general, parents filled in questionnaires at home before the intervention and at the mental health care centre after the last session. Teachers received the questionnaires from the parents and sent them back. Children filled in the questionnaires under supervision at the mental health care centre. The control group filled in pretest questionnaires at school under supervision, because the intake was half a year later. Completion of the questionnaires took about 15–20 minutes.

To motivate parents to fill in the questionnaires at three separate points in time, the training was offered for free (upon the precondition that all measurement occasions were completed) and parents received a report with the results for their child. Children, parents and teachers were all knowledgeable as to who was in the intervention condition and who was not: it was not possible to blind participants, parents or teachers.

Intervention

Topper Training was provided by two trained psychologists with 5 and 7 years experience each in giving this training. The intervention consisted of ten 90-minute group sessions given every two weeks. Training groups contained a maximum of 15 children with internalising and/or externalising problems. In S5 File we give a more detailed description of the intervention.

Measures

Strengths and Difficulties Questionnaire (SDQ)

Parents and teachers reported children’s problem behaviour on the SDQ [25, 26] a 25-item measure of problem behaviour and prosocial behaviour. We used the Emotional Symptoms scale (5 items), Conduct Problems scale (5 items), Peer Problems scale (5 items) and Prosocial Behaviour scale (5 items). We did not use the Attention and Hyperactivity scale in this study, because this is not one of the goals of Topper Training. Items were rated on a scale ranging from 0 (not true) to 2 (certainly true). In our sample, Cronbach’s alpha ranged between .69 and .81 for teacher reports and between .50 (mother report on Emotional Symptoms scale) and .71 for parent reports. Concurrent validity in a Dutch sample was established [26].

We used the extended SDQ with an additional impact supplement. This supplement provides an impact score, which is the sum of the scores on the distress and social incapacity items. The Impact score is found to discriminate better between community and clinic samples than symptom scores [27]. Pre-test scores of mother and father were strongly correlated (r between .51 and .79). We decided to take the average parent score by computing the mean score of father and mother. When the score of only one parent was available at a given point in time, we also used the score of that parent at the other time points for that child to ensure correct within-subject comparisons. This was the case for five training children and two control children.

Child Depression Inventory (CDI)

We assessed depressive symptoms through a Dutch translation [28] of the Children’s Depression Inventory (CDI) [29]. In this translation, one item from the original CDI concerning suicidal ideation (“I want to kill myself”) was replaced by two less precarious questions: I (never/sometimes/-often) think “I wish I was dead” and I (always/sometimes not/do not) think that life is worth living. This resulted in a 28-item questionnaire. For each item, children selected one of three statements indicating how they had felt over the past 2 weeks. The CDI has strong predictive, convergent and construct validity (e.g., [30, 31]), and was shown to have adequate internal consistency and test-retest reliability in previous studies [29, 32]. Cronbach’s alpha in the current sample was .85. On the basis of cut-off scores suggested by Kovacs [29], scores below 13 were rated as normal and scores of 16 or higher were rated as clinically depressed.

Self-Perception Profile for Children (SPPC)

We used the self-esteem scale from the Dutch version [33] of the Self-Perception Profile for Children [34]. This scale consists of 6 items. Each item consists of two opposing descriptions, from which children have to choose one and then indicate whether this is somewhat true or totally true for them. Accordingly, each item is scored on a four-point scale, with a higher score reflecting a more positive view of oneself. The Dutch version was found to be reliable (Cronbach’s alpha = .74 and test-retest reliability after four weeks was .74 [33]. The construct and concurrent validity was established in a Dutch sample [35]. Internal consistency in the current sample was .88. Scores below the 10^th percentile were rated as clinically low and above the 20^th percentile as normal. This translates into different scores for boys and girls: girls scored clinical below 16, boys below 17. Scores of 18 or higher were rated as normal for boys and girls.

Topper questionnaire

We used the Topper questionnaire [36] to measure bullying and self-perceived peer victimisation. Bullying was measured by the question: ‘I bully at school’ and self-perceived peer victimisation was measured by two questions: ‘I am afraid of being bullied’ and ‘I get bullied’, comparable to The Revised Olweus Bully/Victim Questionnaire [37]. For each statement children chose “totally not true,” “not really true,” “a little true” or “totally true” using a four-point Likert scale. Correlations with the other dependent variables of this study were in the expected direction and gave support for the concurrent validity of these questions, see S1 Table.

This questionnaire was filled in at home since supervision was not necessary. All other child questionnaires were completed under the supervision of a test assistant. Clinical relevance was measured by categorising children as ‘bully’ or ‘non-bully’ and ‘victim’ or ‘non—victim’. Children with a score below 3 (“totally not true” and “not really true”) were rated as non-victim or non-bully; children with a score of 3 or higher (“a little true” or “totally true”) were rated as victim or bully. This classification is comparable to the criterion (i.e. more than once or twice) used by Farrington and Ttofi [38] in their meta-analysis. The other scales of the Topper questionnaire were filled in, but we decided not to use them in this study because we already measured these aspects with other measures (CDI and SDQ).

The dependent variables correlated with each other in the expected directions and strength. Pearson correlations varied between r = -.72 (p < .001) and r = .46 (p < .001). For a complete overview of all correlations, see S1 Table.

Data analyses

To test the immediate effects of Topper Training, we used Repeated Measures ANOVA with group (intervention, waiting list) as between group factor and time (T1, T2) as within group factor. A significant group x time interaction effect indicated an intervention effect. Effect sizes (Cohen’s d) were calculated by subtracting the T1-T2 change in the waitlist group from the T1-T2 change in the intervention group. And diving this by the pooled standard deviation of the T1-T2 difference scores (see [39]). Based on these difference scores we computed the confidence interval for Cohen’s d. To determine the clinical relevance of these results, we computed the proportion of children in each group that moved from the clinical to the normal range, based on the normative data of the instruments.

We used the three time points for the waiting list group to test for additional evidence of an intervention effect. The slope between T2 and T3 (the intervention period) was compared with the slope between T1 and T2 (waiting list period). The significance of the difference was tested with a quadratic interaction effect in a repeated measures analysis, while only including the waiting list group. A significant interaction in combination with inspection of the graphs for the direction of the interaction was used as an additional test for the intervention effect. To examine the extent to which immediate post-test change was maintained at the six-month follow-up in the intervention group, Paired-Samples t-tests were used on immediate post-test and follow-up scores.

We checked assumptions before conducting the repeated measures ANOVA. Most variables had a normal distribution. Some did not have a normal distribution, as was expected (e.g. depression). With a sample size of more than 30, this does not give a bias in the analyses (central limit theorem). Sphericity is only applicable when comparing three time points; in this study we only compare two time points in the analyses. Scores are independent since the control and training groups did not have any contact and could not influence each other’s answers on the questionnaires.

Results

Baseline differences between intervention and waiting list groups

At baseline, the groups only differed on self-perceived peer victimisation (t (121) = 1.984, p = .05). The intervention group scored higher at baseline (M = 2.3, SD = 1.0) than the waiting list group (M = 2.0, SD = 0.9). We corrected for these pre-test differences by entering the pre-intervention score as a covariate in an ANCOVA on the intervention effect. Mean scores did not differ between the intervention and control group for any other variable, including bullying, depression, self-esteem or the parent and teacher SDQ scales (all p’s > .05).

Immediate effects

Table 2 provides descriptive statistics for the intervention and waiting list groups at pre-intervention (T1), post-intervention (T2) and half a year later (T3). We plotted these mean scores in Figs 3–9 and in S1 and S2 Figs, calling the intervention group ‘Immediate Topper’ and the waiting list group ‘Waiting list Topper’. Table 3 provides the results from repeated measures analyses. The table shows Cohen’s d and its confidence intervals and the interactions between intervention group and time: the intervention effects. This interaction effect, indicating more positive change in the intervention group compared to the waiting list group, was significant for self-perceived peer victimisation F(1,119) = 6.66, p = .011, d = .62, self-worth F(1, 130) = 6.51, p = .012, d = .45, parent-reported (but not teacher-reported) emotional symptoms F(1,127) = 15.12, p = 1,62 * 10⁻⁴, d = .70, peer relationship problems F(1,127) = 5.14, p = .025, d = .41, and the impact of these problems F(1, 127) = 8.59, p = .004, d = .59) and teacher-reported (but not parent-reported) conduct problems F(1,118) = 4.95, p = .028, d = .42. No significant effects were found on self-reported bullying, depression or prosocial behaviour. No adverse effects were found: in all cases where no effects were found, the intervention group improved as much as the waiting list group did.

Additional within-group analyses of effects in the delayed intervention group

Additional analyses were conducted to test for the effects of Topper Training in the waiting list (i.e. delayed intervention) group. In these analyses changes in the outcome variables during the waiting list period were compared with changes in the same group during the intervention period using repeated measures ANOVA’s with the three time points and a quadratic contrast. Significant quadratic effects (indicating more improvement during the intervention period than during the waiting list period) were found for Self-perceived peer victimisation: F(1,42) = 4.23, p = .046, parent-reported Emotional symptoms: F(1,44) = 5.73, p = .021, and Impact of the problems: F(1,44) = 6.85, p = .011. No quadratic (intervention) effects were found for teachers: improvements experienced during the waiting list period were comparable with improvements during intervention, resulting only in significant linear effects.

Maintenance of effects

Due to the delayed intervention design, no half-year follow-up data were available for the control group. We therefore tested whether outcomes were stable or improved from immediate post-intervention to the six-month follow-up for the intervention group separately. Within-participants t-tests indicated that scores did not change between immediate post-test and six-month follow-up for all scales on which immediate effects were found (see also Figs 3–8, S1 Fig and S2 Fig). While no immediate effect was found for depression (the waiting list group improved as much as the intervention group) the scores at follow-up were significantly lower than the scores at post-intervention, which may tentatively suggest that Topper Training reduces depression in the long run.

Clinical relevance

Besides comparing average scores of the whole sample, it is interesting to test the effect of Topper Training specifically for children who scored in the clinical range. Clinical relevance of the results (i.e. the extent to which children scoring in the clinical range at pre-test showed movement to the normal range at post-test) is shown in Table 4. For most of the problem domains, the proportion of children scoring in the clinical range at baseline that moved to the normal distribution at post-test in the intervention group was substantial (30% to 70% across different measurements). Parent- and child-reported proportions of improvement were significantly higher in the intervention group than in the waiting list group for emotional and conduct problems, impact and self-esteem. Teacher-reported proportions were more similar in the intervention and waiting list groups, resulting in no statistical differences.

Discussion

In this study we evaluated the effects of the indicated preventive transdiagnostical intervention Topper Training in 8- to 11 year olds with mild to severe problems in social interaction in a mental health care setting. We hypothesised main effects on conduct as well as emotional problems. In line with this hypothesis, we found significant effects on parent reported emotional problems and on teacher reported conduct problems. We discuss possible explanations for the difference between parental and teacher report later on. Is it recommendable to give Topper Training to children with clinical emotional or conduct problems? Although only a subsample of the children could be included for this analysis, we found significant effects in our ‘clinical relevance analysis’. Almost half of the parents (47%) of children with clinical emotional problems reported that those problems were reduced to a normal range after the training, compared to 16% in the waiting list group. This is in line with our previous study in a mental health care setting [24], were parents reported a large effect for children with clinical internalising problems (d = .87).

Although parents did not report an effect on conduct problems in the whole sample, we found that 64% of the children with clinical conduct problems scored in the normal range at posttest (13% in waiting list group). This is in line with our previous study [24] that shows significant effects on parental reported clinical externalising problems and aggression. The overall effect on teacher reported conduct problems was in line with effects of Topper Training in a classroom context [23], where the teacher reported a large effect on disruptive behaviour at the classroom level.

Regarding the secondary outcomes, parents perceived a decline in peer problems (in line with our earlier findings [24]) and impact of those problems on the lives of the children, but no decrease in prosocial behaviour (in contrary to our previous study in a classroom context [23]). Additionally, half of the parents reported that the impact of the problems reduced from very high to normal (compared to 13%). Children experienced significant less peer victimisation and an increased self-worth after training (in line with effects in a classroom context [23], but no decrease on bullying (we did not measure this before). The training created a healthy self-worth for 69% of the children with clinical low self-worth (compared to 17%). All benefits of the training sustained up to half a year.

The effects of Topper Training on depression need a closer look. It appears that in the waiting list condition, children become less depressed over time without intervention. During the intervention, Topper Training does not seem to have an additional effect on depression. However, six months after the training, depression had declined even more in the training group. This suggests a sleeper effect. In the current design, we cannot make causal inferences on this, but it might indicate that Topper Training gives children tools to deal in a different way with social situations and that this gives them control over their lives and hence on the long run reduces depressed feelings. This would be in line with the mentioned ideas of Self-Determination Theory: stimulating autonomy, competence and relatedness (which Topper Training does) will contribute to well-being and hence reduce depressed mood.

Taken together, the results provide additional support for the effectiveness of Topper Training in 8- to 11-year-old children with mild to severe psychosocial problems, under real-world conditions. The effects are substantial and are in line with previous research on Topper Training in a mental healthcare setting and in a classroom setting [23, 24]. The discrepancies between parent- and teacher-reported effects are salient in this study. A surprising finding was that parents in the current study did not report significant improvements in their child’s conduct problems, while this was reported by the teachers, and while parent-reported conduct problems of the child were found to decrease in our earlier studies in a mental healthcare setting [24] and in a classroom context [23]. When we take a closer look at the data, it appears that the children in our sample had on average low conduct problems, showing a bottom-effect. Only about 15–18% of the children showed clinical conduct problems at pre-test according to parents. Topper Training was clearly effective for those children: about two-thirds of the children with clinical-level conduct problems at pre-test moved to the normal range at post-test (compared to 13% in the waiting list group). The fact that the improvement in conduct problems was clinically relevant but not statistically significant may be a consequence of the heterogeneity of the sample. To test whether Topper Training is effective for children with behavioural problems, a fully clinical sample would perhaps be more suited.

In contrast to parents, teachers did not seem to experience any effect of Topper Training on emotional symptoms, peer relationship problems and impact. Inspection of the data reveals that teachers experienced improvements in emotional and peer problems in control group children while they were waiting for the intervention. This might indicate that teachers may have been especially attentive to the children who were placed on a waiting list. This extra attention might have had a positive influence on the children, in that they may have felt more noticed and understood by the teacher, which in itself can lead to a reduction in emotional symptoms. Another explanation for the discrepancy between parent- and teacher-reports could be that teachers may be more sensitive to perceiving (changes in) conduct problems in a classroom context than to changes in emotional problems that are not readily observable. Yet another explanation might be that the new child social-emotional skills that have an effect on emotional symptoms and peer interaction are only practiced in the home context and have not yet been generalised into the school setting. This may take more time. Decreases in peer problems perceived by the teacher give support for this notion: six-month follow up scores of the teachers were comparable to post-test scores of the parents.

At first sight, another surprising finding was that Topper Training effectively reduced peer victimisation while the program did not affect levels of self-reported bullying. Perhaps this pattern of findings can be explained by the fact that there was very little self-reported bullying among the children in the current sample at baseline, so improvements could hardly be made. Future studies, with other criteria for inclusion, may test whether Topper Training reduces bullying by children who are selected for bullying behaviour. In addition, contrary to expectations, we did not find any significant effects on prosocial behaviour. Topper Training seems to have more effect on reducing problems than it does on stimulating positive behaviour. An explanation may be that children in this sample scored in the normal range at pretest on prosocial behaviour, on average (which in the SDQ means: being helpful and kind, sharing), which may have resulted in a ceiling effect.

For comparability with other studies, we used a significance level of p < .05, to identify results as either ‘significant’ or not. Our ‘significant’ p-values range from .011 to .046. Taking into account the applied nature of our study, these are fruitful results. However, if we translate those p-values to Bayes Factors (B), representing the amount of evidence supporting alternative hypothesis against the null hypothesis (see [40]), we see that most of our data cannot be considered “strong” evidence (Johnson [41] suggests to test at p < .005 (corresponding to a strong evidence criterion, or Bayes Factor of B > 14). Following that criterion, the effects on parent-reported emotional problems (p = 1,62 * 10⁻⁴ which corresponds to B = 260) and impact of the problems (p = .004: B = 17) give strong evidence for the difference in development between trained children and waitlist children. Thus, with this more stringent criterium we conclude that it is very likely that children receiving Topper Training show reductions in emotional problems and impact of the problems according to parents.

Limitations and strengths

The present study provides a stringent test of the effectiveness of Topper Training, but it is still characterised by some limitations. One limitation of this study is that while we used multi-informant assessments, none of the informants were blind to condition. This might have influenced their responses. Another limitation is that the follow-up data for children who received the intervention directly could not be compared to a control group that did not undergo an intervention. A third limitation is that only a subset of our sample scored in the clinical range at pre-test: this made the sample size for calculating clinical relevance relatively small. While the clinical relevance of the current results is certainly promising, we need to test the effectiveness in a more clinical sample to generalise the results to a more clinical population. A fourth limitation of this study is that although we were generally able to use reliable and valid measures, bullying and peer victimisation were measured by only one and two questions, respectively. However, the Olweus Bully/Victim questionnaire [37] is used in many studies, and it too relies on two main questions (comparable to the ones we used in our study). Finally, to make our results more comparable to those obtained when using Olweus’ complete questionnaire, it would have been better if we had used similar response options to those used in previous studies, such as ‘not at all’, ‘only once or twice’, ‘two or three times a month’, ‘about once a week’, and ‘several times a week’. We did not do this because the questionnaire being used was part of the normal intervention intake procedure, with standard answering categories for all questions.

An important strength of this study is the random assignment of the children to either the intervention or waiting list group, which makes causal inference strong. In addition, the training was given under real-world conditions with routine provision of a training that is already widely implemented in this way. This makes the results significant in practical terms: this intervention in other mental healthcare centres by trained psychologists is likely to be effective. Results of an earlier study in these centres were found to be in line with the present findings [24]. Another strength of the study is the heterogeneity of the sample. This intervention is not only directed at and effective for children with either internalising or externalising problems, but rather is directed at the whole spectrum of psychosocial problems, taking into account one underlying general psychopathology factor: the p factor, as suggested by Caspi et al. [10].

Conclusion and future research

Overall, these findings indicate that cognitive behavioural techniques taught in a peer group with an additional parent training and a focus on autonomy, competence and relatedness can be effective for children aged 8 to 11 years with psychosocial problems. Since Topper Training is widely implemented in the Netherlands and this study was done under real-world conditions, these results are promising in terms of the daily practice of this intervention for children with psychosocial problems. These effects were measured after 10 sessions, taking about five months in total. The intervention does not demand costly diagnostic tests, but can be followed without referral. This makes the intervention feasible.

As an additional step towards examining the effective elements of interventions for children with psychosocial problems, future research might examine the effectiveness of separate elements of the training. Moreover, a larger sample would enable us to examine the effectiveness of Topper Training in subsamples based on gender, age and severity of problems which would yield more information on the question for whom the intervention is more (or less) effective.

Supporting information

S1 Fig [tif]
Significant effect of Topper Training on parent-reported (but not teacher-reported) impact of the problems.

S2 Fig [tif]
No significant effect on self-reported bullying.

S1 File [doc]
Abstract in Dutch.

S2 File [doc]
Trial study protocol in Dutch.

S3 File [doc]
Trial study protocol in English.

S4 File [doc]
CONSORT checklist.

S5 File [doc]
Description of the intervention Topper Training.

S1 Table [doc]
Correlations of all outcome measures.