Measuring Coverage in MNCH: A Prospective Validation Study in Pakistan and Bangladesh on Measuring Correct Treatment of Childhood Pneumonia

Summary

Background:
Antibiotic treatment for pneumonia as measured by Demographic and Health Surveys (DHS) and Multiple Indicator Cluster Surveys (MICS) is a key indicator for tracking progress in achieving Millennium Development Goal 4. Concerns about the validity of this indicator led us to perform an evaluation in urban and rural settings in Pakistan and Bangladesh.

Methods and Findings:
Caregivers of 950 children under 5 y with pneumonia and 980 with “no pneumonia” were identified in urban and rural settings and allocated for DHS/MICS questions 2 or 4 wk later. Study physicians assigned a diagnosis of pneumonia as reference standard; the predictive ability of DHS/MICS questions and additional measurement tools to identify pneumonia versus non-pneumonia cases was evaluated.

Results at both sites showed suboptimal discriminative power, with no difference between 2- or 4-wk recall. Individual patterns of sensitivity and specificity varied substantially across study sites (sensitivity 66.9% and 45.5%, and specificity 68.8% and 69.5%, for DHS in Pakistan and Bangladesh, respectively). Prescribed antibiotics for pneumonia were correctly recalled by about two-thirds of caregivers using DHS questions, increasing to 72% and 82% in Pakistan and Bangladesh, respectively, using a drug chart and detailed enquiry.

Conclusions:
Monitoring antibiotic treatment of pneumonia is essential for national and global programs. Current (DHS/MICS questions) and proposed new (video and pneumonia score) methods of identifying pneumonia based on maternal recall discriminate poorly between pneumonia and children with cough. Furthermore, these methods have a low yield to identify children who have true pneumonia. Reported antibiotic treatment rates among these children are therefore not a valid proxy indicator of pneumonia treatment rates. These results have important implications for program monitoring and suggest that data in its current format from DHS/MICS surveys should not be used for the purpose of monitoring antibiotic treatment rates in children with pneumonia at the present time.

Please see later in the article for the Editors' Summary

This paper is part of the PLOS Medicine “Measuring Coverage in MNCH” Collection.

Introduction

Globally, of the estimated 6.9 million annual deaths in children younger than 5 y, 1.2 million (18%) die from pneumonia [1]. Prompt treatment with appropriate antibiotics in children with pneumonia is an effective intervention for reducing mortality [2],[3]. The proportion of children with pneumonia in a population who receive antibiotic treatment (antibiotic treatment rate) is a key indicator for tracking progress in achieving Millennium Development Goal 4 targets [4]. The validity of this indicator depends on both the correct identification of pneumonia and the use of an antibiotic to treat the condition. Current measures of antibiotic use in pneumonia rely on household-based surveys such as the Demographic and Health Surveys (DHS) and the Multiple Indicator Cluster Surveys (MICS). It is, therefore, crucial that the DHS and MICS measures are reliable and accurate.

Pneumonia indicators in current surveys are based on interviews with mothers (DHS) or primary caregivers (MICS) using structured questions on cough and short, rapid breathing or difficulty in breathing in the previous 2 wk and whether these were chest-related. The intent is to identify the best possible proxy for pneumonia in order to assess treatment coverage based on mother/caregiver recall. Although the DHS survey labels this condition as “symptoms of acute respiratory infection,” the MICS survey uses the term “suspected pneumonia.” Irrespective of the terminology used, for program purposes these cases are classified as pneumonia. Therefore, the accuracy and reliability of these questions and the algorithm as a valid proxy of pneumonia in children has been questioned [5],[6], and there is a need to assess the validity of this approach.

Our primary study objective was to assess the validity of the caregiver's responses to the standard DHS/MICS questions about whether the child had pneumonia in the recent past, and if so, how it was treated. The reference standard was physician-diagnosed pneumonia (as per World Health Organization [WHO] definitions) [7]. A further objective was to determine if these measures can be improved by adding additional questions to the DHS/MICS surveys or using alternative measurement tools. We also aimed to assess if there was any difference in a caregiver's recall at 2 wk (recall period in current DHS/MICS surveys) and 4 wk.

Methods

Ethics Statement

Ethical approval was obtained from the Hospital Ethics Committee (Pakistan Institute of Medical Sciences), the Ethical Review Committee of the International Centre for Diarrhoeal Disease Research, Bangladesh, and the WHO Ethics Review Committee.

We employed a two-stage written consent procedure: caregivers were informed about the study and permission was obtained first at the time of diagnosis and enrollment, and then again at the start of the follow-up home interviews.

Setting

DHS and MICS surveys are conducted very widely throughout low- and middle-income countries and in very diverse settings, which range from rural settings served by community services to urban and peri-urban settings, where the first level health services are provided by local hospital outpatient departments. We thus conducted the study in three diverse urban and rural settings.

The study was conducted in two countries: Pakistan and Bangladesh. In Pakistan, study participants were recruited in an urban setting, from the outpatient department of Children's Hospital, Pakistan Institute of Medical Sciences, Islamabad, Pakistan. In Bangladesh, the participants were recruited from both urban (Dhaka Shishu Hospital, a tertiary pediatric hospital) and rural (community-based recruitment in four unions [Mirzapur, Gorai, Bhatgram, and Jamurki] of Mirzapur, 63 km north of Dhaka) settings. The latter setting involved initial identification of pneumonia and “no pneumonia” cases during weekly home visits by Village Health Workers, later reassessed by study physicians according to WHO guidelines and enrolled.

Study Design

The study identified and recruited two groups of children with acute respiratory infections: those who were confirmed to have pneumonia and those who did not have pneumonia. Two to four weeks after recruitment caregivers were surveyed to assess the accuracy of their recall of the diagnosis and treatment given. This enabled an assessment to be made of the degree to which DHS and MICS measures of antibiotic treatment in those with reported symptoms of pneumonia were valid measures of the antibiotic treatment of true pneumonia in a study population.

The study had two phases. Phase 1 involved recruiting large numbers of children with symptoms of acute respiratory infection and having trained study physicians establish whether pneumonia (the reference standard for the test) was present or absent. We purposefully selected hospital outpatient departments as two of the three study sites to increase the probability that we would find sufficient numbers of children presenting with symptoms of acute respiratory infection, and to allow careful training and monitoring of the study physicians to ensure that the reference standard was robust. At the third, rural site in Bangladesh, patients were also identified in the community. Phase 2 involved having trained field workers interview the caregiver of each child in their home using DHS and MICS algorithms and alternative tools, either 2 or 4 wk after their recruitment.

Participants and Selection Criteria

The study participants were children 0–59 mo with physician-diagnosed clinical pneumonia or “no pneumonia” and their caregivers. A single WHO technical expert (S. A. Q.) oversaw staff training and study monitoring to ensure comparability of case definitions across sites.

Enrollment and Follow-Up

Enrollment procedure

In the two urban settings (Islamabad and Dhaka) all children aged 0 to 5 y who were assessed and managed by hospital outpatient department physicians were referred to study physicians, who then screened, reassessed, and diagnosed the children using WHO acute respiratory infection guidelines and determined their eligibility for inclusion in the study. Children were enrolled in one of two groups: those with pneumonia and those with “no pneumonia” [7]. Treatment provided by outpatient department physicians was recorded. Children with the following conditions were excluded: recurrent wheezing (airway disorder/asthma), severe pneumonia requiring hospitalization (since DHS and MICS surveys typically gather data on very few of these cases due to their very low period prevalence in community surveys and since hospitalization, being a dramatic event, could induce recall bias), symptoms of chronic cough of more than 4 wk duration, history of pneumonia within past 10 d, history of congenital heart disease, and nonresident of study catchment area. Children with “cough or cold/no pneumonia” were frequency matched to pneumonia cases so that in any one week the age (and sex, in the case of the Bangladesh sites) distribution of cases of pneumonia and “cough or cold” was similar.

For each pneumonia case enrolled, a corresponding age-matched (and sex-matched, in case of Bangladesh sites) “no pneumonia” case was selected using a computer-generated randomization list. Details of the recruitment procedures for each site are given in Box 2.

In the rural setting (Mirzapur), the Village Health Workers identified all children with pneumonia or “no pneumonia” through their weekly household visits and informed the study physicians by mobile phone. The study physicians visited at home all children with “possible pneumonia” and a sample of children with “no pneumonia” who lived nearest to the case of “possible pneumonia.” They reassessed using Integrated Management of Childhood Illness guidelines, and screened and enrolled eligible children.

DHS and MICS questionnaires and alternative measurement tools

DHS and MICS surveys both have an algorithm of questions about the presence or absence of specific signs and symptoms of suspected pneumonia (denoted as “symptoms of acute respiratory infection” in the DHS survey and “suspected pneumonia” in the MICS questionnaire; copies of questionnaires are given in Texts S1, S2, S3). Additional study tools were developed by ARI Research Cell, Children's Hospital, Pakistan Institute of Medical Sciences, Islamabad, and field tested in both Pakistan and Bangladesh. These tools included a pneumonia score questionnaire, which consisted of questions on 20 commonly reported signs and symptoms of pneumonia (Box 3), and a video depicting children in three scenarios: with signs and symptoms of pneumonia, with severe pneumonia, and with cough and cold but without pneumonia. The video had nine different clips of no pneumonia, pneumonia (fast breathing), and severe pneumonia (lower chest indrawing), one set for each of three age groups, i.e., up to 2 mo, 2–11 mo, and 12–59 mo. Country-specific drug charts (flip chart and computer-based) were developed showing medicines (especially antibiotics) commonly used in pneumonia and other febrile illnesses in children 0–59 mo old. All questionnaires, including standard DHS and MICS questionnaires, were translated into local languages (Urdu and Bangla—see Texts S1, S2, S3 for copies of questionnaires).

Home follow-up

Using computer-generated randomization lists, two-thirds of enrolled children were randomly allocated to follow-up at 2 wk, and one-third to follow-up at 4 wk. The mothers/caretakers were (i) interviewed using DHS/MICS questions on cough and chest-related short, rapid breathing and difficult breathing, (ii) asked about specific signs of pneumonia using a pneumonia score questionnaire, (iii) shown video clips to find which clip best represented their child's respiratory illness episode, and (iv) shown a “computer-based drug chart” and “a drug flip chart” to identify the drugs the child was treated with. The order of administering the DHS and MICS questionnaires was alternated with each child, so that each questionnaire was administered first on 50% of occasions.

Data Management and Analysis

After double data entry and data cleaning, descriptive statistics were used to assess the socio-demographic and clinical characteristics of the study children. Sensitivity and specificity were calculated to assess the discriminative power of each measurement tool (DHS, MICS, pneumonia score questionnaire, pneumonia video) and for various combinations of tools.

For the pneumonia score, a composite score was developed by adding one point for each symptom or sign present. Sensitivity and specificity were calculated for each of the composite scores (ranging from lowest to highest). The intent was to identify a cutoff that can be used to reach a decision on the presence or absence of pneumonia. SPSS 11.0 (SPSS Inc.) and STATA 10 (StataCorp) were used in Pakistan and Bangladesh, respectively.

Sample Size Calculation

Based on baseline estimates of sensitivity of 60%–70% and specificity of 70%–90% of mother/caregiver recall of symptoms of acute respiratory infection or suspected pneumonia in predicting true pneumonia, it was estimated that 300 children under 5 y with physician-diagnosed pneumonia and 300 with “no pneumonia” should be enrolled at each site in order to estimate sensitivity and specificity with a precision of ±5%.

The study is reported in line with the STARD statement (checklist in Text S4). The protocol is provided in Text S5.

Results

Enrollment and Follow-Up Status

At the Pakistan site, 752 children were enrolled in the study—361 with pneumonia and 391 with “no pneumonia”—between October 2010 and February 2011 (Figure 1). Follow-up of 329 pneumonia and 343 “no pneumonia” cases was successfully carried out; 456 at 2 wk and 216 at 4 wk.

In Bangladesh, between March and August 2011, 1,178 children—589 each in the pneumonia and “no pneumonia” groups—were enrolled, of which 700 were from Dhaka Shishu Hospital (Figure 2) and 478 from rural Mirzapur (Figure 3). “No pneumonia” cases were enrolled using recruitment procedures similar to those used in Pakistan.

Baseline Characteristics of the Study Children

The socio-demographic characteristics and the clinical features of the study children are presented in Table 1.

Discriminative Power of Survey Tools

Sensitivity, specificity, and 95% CIs for the various instruments that were studied are presented in Tables 2 and 3.

Data are given as percent (95% CI).

DHS/MICS questions

Results at both sites showed poor discriminative power for DHS and MICS questions. Individual patterns of sensitivity and specificity varied substantially by study site and between urban and rural sites (Table 2). Results were similar for DHS and MICS questions, although not identical, because of some differences in questionnaire skip patterns. There was no significant difference in the validity of DHS and MICS questions at 2- versus 4-wk follow-up intervals (Table 2).

Pneumonia score

As expected, specificity increased and sensitivity declined with increasing pneumonia score, i.e., the number of positive symptoms from among the pneumonia score questions. An area-under-the-curve analysis was performed for the pneumonia score (see below) and is presented in Figure 4. Thresholds could be chosen so that the performance was comparable to DHS/MICS questions (e.g., >9 in Bangladesh) or to selectively increase test specificity.

Video tool

In Pakistan (where the video was developed and child images recorded), the video tool had a much higher discriminative power than in Bangladesh. There was no significant difference in recall with the video tool at 2 versus 4 wk in any of the three sites (Table 2).

Recall of Antibiotic Use

In children with pneumonia, antibiotic treatment was recalled correctly by 66.0% and 66.8% of caregivers using DHS or MICS questions in Pakistan and Bangladesh, respectively, with no significant difference between 2- and 4-wk recall. Correct recall increased to 72.0% using a drug flip chart/photo album and to 71.3% using a computer-based drug chart in Pakistan, and to 78.8% and 81.2%, respectively, in Bangladesh.

Association of Caregiver's Recall with Socio-Demographic Characteristics

In both Bangladesh study sites, but not in Pakistan, test specificities were higher for mothers who had higher levels of education. At the Bangladesh rural site the test specificity was 59% for mothers who had secondary or higher education, compared to a specificity of 37% for mothers whose education level was below primary. However, at the Bangladesh urban site these figures were 86% and 73%, respectively.

Discussion

Importance of Monitoring Antibiotic Treatment of Pneumonia

Despite recent falls in pneumonia mortality over the past ten years, it remains the largest single cause of death in children [8]. It is therefore essential that global and national programs track the coverage of effective interventions against pneumonia if Millennium Development Goal 4 is to be achieved. The antibiotic treatment of pneumonia is highly effective and, together with immunization, is one of the two main control strategies [9]. DHS and MICS household surveys are the primary tools used to measure intervention coverage in low- and middle-income countries where health information systems are weak, and are the primary source of information on common childhood illnesses and treatment coverage. We report here on the first study evaluating the validity of the estimates of pneumonia treatment coverage produced by the DHS and MICS survey instruments and discuss the implications of these findings for program monitoring.

Validity of DHS/MICS Data for Estimation of Pneumonia Prevalence

Community-based epidemiological studies have estimated pneumonia incidence to be about 0.3 episodes per child per year in children under 5 y in low- and middle-income countries, which is some 12–18 times lower than the reported incidence of upper respiratory infections in this age group [10]. This low prevalence of pneumonia among all children with cough who are surveyed by DHS and MICS requires that the survey instruments must have very high specificity to identify pneumonia, otherwise the great majority of cases identified in the survey will not represent true cases of pneumonia (as reviewed by Campbell and colleagues in this PLOS Medicine Collection [11]). An example of this effect can be seen in a recent Pakistan DHS survey, which reported in 2006–2007 that 2,508/8,367 (29.9%) children under 5 y had cough and 1,178/8,367 (14.1%) had symptoms consistent with pneumonia [12]. This ratio of children with reported symptoms and signs of “cough and cold only” versus “suspected pneumonia” of 2∶1 is in marked contrast to that reported in community studies, where this ratio is typically very much higher, as noted above [13].

DHS and MICS reports clearly caution that these data (denoted as “suspected pneumonia” in MICS surveys) should not be used as a proxy measure of the prevalence of pneumonia in the community. However, many pneumonia control programs in the developing world, faced with a lack of data on this important parameter for planning purposes, use the information in this way. The results of our study show that the specificity of these survey tools is well below the very high levels required for this proxy to give an accurate estimation of pneumonia prevalence, and reinforce that these data should not be used for this purpose as they will lead to large overestimations of pneumonia prevalence [11].

Validity of DHS/MICS Data for Estimation of Proportion of Children with Pneumonia Who Receive Antibiotic Treatment

DHS and MICS surveys identify children whose caregivers report that they had symptoms and signs consistent with pneumonia, and then ask whether these children were treated with an antibiotic. The validity of this proxy indicator of the proportion of children with pneumonia who receive antibiotic treatment is therefore entirely dependent on the validity of DHS/MICS-reported symptoms and signs of pneumonia as a measure of true pneumonia. For this to represent a valid denominator for this important indicator, it is important that a high proportion of children with “symptoms of acute respiratory infection” (DHS)/“suspected pneumonia” (MICS) actually have “true” pneumonia. If this proportion is low, then results of monitoring antibiotic coverage among these children will not only be inaccurate but also misleading. The results of this study show that DHS and MICS surveys at both sites have poor discriminative power for the identification of episodes of pneumonia in young children. Previous studies have shown that fever is a significant predictor of pneumonia and that adding fever to WHO criteria increased the specificity of pneumonia diagnosis [14],[15]. Our findings support this observation, as specificity increased when a question on the presence of fever was included (Table 3). Since DHS/MICS surveys currently collect information on the presence of fever in the last 2 wk, these data could be readily incorporated into the definition of “suspected pneumonia.” However, the addition of questions on the presence of fever made only modest improvements to the discriminative power. Thus, most children with “suspected pneumonia” (MICS) or “symptoms of acute respiratory infection” (DHS) who form the denominator for the “antibiotic treatment rate” do not truly have pneumonia. We conclude, therefore, that the addition of questions on the presence of these symptoms and signs would not make sufficiently large improvements in instrument validity for these to be adopted at present.

We found no significant difference in caregivers' recall at 2 and 4 wk, which is contrary to the findings in some previous studies. The recall sensitivity of caregivers' reports dropped when the interview took place after more than 2 wk in a respiratory questionnaire validation study conducted in Peru [6]. Some previous studies on maternal recall of breastfeeding duration showed a drop in mothers' recall accuracy with time [16],[17]. A more recent study conducted in Kenya also assessed the accuracy of caregivers' recall over time and found that it decreased when the recall period was increased to 2 wk from 3–4 d [18]. In our study, the lack of fall in performance with the longer recall period suggests that DHS/MICS surveys may be able to adopt a longer recall period for this question, thus increasing the period prevalence of true pneumonia cases detected in the survey. This possibility should be studied further, because since increasing the recall period, and thus the true period prevalence, can be expected to improve the validity of the indicator [11].

The sensitivities and specificities of the DHS/MICS survey instruments varied substantially by study site and by urban and rural setting (Table 2), reflecting different cultural factors, such as caregiver education, and making comparisons across countries or time periods difficult to interpret. These results have important implications for monitoring programs because these instruments are used to track progress over time (typically with serial surveys in different populations within a country) and to identify countries that have low intervention coverage in comparison to others. These findings urge caution in using DHS/MICS data for these purposes.

We show that the use of video material shows some potential to complement current DHS/MICS surveys and result in increased specificity for identification of episodes of pneumonia. However, substantial improvements in test specificity were found only in Pakistan, where the video was made, and were not found in Bangladesh. This finding is not unexpected and may relate to the fact that Pakistani children look somewhat different from Bangladeshi children, which is likely to have influenced caregiver responses. Moreover, we elected a priori to focus the video on showing fast breathing and lower chest indrawing. Ethnographic studies from this region have shown that mothers have varying perceptions and concepts regarding childhood pneumonia, and many caregivers do not associate fast breathing with pneumonia [19]–[21]. The fixed sequence of clips might also have influenced caregiver responses. The potential use of this video tool presents practical and logistical challenges, not least of which is that each individual country may be required to develop their own video. The improved discriminative power with the use of the video in Pakistan is promising, and we recommend that video presentation be explored further in an attempt to further improve its performance.

In children with pneumonia, antibiotic treatment was recalled correctly by two-thirds of caregivers using DHS or MICS questions in Pakistan and Bangladesh, with no significant difference between 2- and 4-wk recall. Correct recall increased with a drug flip chart/photo album or a computer-based drug chart. Thus, the validity of the DHS and MICS questions in correctly identifying antibiotic treatment for pneumonia improved with specific and more structured questions about the medicines or with the use of illustrations of common treatments. Although drug charts performed better than DHS/MICS questions at all three sites, using them in periodic household surveys may be challenging. There may be a very large number of available medicines in any given area, and new medicine brands are introduced often, so frequent revisions would be required.

Study Limitations

The findings of the survey may have been biased by some aspects of the study design. DHS and MICS surveys are conducted in very diverse urban and rural settings. We attempted to reflect this through the urban and rural study sites selected for study. However, it is possible that the rural site in Bangladesh is not typical because of the ongoing research studies in that population. Furthermore, the urban site in Bangladesh included patients from an expanded urban and peri-urban catchment area, resulting in a relatively high rate of loss to follow-up and delayed follow-up. Thus, our findings may not be generalizable to some settings in low- and middle-income countries. Further studies, in particular in settings with high Plasmodium falciparum malaria transmission to explore the generalizability and to check the consistency of these findings, are recommended on this important issue. Nevertheless, the fact that our main study conclusions were confirmed in all three study settings suggests that the main findings may be widely relevant.

We selected study sites with investigators highly experienced in similar studies and employed a single technical expert to conduct the clinical training and oversee the monitoring of research staff in an attempt to reduce the potential for artifactual differences in study implementation across sites. Nevertheless, some of the differences shown between study sites may reflect some variation in application of methods across sites.

Mothers who take their children to a secondary or tertiary care center do not represent a random sample of all mothers who take part in DHS and MICS surveys. This is likely to influence results. We expect that these mothers will be more educated and show better recognition of signs of pneumonia in their children. To the extent that this is true, we may have overestimated the sensitivity and specificity of DHS/MICS questions. Furthermore, it is also likely that attendance at a hospital makes the episodes more memorable (and thus subject to better recall by the caregiver) and thus may once again lead to overestimation of specificity.

It is possible that the “case mix” in the hospital setting is different from that found in community surveys. We attempted to minimize this by excluding cases of severe pneumonia (rarely found in community surveys due to very low 2-wk period prevalence) and by adopting a matched group who had a respiratory illness (cough and cold) rather than studying healthy children. Nevertheless, this may have influenced our estimates of sensitivity and specificity. Thus, study procedures may have influenced caregiver recall. However, our main finding is that specificity levels associated with DHS and MICS questions are much too low to provide a robust basis for a pneumonia treatment rate indicator. We believe that the biases noted above tend to lead to an overestimation of specificity, and so we consider that our main conclusions about poor discriminative ability should remain valid. In addition, the level of specificity that we found is much lower than would be required, suggesting moderate effects of bias.

Conclusions

Monitoring antibiotic treatment of pneumonia is essential for national and global programs. Despite this, we believe this is the first study to assess the validity of current surveys to measure this program indicator. Given the very large investment of human and financial resources in these surveys and their importance to child health programs, it is remarkable that there has been so little published data from research on this key issue. In the context of a low true prevalence of pneumonia, the DHS/MICS questions to define children with “suspected pneumonia” (MICS term) has a low yield for pneumonia, and thus most of these children do not have true pneumonia. This finding reinforces DHS and MICS recommendations that these measures not be used as a proxy for pneumonia prevalence, as using these measures will result in substantial overestimates.

We found the discriminative power (in particular the specificity) of a construct of caregiver-reported symptoms and signs of acute respiratory infection to be low for true pneumonia, as measured by DHS and MICS questionnaires; therefore, the use of such data from these surveys can be misleading for measuring antibiotic treatment rates. Using such data could lead to incorrect policy decisions having programmatic implications. Although the alternative tools evaluated in this study did not perform markedly better than DHS and MICS questions, we have presented some options for improving their specificity. Furthermore, the sensitivities and specificities of DHS and MICS survey instruments varied substantially by site, reflecting different cultural factors, such as caregiver education, and making comparisons across countries or time periods difficult to interpret. These results suggest that data from these surveys should not be used for the purpose of monitoring antibiotic treatment rates in children with pneumonia at the present time.

Supporting Information

Attachment 1

Attachment 2

Attachment 3

Attachment 4

Attachment 5