Paper-and-pencil versus computerized administration mode: Comparison of data quality and risk behavior prevalence estimates in the European school Survey Project on Alcohol and other Drugs (ESPAD)

English version

Autoři: Emanuela Colasante ^aff001; Elisa Benedetti ^aff001; Loredana Fortunato ^aff001; Marco Scalese ^aff001; Roberta Potente ^aff001; Arianna Cutilli ^aff001; Sabrina Molinaro ^aff001
Působiště autorů: National Research Council of Italy, Institute of Clinical Physiology, Epidemiology and Health Research Lab, Pisa, Italy ^aff001
Vyšlo v časopise: PLoS ONE 14(11)
Kategorie: Research Article
prolekare.web.journal.doi_sk: https://doi.org/10.1371/journal.pone.0225140

Souhrn

Purpose

The aim of this experimental study was to investigate whether paper-and-pencil and computerized surveys administered in the school setting yield equivalent data quality indicators and risk behavior prevalence estimates.

Methods

Data were drawn from the European School Survey Project on Alcohol and Other Drugs (ESPAD®) carried out in Italy to monitor drug, alcohol, tobacco use and other risk-behaviors among Italian high school students aged 15–19 years.

A sub-sample of schools was recruited for the study (1673 pupils). For each school, two entire randomly selected courses (from the first to the fifth grade) participated and were assigned randomly to the self-administered paper-and-pencil (N = 811) or computerized survey (N = 862). Differences in data quality were assessed using the following indicators: questionnaire completeness (missing gender and/or 50% of missing answers) and internal consistency (repetitive extreme response patterns). Separate logistic regression models were used to estimate the mode effect on the reporting of each risk behavior, controlling for gender and age. Finally, the prevalence estimates of the experimental study were compared to the results of the national ESPAD® study.

Results

The computerized administration mode produced a higher proportion of invalid questionnaires, but the prevalence estimates generated from responses to the paper-and-pencil and computerized surveys were generally equivalent. Nevertheless, comparing these results with those of the national ESPAD® study, some differences in the prevalence rates were found.

Conclusions

The findings suggest that in a proctored school setting, the computerized survey mode yields almost the same results as the paper-and-pencil mode. However, because of the reliance on existing informatics facilities until when all schools in the country will be sufficiently equipped for the computerized data collection, they should be given the opportunity to choose between paper-and-pencil and computerized survey modes, in order to avoid a possible selection bias.

Klíčová slova:

Europe – Alcohol consumption – Behavior – Schools – Surveys – Questionnaires – Italian people – Drug administration

Zdroje

1. Ebert JF, Huibers L, Christensen B, Christensen MB. Paper- or Web-Based Questionnaire Invitations as a Method for Data Collection: Cross-Sectional Comparative Study of Differences in Response Rate, Completeness of Data, and Financial Cost. J Med Internet Res 2018;20(1).

2. Denniston MM, Brener ND, Kann L, Eaton DK, McManus T, Kyle TM et al. Comparison of paper-and-pencil versus Web administration of the Youth Risk Behavior Survey (YRBS): Participation, data quality, and perceived privacy and anonymity. Comput Human Behav 2010;26:1054–1060.

3. Gnambs T, Kaspar K. Disclosure of sensitive behaviors across self-administered survey modes: a meta-analysis. Behav Res Methods 2015;47(4):1237–1259. doi: 10.3758/s13428-014-0533-4 25410404

4. Brener ND, Billy JOG, Grady WR. Assessment of Factors Affecting the Validity of Self-Reported Health-Risk Behavior Among Adolescents: Evidence From the Scientific Literature. J Adolesc Health 2003;33:436–457. doi: 10.1016/s1054-139x(03)00052-1 14642706

5. Bowling A. Mode of Questionnaire Administration Can Have Serious Effects on Data Quality. J Public Health 2005;27(3): 281–91.

6. Beebe TJ, Harrison PA, McRae JA Jr, et al. An Evaluation of Computer-Assisted Self-Interviews in a School Setting. Public Opin Q 1998;62(4):623–32.

7. Beebe TJ, Harrison PA, Park E, McRae JA Jr, Evans J. The Effects of Data Collection Mode and Disclosure on Adolescent Reporting of Health Behavior. Soc Sci Comput Rev 2006;24(4):476–488

8. Bates SC, Cox JM. The impact of computer versus paper–pencil survey, and individual versus group administration, on self-reports of sensitive behaviors. Comput Human Behav 2008;24:903–96.

9. Gnambs T, Kaspar K. Socially Desirable Responding in Web-Based Questionnaires: A Meta-Analytic Review of the Candor Hypothesis. Assessment 2017;24(6):746–762. doi: 10.1177/1073191115624547 26739360

10. Tourangeau R, Rips LJ, Raskinki K. The Psychology of Survey Response. Cambridge, UK: Cambridge University Press, 2000.

11. Sudman S. Examining substance abuse data collection methodologies. J Drug Issues 2001;31:695–716.

12. Van de Looij-Jansen PM, de Wilde EJ. Comparison of Web-Based versus paper-and-pencil Self-Administered Questionnaire: Effects on Health Indicators in Dutch Adolescents. Health Serv Res 2008;43(5):1708–1721.

13. Hallfors D, Khatapoush S, Kadushin C, Watson K, Saxe L. A Comparison of Paper vs Computer-Assisted Self Interview for School Alcohol, Tobacco, and Other Drug Surveys. Eval Program Plann 2000;23(2):149–55.

14. Lucia S, Herrmann L, Killias M. How important are interview methods and questionnaire designs in research on self-reported juvenile delinquency? An experimental comparison of Internet vs paper-and-pencil questionnaires and different definitions of the reference period. J Exp Criminol 2007;3:39–64.

15. Raat H, Mangunkusumo RT, Landgraf JM, Kloek G, Brug J. Feasibility, reliability, and validity of adolescent health status measurement by the Child Health Questionnaire Child Form (CHQ-CF): internet administration compared with the standard paper version. Qual Life Res 2007;16:675–685. doi: 10.1007/s11136-006-9157-1 17286197

16. Hays S, McCallum RS. A Comparison of the Pencil-and-Paper and Computer-Administered Minnesota Multiphasic Personality Inventory-Adolescent. Psychol Sch 2005;42(6):605–13.

17. Vosylis R, Malinauskienė O, Žukauskienė R. Comparison Of Internet-Based Versus paper-and-pencil Administered Assessment Of Positive Development Indicators In: Adolescents’ Sample. ISSN 1392-0359. PSICHOLOGIJA. 2012. p 45.

18. Vereecken CA, Maes L. Comparison of a Computer-Administered and Paper-and-Pencil-Administered Questionnaire on Health and Lifestyle Behaviors. J Adolesc Health 2006;38(4):426–32. doi: 10.1016/j.jadohealth.2004.10.010 16549304

19. Eaton DK, Brener ND, Kann L, Denniston MM, McManus T, Kyle TMet al. Comparison of paper-and-pencil Versus Web Administration of the Youth Risk Behavior Survey (YRBS): Risk Behavior Prevalence Estimates. Eval Rev 2010;34(2):137–153. doi: 10.1177/0193841X10362491 20234000

20. McCabe SE, Boyd CJ, Young A, Crawford S, Pope D. Mode Effects for Collecting Alcohol and Tobacco Data among 3rd and 4th Grade Students: A Randomized Pilot Study of Web-Form versus Paper-Form Surveys. Addict Behav 2005;30(4):663–71. doi: 10.1016/j.addbeh.2004.08.012 15833572

21. Turner CF, Ku L, Rogers SM, Lindberg LD, Pleck JH, Sonenstein FL. Adolescent sexual behavior, drug use, and violence: Increased reporting with computer survey technology. Science 1998;280:867–73. doi: 10.1126/science.280.5365.867 9572724

22. Wright DL, Aquilino WS, Supple AJ. A comparison of computer-assisted and paper-and-pencil self-administered questionnaires in a survey on smoking, alcohol, and drug use. Public Opin Q 1998;62331–53.

23. Wang YC, Lee CM, Lew-Ting CY, Hsiao CK, Chen DR, Chen WJ. Survey of substance use among high school students in Taipei: Web-based questionnaire versus paper-and-pencil questionnaire. J Adolesc Health 2005;37(4):289–95. doi: 10.1016/j.jadohealth.2005.03.017 16182139

24. Brener ND, Eaton DK, Kann L, Grunbaum JA, Gross LA, Kyle TM, et al. The Association Of Survey Setting And Mode With Self-Reported Health Risk Behaviors Among High School Students. Public Opin Q 2006;70(3):354–374.

25. Kraus L, Nociar A. ESPAD report 2015: results from the European school survey project on alcohol and other drugs. Lisbon: Luxembourg: 2016.

26. Guttormsson U, Leifman H, Kraus L, Arpa S, Molinaro S, Monshouwer K et al. Espad 2015 Methodology. Luxembourg: Publications Office of the European Union. 2016. Available online at: http://www.webcitation.org/73s0W73bi

27. Molinaro S, Siciliano V, Curzio O, Denoth F, Mariani F. Concordance and consistency of answers to the self-delivered ESPAD questionnaire on use of psychoactive substances. Int J Methods in Psychiatr Res 2012;21(2):158–168.

28. Mangunkusumo RT, Duisterhout JS, De Graaff N, Maarsingh EJ, De Koning HJ, Raat H. Internet versus Paper Mode of Health and Health Behavior Questionnaires in Elementary Schools: Asthma and Fruit as Examples. J Sch Health 2006;76(2):80–86. doi: 10.1111/j.1746-1561.2006.00072.x 16466471

29. Weigold A, Weigold IK, Natera SN. Mean scores for self-report surveys completed using paper-and-pencil and computers: A meta-analytic test of equivalence. Comput Human Behav 2018;86:153–164.

30. Mauz E, Hoffmann R, Houben R, Krause L, Kamtsiuris P, Gößwald A. Mode Equivalence of Health Indicators Between Data Collection Modes and Mixed-Mode Survey Designs in Population-Based Health Interview Surveys for Children and Adolescents: Methodological Study. J Med Internet Res 2018;20(3).

31. Baier D. Computer-assisted versus paper-and-pencil self-report delinquency surveys: Results of an experimental study. Eur J Criminol 2018;15(4):385–402.

32. Mangunkusumo RT, Moorman PW, Van Den–de Ruiterde AE, Van Der Lei J, De Koning HJ, Raat H. Internet-Administered Adolescent Health Questionnaires Compared with a Paper Version in a Randomized Study. J Adolesc Health 2005;36(1):70.

33. Paperny DM, Aono JY, Lehman RM, Hammar SL, Risser J. Computer-assisted detection and intervention in adolescent high-risk health behaviors. J Pediatr 1990;116:456–62. doi: 10.1016/s0022-3476(05)82844-6 2308041