Measuring individual differences in cognitive abilities in the lab and on the web

English version

Autoři: Simón Ruiz ^aff001; Xiaobin Chen ^aff001; Patrick Rebuschat ^aff001; Detmar Meurers ^aff001
Působiště autorů: LEAD Graduate School and Research Network, University of Tübingen, Tübingen, Germany ^aff001; Department of Linguistics and English Language, Lancaster University, Lancaster, United Kingdom ^aff002; Department of Linguistics, University of Tübingen, Tübingen, Germany ^aff003
Vyšlo v časopise: PLoS ONE 14(12)
Kategorie: Research Article
prolekare.web.journal.doi_sk: https://doi.org/10.1371/journal.pone.0226217

Souhrn

The present study compared lab-based and web-based versions of cognitive individual difference measures widely used in second language research (working memory and declarative memory). Our objective was to validate web-based versions of these tests for future research and to make these measures available for the wider second language research community, thus contributing to the study of individual differences in language learning. The establishment of measurement equivalence of the two administration modes is important because web-based testing allows researchers to address methodological challenges such as restricted population sampling, low statistical power, and small sample sizes. Our results indicate that the lab-based and web-based versions of the tests were equivalent, i.e., scores of the two test modes correlated. The strength of the relationships, however, varied as a function of the kind of measure, with equivalence appearing to be stronger in both the working memory and the verbal declarative memory tests, and less so in the nonverbal declarative memory test. Overall, the study provides evidence that web-based testing of cognitive abilities can produce similar performance scores as in the lab.

Klíčová slova:

Learning – Cognition – Memory – Vision – Working memory – Language – Memory recall – Language acquisition

Zdroje

1. Kidd E, Donnelly S, Christiansen MH. Individual differences in language acquisition and processing. Trends in Cognitive Sciences. 2018;22(2):154–69. doi: 10.1016/j.tics.2017.11.006 29277256

2. Hamrick P. Declarative and procedural memory abilities as individual differences in incidental language learning. Learning and Individual Differences. 2015;44:9–15. doi: 10.1016/j.lindif.2015.10.003

3. Ruiz S, Tagarelli KM, Rebuschat P. Simultaneous acquisition of words and syntax: Effects of exposure condition and declarative memory. Frontiers in Psychology. 2018 12;9:1168. doi: 10.3389/fpsyg.2018.01168 30050480

4. Li S. Cognitive differences and ISLA. In: Loewen S, Sato M, editors. The Routledge handbook of instructed second language acquisition. New York: Routledge; 2017. pp. 396–417.

5. Pawlak M. Overview of learner individual differences and their mediating effects on the process and outcome of interaction. In Gurzynski-Weiss L., editor. Expanding individual difference research in the interaction approach: Investigating learners, instructors, and other interlocutors. Amsterdam: John Benjamins; 2017. pp. 19–40.

6. Larsen‐Freeman D. Looking ahead: Future directions in, and future research into, second language acquisition. Foreign language annals. 2018;51(1):55–72. doi: 10.1111/flan.12314

7. Hamrick P, Lum JA, Ullman MT. Child first language and adult second language are both tied to general-purpose learning systems. Proceedings of the National Academy of Sciences. 2018;115(7):1487–92.

8. Lado B. Aptitude and pedagogical conditions in the early development of a nonprimary language. Applied Psycholinguistics. 2017;38(3):679–701.

9. Faretta-Stutenberg M, Morgan-Short K. The interplay of individual differences and context of learning in behavioral and neurocognitive second language development. Second Language Research. 2018;34 (1): 67–101. doi: 10.1177/0267658316684903

10. Tagarelli KM, Ruiz S, Moreno Vega JL, Rebuschat P. Variability in second language learning: The roles of individual differences, learning conditions, and linguistic complexity. Studies in Second Language Acquisition. 2016;38(2):293–316. doi: 10.1017/S0272263116000036

11. Buffington J, Morgan-Short K. Declarative and procedural memory as individual differences in second language aptitude. In: Wen Z, Skehan P, Biedroń A, Li S, Sparks R, editors. Language aptitude: Multiple perspectives and emerging trends. New York: Routledge; 2019. pp. 215–237.

12. Marsden E, Morgan‐Short K, Thompson S, Abugaber D. Replication in second language research: Narrative and systematic reviews and recommendations for the field. Language Learning. 2018;68(2): 321–91. doi: 10.1111/lang.12286

13. Plonsky L. Study quality in SLA: An assessment of designs, analyses, and reporting practices in quantitative L2 research. Studies in Second Language Acquisition. 2013;35(4): 655–87. doi: 10.1017/S0272263113000399

14. Plonsky L. Quantitative research methods. In: Loewen S, Sato M, editors. The Routledge handbook of instructed second language acquisition. New York: Routledge; 2017. pp. 505–521.

15. Lindstromberg S. Inferential statistics in Language Teaching Research: A review and ways forward. Language Teaching Research. 2016;20(6): 741–68. doi: 10.1177/1362168816649979

16. Tackett JL, Brandes CM, King KM, Markon KE. Psychology's replication crisis and clinical psychological science. Annual review of clinical psychology. 2019;15:579–604. doi: 10.1146/annurev-clinpsy-050718-095710 30673512

17. Krantz JH, Reips UD. The state of web-based research: A survey and call for inclusion in curricula. Behavior Research Methods. 2017;49(5): 1621–1619. doi: 10.3758/s13428-017-0882-x 28409484

18. Roever C. Web-based language testing. Language Learning & Technology. 2001;5(2):84–94.

19. Domínguez C, López-Cuadrado J, Armendariz A, Jaime A, Heras J, Pérez TA. Exploring the differences between low-stakes proctored and unproctored language testing using an Internet-based application. Computer Assisted Language Learning. 2019:1–27.

20. Diaz Maggioli GH. Web‐Based Testing. The TESOL Encyclopedia of English Language Teaching. 2018:1–6.

21. Birnbaum MH. Human research and data collection via the Internet. Annu. Rev. Psychol. 2004;55:803–32. doi: 10.1146/annurev.psych.55.090902.141601 14744235

22. Hicks KL, Foster JL, Engle RW. Measuring working memory capacity on the web with the online working memory lab (the OWL). Journal of Applied Research in Memory and Cognition. 2016;5(4): 478–89. doi: 10.1016/j.jarmac.2016.07.010

23. Wolfe CR. Twenty years of Internet-based research at SCiP: A discussion of surviving concepts and new methodologies. Behavior research methods. 2017;49(5): 1615–1620. doi: 10.3758/s13428-017-0858-x 28176258

24. Gwaltney CJ, Shields AL, Shiffman S. Equivalence of electronic and paper-and-pencil administration of patient-reported outcome measures: a meta-analytic review. Value in Health. 2008;11(2): 322–333. doi: 10.1111/j.1524-4733.2007.00231.x 18380645

25. Button KS, Ioannidis JP, Mokrysz C, Nosek BA, Flint J, Robinson ES, Munafò MR. Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience. 2013;14(5):365. doi: 10.1038/nrn3475 23571845

26. Branch MN. The “Reproducibility Crisis:” Might the Methods Used Frequently in Behavior-Analysis Research Help?. Perspectives on Behavior Science. 2019;42(1):77–89.

27. Laraway S, Snycerski S, Pradhan S, Huitema BE. An overview of scientific reproducibility: Consideration of relevant issues for behavior science/analysis. Perspectives on Behavior Science. 2019;42(1):33–57.

28. Shrout PE, Rodgers JL. Psychology, science, and knowledge construction: Broadening perspectives from the replication crisis. Annual review of psychology. 2018;69:487–510. doi: 10.1146/annurev-psych-122216-011845 29300688

29. Cohen J. Statistical power analysis for the behavioral sciences. 2nd ed. Hillsdale, NJ: Erlbaum; 2013.

30. Stewart N, Chandler J, Paolacci G. Crowdsourcing samples in cognitive science. Trends in cognitive sciences. 2017;21(10):736–48 doi: 10.1016/j.tics.2017.06.007 28803699

31. Cowan N. Working memory maturation: Can we get at the essence of cognitive growth?. Perspectives on Psychological Science. 2016;11(2): 239–64. doi: 10.1177/1745691615621279 26993277

32. Baddeley AD. Modularity, working memory and language acquisition. Second Language Research. 2017;33(3): 299–311. doi: 10.1177/0267658317709852

33. Roehr K. Linguistic and metalinguistic categories in second language learning. Cognitive Linguistics. 2008;19(1): 67–106. doi: 10.1515/COG.2008.005

34. Grundy JG, Timmer K. Bilingualism and working memory capacity: A comprehensive meta-analysis. Second Language Research. 2017;33(3): 325–40. doi: 10.1177/0267658316678286

35. Jeon EH, Yamashita J. L2 reading comprehension and its correlates: A meta‐analysis. Language Learning. 2014;64(1):160–212. doi: 10.1111/lang.12034

36. Linck JA, Osthus P, Koeth JT, Bunting MF. Working memory and second language comprehension and production: A meta-analysis. Psychonomic Bulletin & Review. 2014;21(4): 861–83. doi: 10.3758/s13423-013-0565-2 24366687

37. Bailey H, Dunlosky J, Kane MJ. Contribution of strategy use to performance on complex and simple span tasks. Memory & cognition. 2011;39(3): 447–61. doi: 10.3758/s13421-010-0034-3 21264605

38. Turner ML, Engle RW. Is working memory capacity task dependent?. Journal of memory and language. 1989;28(2): 127–54. doi: 10.1016/0749-596X(89)90040-5

39. Conway ARA, Kane MJ, Bunting MF, Hambrick DZ, Wilhelm O, et al. (2005) Working memory span tasks: A methodological review and user's guide. Psychonomic Bulletin and Review 12(12): 769–786. doi: 10.3758/BF03196772 16523997

40. Zhou H, Rossi S, Chen B. Effects of working memory capacity and tasks in processing L2 complex sentence: evidence from Chinese-English bilinguals. Frontiers in psychology. 2017;8: 595. doi: 10.3389/fpsyg.2017.00595 28473786

41. Reber PJ, Knowlton BJ, Squire LR. Dissociable properties of memory systems: differences in the flexibility of declarative and nondeclarative knowledge. Behavioral Neuroscience. 1996;110(5): 861. doi: 10.1037//0735-7044.110.5.861 8918990

42. Squire LR. Memory systems of the brain: a brief history and current perspective. Neurobiology of learning and memory. 2004;82(3): 171–7. doi: 10.1016/j.nlm.2004.06.005 15464402

43. Eichenbaum H. Hippocampus: cognitive processes and neural representations that underlie declarative memory. Neuron. 2004;44(1):109–20. doi: 10.1016/j.neuron.2004.08.028 15450164

44. Squire LR. Memory systems of the brain: a brief history and current perspective. Neurobiology of learning and memory. 2004;82(3): 171–7. doi: 10.1016/j.nlm.2004.06.005 15464402

45. Knowlton BJ, Siegel AL, Moody TD. Procedural learning in humans. In Byrne JH, editor. Learning and memory: A comprehensive reference. 2nd ed. Oxford: Academic Press; 2017. pp. 295–312.

46. Hamrick P, Lum JA, Ullman MT. Child first language and adult second language are both tied to general-purpose learning systems. Proceedings of the National Academy of Sciences. 2018;115(7): 1487–1492. doi: 10.1073/pnas.1713975115 29378936

47. Ullman MT. The declarative/procedural model: A neurobiologically motivated theory of first and second language. In: VanPatten B, Williams J, editors. Theories in second language acquisition: An introduction. 2nd ed. New York: Routledge; 2015. pp. 135–158.

48. Ullman MT. The declarative/procedural model: A neurobiological model of language learning, knowledge, and use. In: Hickok G, Small SA, editors. Neurobiology of language. Amsterdam: Elsevier; 2016. pp. 498–505.

49. Morgan-Short K, Faretta-Stutenberg M, Brill-Schuetz KA, Carpenter H, Wong PC. Declarative and procedural memory as individual differences in second language acquisition. Bilingualism: Language and Cognition. 2014;17(1):56–72. doi: 10.1017/S1366728912000715

50. Carpenter, HS. A behavioral and electrophysiological investigation of different aptitudes for L2 grammar in learners equated for proficiency level. Ph.D. Thesis, Georgetown University. 2008. Available from: http://hdl.handle.net/10822/558127.

51. Carroll JB, Sapon SM. Modern Language Aptitude Test: Manual. New York: Psychological Corporation; 1959.

52. Trahan DE, Larrabee GJ. Continuous visual memory test. Odessa, FL: Assessment Resources. 1988.

53. Schneider W, Eschman A, Zuccolotto A. EPrime user’s guide. Pittsburgh, PA: Psychology Software Tools Inc. 2002.

54. Unsworth N, Heitz RP, Schrock JC, Engle RW. An automated version of the operation span task. Behavior Research Methods. 2005;37(3): 498–505. doi: 10.3758/bf03192720 16405146

55. Wickens TD. Elementary signal detection theory. New York: Oxford University Press; 2002.

56. R Development Core Team (2016) R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. http://www.rproject.org.

57. Bates D, Mächler M, Bolker B, Walker S. Fitting linear mixed-effects models using lme4. Journal of Statistical Software. 2015;67(1), 1–48. doi: 10.18637/jss.v067.i01

58. Kane MJ, Hambrick DZ, Tuholski SW, Wilhelm O, Payne TW, Engle RW. The generality of working memory capacity: a latent-variable approach to verbal and visuospatial memory span and reasoning. Journal of Experimental Psychology: General. 2004;133(2): 189–217. doi: 10.1037/0096-3445.133.2.189 15149250

59. Gelman A. The failure of null hypothesis significance testing when studying incremental changes, and what to do about it. Personality and Social Psychology Bulletin. 2018;44(1): 16–23. doi: 10.1177/0146167217729162 28914154

60. Leidheiser W, Branyon J, Baldwin N, Pak R, McLaughlin A. Lessons learned in adapting a lab-based measure of working memory capacity for the web. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting 2015. Los Angeles: Sage CA; 2015. pp. 756–760.

61. Reips UD, Krantz JH. Conducting true experiments on the Web. In: Gosling SD, Johnson JA, editors. Advanced methods for conducting online behavioral research. Washington, DC: American Psychological Association; 2010. pp. 193–216.

62. MacWhinney B. A shared platform for studying second language acquisition. Language Learning. 2017;67(S1): 254–75. doi: 10.1111/lang.12220

63. Meurers D, Dickinson M. Evidence and interpretation in language learning research: Opportunities for collaboration with computational linguistics. Language Learning. 2017;67(S1): 66–95. doi: 10.1111/lang.12233

64. Ziegler N, Meurers D, Rebuschat P, Ruiz S, Moreno‐Vega JL, Chinkina M, Li W, Grey S. Interdisciplinary research at the intersection of CALL, NLP, and SLA: Methodological implications from an input enhancement project. Language Learning. 2017;67(S1): 209–231. doi: 10.1111/lang.12227

Measuring individual differences in cognitive abilities in the lab and on the web

Souhrn

Klíčová slova:

Zdroje

PLOS One

Aktuální možnosti diagnostiky a léčby litiáz