[en] Accurate measurement of metacognitive knowledge in reading is important. Different instruments and scoring methods have been proposed but not systematically compared for their measurement comparability across cultures and validity. Given student data from 34 OECD countries in the Programme for International Student Assessment (PISA) in 2009, we compared two scoring methods for metacognitive knowledge in reading based on pair-wise comparisons of strategies and with conventional Likert-scale responses of selected items. Metacognitive knowledge scored with conventional Likert-scale responses demonstrated higher cross-cultural comparability than the pair-wise comparison method. Linked with reading competence, moti- vation and control strategy in reading, scores from the two scoring methods showed differential criterion validity, possibly related to the types of tasks (understanding and remembering versus summarising), item content (complexity and discrimination between preferred strategies in reading) and common method variance (e.g., individuals’ stable response style in rating scales). Theoretical and methodological implications are discussed..
Disciplines :
Education & instruction
Author, co-author :
Zhou, Ji
He, Jia
Lafontaine, Dominique ; Université de Liège - ULiège > Département des Sciences de l'éducation > Analyse des systèmes et pratiques d'enseignement
Language :
English
Title :
Cross-cultural comparability and validity of metacognitive knowledge in reading in PISA 2009: a comparison of two scoring methods
Publication date :
2020
Journal title :
Assessment in Education: Principles, Policy and Practice
Akturk, A. O., & Sahin, I., (2011). Literature review on metacognition and its measurement. Procedia - Social and Behavioral Sciences, 15, 3731–3736.
Allen, B. A., & Armour-Thomas, E., (1993). Construct validation of metacognition. The Journal of Psychology, 127 (2), 203–211. https://doi.org/10.1080/00223980.1993.9915555
Artelt, C., Beinicke, A., Schlagmüller, M., & Schneider, W., (2009). Diagnose von Strategiewissen beim Textverstehen. Zeitschrift für Entwicklungspsychologie und Pädagogische Psychologie, 41 (2), 96–103. https://doi.org/10.1026/0049-8637.41.2.96
Artelt, C., & Schneider, W., (2015). Cross-country generalizability of the role of metacognitive knowledge in students’ strategy use and reading competence. Teachers College Record, 117 (1), 1–32. http://www.tcrecord.org/Content.asp?ContentId=17695
Authors. (2013).
Authors. (2015).
Authors. (2019).
Baumgart, N., & Halse, C., (1999). Approaches to learning across cultures: The role of assessment. Assessment in Education: Principles, Policy & Practice, 6 (3), 321–339. https://doi.org/10.1080/09695949992775
Boer, D., Hanke, K., & He, J., (2018). On detecting systematic measurement error in cross-cultural research: A review and critical reflection on equivalence and invariance tests. Journal of Cross-Cultural Psychology, 49(5), 713–734. https://doi.org/10.1177/0022022117749042
Carrell, P. L., Gajdusek, L., & Wise, T., (1998). Metacognition and EFL/ESL reading. Instructional Science, 26 (1/2), 97–112. https://doi.org/10.1023/A:1003092114195
Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9 (2), 233–255.
Cieciuch, J., Davidov, E., Schmidt, P., Algesheimer, R., & Schwartz, S. H., (2014). Comparing results of an exact vs. an approximate (Bayesian) measurement invariance test: A cross-country illustration with a scale to measure 19 human values. Frontiers in Psychology, 5, 982. https://doi.org/10.3389/fpsyg.2014.00982
Cubukcu, F., (2008). Enhancing vocabulary development and reading comprehension through metacognitive strategies. Issues in Educational Research, 18 (1), 1–11. http://www.iier.org.au/iier18/cubukcu.html
Flavell, J. H., (1976). Metacognitive aspects of problem solving. In L. B., Resnick (Ed.), The nature of intelligence (pp. 231–235). Lawrence Erlbaum.
Goswami, U., (2008). Cognitive development: The learning brain. Psychology Press.
Green, J. M., & Oxford, R. L., (1995). A closer look at learning strategies: L2 Proficiency and gender. TESOL Quarterly, 29 (2), 261–297. https://doi.org/10.2307/3587625
Hacker, D. J., Dunlosky, J., & Graesser, A., (Eds.). (2009). Handbook of metacognition in education. Taylor & Francis.
Hakel, M. D., (1968). Task difficulty and personality test validity. Psychological Reports, 22 (2), 502. https://doi.org/10.2466/pr0.1968.22.2.502
IDB. (2009). IDB Analyzer (Version 3.1). International Association for the Evaluation of Educational Achievement.
Jacobs, J., & Paris, S., (1987). Children’s metacognition about reading: Issues in definition, measurement, and instruction. Educational Psychologist, 22(3–4), 255–278. https://doi.org/10.1080/00461520.1987.9653052
Kember, D., (1996). The intention to both memorise and understand: Another approach to learning. Higher Education, 31 (3), 254–341. https://doi.org/10.1007/BF00128436
Lau, K., & Chan, D. W., (2003). Reading strategy use and motivation among Chinese good and poor readers in Hong Kong. Journal of Research in Reading, 26 (2), 177–190. https://doi.org/10.1111/1467-9817.00195
Love, S., Kannis-Dymand, L., & Lovell, G. P., (2019). Development and validation of the metacognitive processes during performances questionnaire. Psychology of Sport and Exercise, 41, 91–98. https://doi.org/10.1016/j.psychsport.2018.12.004
Mak, S., Cheung, K., Soh, K., Sit, P., & Leong, M., (2017). An examination of student- and across-level mediation mechanisms accounting for gender differences in reading performance: A multilevel analysis of reading engagement. Educational Psychology: An International Journal of Experimental Educational Psychology, 37 (10), 1206–1211. https://doi.org/10.1080/01443410.2016.1242712
Marambe, K., Vermunt, J., & Boshuizen, H., (2012). A cross-cultural comparison of student learning patterns in higher education. Higher Education, 64 (3), 299–316. https://doi.org/10.1007/s10734-011-9494-z
Marton, F., Wen, Q., & Wong, K. C., (2005). “Read a hundred times and the meaning will appear … “ Changes in Chinese University students’ views of the temporal structure of learning. Higher Education, 49 (3), 291–318. https://doi.org/10.1007/s10734-004-6667-z
Nardi, E., (2008). Cultural biases: A non-Anglophone perspective. Assessment in Education: Principles, Policy & Practice, 15(3), 259–266.
OECD. (2010). PISA 2009 results: Learning to learn: Student engagement, strategies and practices (Volume III).
Paris, S. G., Lipson, M. Y., & Wixson, K. K., (1983). Becoming a strategic reader. Contemporary Educational Psychology, 8 (3), 293–316. https://doi.org/10.1016/0361-476X(83)90018-8
Pintrich, P., Smith, D., García, T., & McKeachie, W., (1991). A manual for the use of the motivated strategies for learning questionnaire (MSLQ). University of Michigan.
Pintrich, P. R., & de Groot, E. V., (1990). Motivational and self-regulated learning components of classroom academic performance. Journal of Educational Psychology, 82 (1), 33–40. https://doi.org/10.1037/0022-0663.82.1.33
Podsakoff, P. M., MacKenzie, S. B., Lee, J.-Y., & Podsakoff, N. P., (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88 (5), 879–903. https://doi.org/10.1037/0021-9010.88.5.879
Pressley, M., Brown, R., El-Dinary, P. B., & Afflerbach, P. (1995). The comprehension instruction that students need: Instruction fostering constructively responsive reading. Learning Disabilities Research and Practice, 10 (4), 215–224.
Roeschl-Heils, A., Schneider, W., & van Kraayenoord, C., (2003). Reading, metacognition and motivation: A follow-up study of German students in Grades 7 and 8. European Journal of Psychology of Education, 18 (1), 75–86. https://doi.org/10.1007/BF03173605
Rosseel, Y., (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48 (2), 1–36. https://doi.org/10.18637/jss.v048.i02
Rutkowski, L., & Svetina, D., (2014). Assessing the hypothesis of measurement invariance in the context of large-scale international surveys. Educational and Psychological Measurement, 74 (1), 31–57. https://doi.org/10.1177/0013164413498257
Säälik, Ü., (2015). Reading performance, learning strategies, gender and school language as related issues–PISA 2009 findings in Finland and Estonia. International Journal of Teaching and Education, 3 (2), 17–30. https://doi.org/10.20472/TE.2015.3.2.002
Schmitt, M. C., (1990). A questionnaire to measure children’s awareness of strategic reading processes. The Reading Teacher, 43(7), 454–461. https://www.jstor.org/stable/20200439
Schneider, W., & Artelt, C., (2010). Metacognition and mathematics education. ZDM - the International Journal on Mathematics Education, 42 (2), 16–149. https://doi.org/10.1007/s11858-010-0240-2
Sitzmann, T., & Ely, K., (2011). A meta-analysis of self-regulated learning in work-related training and educational attainment: What we know and where we need to go. Psychological Bulletin, 137 (3), 421–442. https://doi.org/10.1037/a0022777
Taraban, R., Kerr, M., & Rynearson, K., (2004). Analytic and pragmatic factors in college students’ metacognitive reading strategies. Reading Psychology, 25 (2), 67–82. https://doi.org/10.1080/02702710490435547
Van de Vijver, F. J. R., & Leung, K., (1997). Methods and data analysis for cross-cultural research. Sage Publications.
Van de Vijver, F. J. R., & Leung, K., (2000). Methodological issues in psychological research on culture. Journal of Cross-cultural Psychology, 31 (1), 33–51. https://doi.org/10.1177/0022022100031001004
Van Gelderen, A., Schoonen, R., de Glopper, K., Hulstjn, J., Simis, A., Snellings, P., & Stevenson, M., (2004). Linguistic knowledge, processing speed, and metacognitive knowledge in first-and second-language reading comprehension: A componential analysis. Journal of Educational Psychology, 96 (1), 19–30. https://doi.org/10.1037/0022-0663.96.1.19
Vieluf, S., Kunter, M., & van de Vijver, F. J. R., (2013). Teacher self-efficacy in cross-national perspective. Teaching and Teacher Education, 35, 92–103. https://doi.org/10.1016/j.tate.2013.05.006
Weinstein, C. E., Palmer, D., & Schulte, A. C., (1987). Learning and Study Strategies Inventory (LASSI). H & H Publishing.
Witner, S., & Tepner, O. (2011). Entwicklung geschlossener Testaufgaben zur Erhebung des fachdidaktischen Wissens von Chemielehrkräften [Development of closed-ended questions to measure the knowledge of chemistry teachers]. Chimica et ceterae artes rerum naturae didacticae, 37, 113–137.
Wolters, C. A., (1999). The relationship between high school students’ motivational regulation and their use of learning strategies, effort, and classroom performance. Learning and Individual Differences, 11 (3), 281–301. https://doi.org/10.1016/S1041-6080(99)80004-1
