Not-reached Items: An Issue of Time and of test-taking Disengagement? the Case of PISA 2015 Reading Data

[en] Many low-stakes assessments, such as international large-scale surveys, are administered during time-limited testing sessions and some test-takers are not able to endorse the last items of the test, resulting in not-reached (NR) items. However, because the test has no consequence for the respondents, these NR items can also stem from quitting the test. This article, by means of mixture modeling, investigates heterogeneity in the onset of NR items in reading in PISA 2015. Test-taking behavior, assessed by the response times on the first items of the test, and the risk of NR item onset are modeled simultaneously in a 3-class model that distinguishes rapid, slow and typical respondents. Results suggest that NR items can come from a lack of time or from disengaged behaviors and that the relationship between the number of NR items and ability estimate can be affected by these non-effortful NR responses.

Disciplines :

Education & instruction

Author, co-author :

Pools, Elodie ; Université de Liège - ULiège > Evaluation et qualité de l'enseignement (EQUALE)

Language :

English

Title :

Not-reached Items: An Issue of Time and of test-taking Disengagement? the Case of PISA 2015 Reading Data

Publication date :

2022

Journal title :

Applied Measurement in Education

ISSN :

0895-7347

eISSN :

1532-4818

Publisher :

Routledge

Volume :

Issue :

Pages :

197 - 221

Peer reviewed :

Peer Reviewed verified by ORBi

Additional URL :

https://www.tandfonline.com/doi/pdf/10.1080/08957347.2022.2103136

Available on ORBi :

since 16 January 2023

Statistics

Number of views

44 (7 by ULiège)

Number of downloads

1 (1 by ULiège)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenCitations

OpenAlex citations

Bibliography

Brophy, J., & Ames, C. (2005). NAEP testing for twelfth graders: Motivational issues. Washington, DC: National Assessment Governing Board.
Debeer, D., Janssen, R., & De Boeck, P. (2017). Modeling skipped and not-reached items using IRTrees. Journal of Educational Measurement, 54 (3), 333–363. doi: 10.1111/jedm.12147
DeMars, C. E., Bashkov, B. M., & Socha, A. B. (2013). The role of gender in test-taking motivation under low-stakes conditions. Research & Practice in Assessment, 8, 69–82.
Eklof, H. (2010). Skill and will: Test-taking motivation and assessment quality. Assessment in Education: Principles, Policy and Practice, 17 (4), 345–356.
Ferguson, S. L., Moore, E. W. G., & Hull, D. M. (2020). Finding latent groups in observed data: A primer on latent profile analysis in Mplus for applied researchers. International Journal of Behavioural Development, 44 (5), 458–468. doi: 10.1177/0165025419881721
Finn, B. (2015). Measuring motivation in low-stakes assessments (research report RR-15-19). Princeton, NJ: Educational Testing Service.
Foy, P., & Yin, L. (2016). Chapter 13: Scaling the TIMSS 2015 achievement data. In M. O. Martin, I. V. S. Mullis, & M. Hooper (Eds.), Methods and procedures in TIMSS 2015 (pp. 13-1- 13-62). Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Lynch School of Education, Boston College.
Glas, C. A. W., & Pimentel, J. L. (2008). Modeling nonignorable missing data in speeded tests. Educational and Psychological Measurement, 68 (6), 907–922. doi: 10.1177/0013164408315262
Goldhammer, F. (2015). Measuring ability, speed, or both? Challenges, psychometric solutions, and what can be gained from experimental control. Measurement: Interdisciplinary Research and Perspectives, 13 (3–4), 133–164. doi: 10.1080/15366367.2015.1100020
Goldhammer, F., Martens, T., Christoph, G., & Lüdtke, O. (2016). Test-taking engagement in PIAAC. (OECD education working papers, n°133). Paris, France: OECD publishing.
Goldhammer, F., Martens, T., & Lüdtke, O. (2017). Conditioning factors of test-taking engagement in PIAAC: An exploratory IRT modelling approach considering person and item characteristics. Large-scale Assessments in Education, 5 (1). doi: 10.1186/s40536-017-0051-9
Hallquist, M. N., & Wiley, J. F. (2018). MplusAutomation: An R Package for Facilitating Large-Scale Latent Variable Analyses in Mplus. Structural Equation Modeling: A Multidisciplinary Journal, 25 (4), 621–638. doi: 10.1080/10705511.2017.1402334
Jakwerth, P. M., Stancavage, F. B., & Reed, E. D. (2003). NAEP validity studies: An investigation of why students do not respond to questions. Washington, DC: National Center for Education Statistics.
Köhler, C., Pohl, S., & Carstensen, C. H. (2015). Investigating mechanisms for missing responses in competence tests. Psychological Test and Assessment Modelling, 57 (4), 499–522.
Kong, X. J., Wise, S. L., & Bhola, D. S. (2007). Setting the response time threshold parameter to differentiate solution behaviour from rapid-guessing behaviour. Educational and Psychological Measurement, 67 (4), 606–619. doi: 10.1177/0013164406294779
List, M. K., Köller, O., & Nagy, G. (2019). A semiparametric approach for modeling not-reached items. Educational and Psychological Measurement, 79 (1), 170–199. doi: 10.1177/0013164417749679
Lu, Y., & Sireci, S. (2007). Validity issues in test speededness. Educational Measurement: Issues and Practice, 26 (4), 29–37. doi: 10.1111/j.1745-3992.2007.00106.x
Ludlow, L. H., & O’Leary, M. (1999). Scoring omitted and not-reached items: Practical data analysis implications. Educational and Psychological Measurement, 59 (4), 615–630. doi: 10.1177/0013164499594004
Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47 (2), 149–174. doi: 10.1007/BF02296272
Masyn, K. E. (2013). Latent class analysis and mixture modelling. In T. D. Little (Ed.), The Oxford handbook of quantitative methods (pp. 551–611). New York, NY: Oxford University Press.
Muthén, B., & Masyn, K. (2005). Discrete-time survival mixture analysis. Journal of Educational and Behavioural Statistics, 30 (1), 27–58. doi: 10.3102/10769986030001027
Muthén, L. K., & Muthén, B. O. (2007). Mplus user’s guide (5th ed.). Los Angeles, CA: Muthén & Muthén.
OECD. (2009). PISA data analysis manual: SAS (2nd ed.). Paris, France: Author.
OECD. (2017). PISA 2015 technical report. Paris, France: Author.
OECD. (2019). PISA 2018 results (Volume I): What students know and can do. Paris, France: Author.
Pohl, S., Gräfe, L., & Rose, N. (2014). Dealing with omitted and not-reached items in competence tests: Evaluating approaches accounting for missing responses in item response theory models. Educational and Psychological Measurement, 74 (3), 423–452. doi: 10.1177/0013164413504926
Pools, E., & Monseur, C. (2021). Student test-taking effort in low-stakes assessments: Evidence from the English version of the PISA 2015 science test. Large-scale Assessments in Education, 9. doi: 10.1186/s40536-021-00104-6
Schnipke, D. L., & Scrams, D. J. (1997). Modeling item response time with a two-state mixture model: A new method of measuring speededness. Journal of Educational Measurement, 34 (3), 213–232. doi: 10.1111/j.1745-3984.1997.tb00516.x
Schnipke, D. L., & Scrams, D. J. (1999). Representing response-time information in item banks (Law school admission council computerized testing report 97-09). Newton, PA: Law school admission council.
Silm, G., Pedaste, M., & Täht, K. (2020). The relationship between performance and test-taking effort when measured with self-report or time-based instruments: A meta-analytic review. Educational Research Review, 31. doi: 10.1016/j.edurev.2020.100335
Tijmstra, J., & Bolsinova, M. (2018). On the importance of the speed-ability trade-off when dealing with not reached items. Frontiers in Psychology, 9, 9. doi: 10.3389/fpsyg.2018.00964
Ulitzsch, E., von Davier, M., & Pohl, S. (2020). A multiprocess item response model for not-reached items due to time limits and quitting. Educational and Psychological Measurement, 80 (3), 522–547. doi: 10.1177/0013164419878241
Van der Linden, W. J. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72 (3), 287–308. doi: 10.1007/s11336-006-1478-z
Van der Linden, W. J. (2009). Conceptual issues in response-time modeling. Journal of Educational Measurement, 46 (3), 247–272. doi: 10.1111/j.1745-3984.2009.00080.x
Wigfield, A., & Eccles, J. A. (2000). Expectancy-value theory of achievement motivation. Comtemporary Educational Psychology, 25, 68–81.
Wise, S. L., & Kong, X. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18 (2), 163–183. doi: 10.1207/s15324818ame1802_2
Wise, S. L., Pastor, D. A., & Kong, X. J. (2009). Correlates of rapid-guessing behavior in low-stakes testing: Implications for test development and measurement practice. Applied Measurement in Education, 22 (2), 185–205. doi: 10.1080/08957340902754650
Wise, S. L., & DeMars, C. E. (2010). Examinee noneffort and the validity of program assessment results. Educational Assessment, 15 (1), 27–41. doi: 10.1080/10627191003673216
Wise, S. L., & Gao, L. (2017). A general approach to measuring test-taking effort on computer-based tests. Applied Measurement in Education, 30 (4), 343–354. doi: 10.1080/08957347.2017.1353992
Wise, S. L. (2017). Rapid-guessing behavior: Its identification, interpretation and implications. Educational Measurement: Issues and Practice, 36 (4), 52–61. doi: 10.1111/emip.12165
Wu, M., Adams, R., Wilson, M., & Haldane, S. (2007). ACER ConQuest version 2.0. Generalised Item Response Modelling software. Camberwell, Australia: ACER Press.