Identification of differential item functioning in multiple-group settings: A multivariate outlier detection approach

[en] This paper focuses on the identification of differential item functioning (DIF) when more than two groups of examinees are considered. We propose to consider items as elements of a multivariate space, where DIF items are outlying elements. Following this approach, the situation of multiple groups is a quite natural case. A robust statistics technique is proposed to identify DIF items as outliers in the multivariate space. For low dimensionalities, up to two three groups, also a simple graphical tool is derived. We illustrate our approach with a re-analysis of data from Kim, Cohen, and Park (1995) on using calculators for a mathematics test.

Disciplines :

Mathematics

Author, co-author :

Magis, David ; Université de Liège - ULiège > Département de mathématique > Statistique mathématique

De Boeck, Paul

Language :

English

Title :

Identification of differential item functioning in multiple-group settings: A multivariate outlier detection approach

Publication date :

2011

Journal title :

Multivariate Behavioral Research

ISSN :

0027-3171

eISSN :

1532-7906

Publisher :

Lawrence Erlbaum Associates, Inc.

Volume :

Pages :

733-755

Peer reviewed :

Peer Reviewed verified by ORBi

Available on ORBi :

since 18 May 2011

Statistics

Number of views

111 (5 by ULiège)

Number of downloads

2 (0 by ULiège)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenCitations

OpenAlex citations

Bibliography

Ackerman, T. A. (1992). A didactic explanation of item bias, item impact, and item validity from a multidimensional perspective. Journal of Educational Measurement, 29, 67-91.
Angoff, W. H., & Ford, S. F. (1973). Item-race interaction on a test of scholastic aptitude. Journal of Educational Measurement, 10, 95-106.
Clauser, B. E., & Mazor, K. M. (1998). Using statistical procedures to identify differential item functioning test items. Educational Measurement: Issues and Practice, 17, 31-44.
Conover, W. J. (1971). Practical nonparametric statistics. New York, NY: Wiley.
Dorans, N. J., & Kulick, E. (1986). Demonstrating the utility of the standardization approach to assessing unexpected differential item performance on the Scholastic Aptitude Test. Journal of Educational Measurement, 23, 355-368.
Gnanadesikan, R., & Kettenring, J. (1972). Robust estimates, residuals, and outlier detection with multiresponse data. Biometrics, 28, 81-124.
Hardin, J., & Rocke, D. M. (2005). The distribution of robust distances. Journal of Computational and Graphical Statistics, 14, 928-946.
Holland, P. W., & Thayer, D. T. (1988). Differential item performance and the Mantel-Haenszel procedure. In H. Wainer & H. I. Braun (Eds.), Test validity (pp. 129-145). Hillsdale, NJ: Erlbaum.
Holland, P. W., & Wainer, H. (1993). Differential item functioning. Hillsdale, NJ: Erlbaum.
Huber, P. J., & Ronchetti, E. M. (2009). Robust statistics (second edition). Hoboken, NJ: Wiley.
Johnson, R. A., & Wichern, D. W. (1998). Applied multivariate statistical analysis (4th ed.). Upper Saddle River, NJ: Prentice-Hall.
Kim, S.-H., Cohen, A. S., & Park, T.-H. (1995). Detection of differential item functioning in multiple groups. Journal of Educational Measurement, 32, 261-276.
Lord, F. M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Erlbaum.
Magis, D., & De Boeck, P. (2011). A robust outlier approach to prevent Type I error inflation in DIF. Research report, Department of Mathematics, University of Liège, Belgium.
Mahalanobis, P. C. (1936). On the generalized distance in statistics. Proceedings of the National Institute of Science of India, 2, 49-55.
Mardia, K., Kent, J., & Bibby, J. (1979). Multivariate Analysis. New York, NY: Academic Press.
Maronna, R. A., Martin, D., & Yohai, V. J. (2006). Robust statistics: Theory and methods. New York, NY: Wiley.
Maronna, R. A., & Yohai, V. J. (1995). The behavior of the Stahel-Donoho robust multivariate estimator. Journal of the American Statistical Association, 90, 330-341.
Maronna, R. A., & Zamar, R. H. (2002). Robust estimates of location and dispersion for highdimensional datasets. Technometrics, 44, 307-317.
Miller, R. G. (1981). Simultaneous statistical inference (2nd ed.). New York, NY: Springer-Verlag.
Millsap, R. E., & Everson, H. T. (1993). Methodology review: Statistical approaches for assessing measurement bias. Applied Psychological Measurement, 17, 297-334.
Osterlind, S. J., & Everson, H. T. (2009). Differential item functioning (2nd ed.). Thousand Oaks, CA: Sage.
Penfield, R. D. (2001). Assessing differential item functioning among multiple groups: a comparison of three Mantel-Haenszel procedures. Applied Measurement in Education, 14, 235-259.
Penfield, R. D., & Camilli, G. (2007). Differential item functioning and item bias. In C. R. Rao & S. Sinharray (Eds.), Handbook of statistics 26: Psychometrics (pp. 125-167). Amsterdam, the Netherlands: Elsevier.
R Development Core Team. (2010). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing.
Raju, N. S. (1988). The area between two item characteristic curves. Psychometrika, 53, 495-502.
Raju, N. S. (1990). Determining the significance of estimated signed and unsigned areas between two item response functions. Applied Psychological Measurement, 14, 197-207.
Rocke, D. M. (1996). Robustness properties of S-estimates of multivariate location and shape in high dimension. Annals of Statistics, 24, 1327-1345.
Rousseeuw, P. J., & Croux, C. (1993). Alternatives to the median absolute deviation. Journal of the American Statistical Association, 88, 1273-1283.
Rousseeuw, P. J., & Leroy, A. M. (1987). Robust regression and outlier detection. New York, NY: Wiley.
Rousseeuw, P. J., & van Driessen, K. (1999). A fast algorithm for the minimum covariance determinant estimator. Technometrics, 41, 212-223.
Shealy, R. T., & Stout, W. (1993). A model based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DIF as well as item bias/DIF. Psychometrika, 58, 159-194.
Swaminathan, H., & Rogers, H. J. (1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27, 361-370.
Thissen, D., Steinberg, L., & Wainer, H. (1988). Use of item response theory in the study of group difference in trace lines. In H. Wainer & H. Braun (Eds.), Test validity (pp. 147-170). Hillsdale, NJ: Erlbaum.
Todorov, V., & Filzmoser, P. (2009). An object-oriented framework for robust multivariate analysis. Journal of Statistical Software, 32, 1-47.
Woodruff, D. L., & Rocke, D. M. (1994). Computable robust estimation of multivariate location and shape on high dimension using compound estimators. Journal of the American Statistical Association, 89, 888-896.
Zijlstra, W. P., van der Ark, L. A., & Sijstma, K. (2007). Outlier detection in test and questionnaire data. Multivariate Behavioral Research, 42, 531-555.
Zijlstra, W. P., van der Ark, L. A., & Sijstma, K. (2011). Robust Mokken scale analysis by means of the forward search algorithm for outlier detection. Multivariate Behavioral Research, 46, 58-89.