On the use of ROC curves in DIF simulation studies

Simulation studies are often used to compare methods to detect differential item functioning (DIF). However, comparing the performance of such methods can become complicated when the identification of DIF items relies on statistics based on pre-defined significance level or on pre-established cutoff values. DIF methods based on conceptually different approaches may therefore become incomparable in terms of summary DIF statistics such as false alarm rate or hit rate. The purpose of this talk is to overcome this analytic issue by introducing receiver operating characteristic (ROC) curves in this context. ROC curves allow for global comparison of methods’ performances by computing pairs of (false alarm, hit) rates and representing them on a common scatter plot. Several summary ROC statistics can be considered for further analysis. The application of the ROC curve methodology, together with its limitation and possible extensions, is illustrated by a simple simulation study that compares three score-based DIF methods (Mantel-Haenszel, standardization and Delta plot).