Examine the effects of two adjustments to the lz statistic

Conformity to a known distribution and sensitivity to response aberrance are desirable properties of person-fit statistics. This simulation study examined the joint and independent effects of two adjustments to the standardized log-likelihood statistic (lz): (1) correction of the negatively skewed distribution of lz (Snijders, 2001), and (2) improving the sensitivity of the statistic by employing more accurate estimates of item response probability using symmetric functions (Dimitrov and Smith, 2006). Data were simulated using three test lengths (10, 20, 30 items). Data containing misfitting response patterns were simulated using three aberrant response patterns (cheating, guessing, and inattentiveness), and three levels of aberrance (i.e., proportion of item responses affected by misfit; 10%, 30% and 50%). Data containing no simulated misfitting response patterns were also generated for each test length. Non-misfitting responses were generated using the dichotomous Rasch measurement model. For each combination of independent variables, a dataset was generated consisting of 5,000 simulees. Four fit statistics were compared: lz, lz* (Snijders adjustment), lzSYM (Dimitrov and Smith adjustment), and lzSYM* (both adjustments). Mean Type I error rates were ≤ 0.1 across all conditions. The lz* statistic produced the best control of Type I error, which was often below the nominal Type I error rate, whereas the empirical Type I error rate for the unadjusted lz statistic most closely approximated the nominal rate. In contrast, lzSYM and lzSYM* yielded empirical Type I error rates larger than the nominal rate, with the discrepancy being particularly pronounced as the length of the test decreased. As might be expected, power to detect misfitting response patterns increased with test length and with the percentage of misfitting response patterns in the sample. Both lzSYM and lzSYM* evidenced improved power in detecting misfitting response patterns compared to lz and lz*, particularly for guessing response patterns and/or on shorter (i.e., 10 item) tests.