The contribution of Multidimensional IRT for modeling applicants performance on a SJT.

Situational judgment tests (SJTs) are testing methods often used in Personnel Selection to appraise different constructs (Whetzel & McDaniel, 2009). In the framework of an overall career progression program in a Public Administration, SJTs were developed to assess several competencies relevant to both task (e.g. planning, managing, etc.) and contextual performance (e.g. helping behaviour, organizational loyalty, etc.). Each competency is a combination of KSAOs needed to perform efficiently in a specific context (Campion & al., 2012; Le Boterf 2008).

Given the lack of internal consistency of SJT and the dichotomous nature of the data, we decide to use Multidimensional Item Response Theory (MIRT) model to examine the issue of construct validity. Based on Hartig & Höhler (2009), we use MIRT compensatory model to estimate the latent covariance structures of the SJT. Different model are tested and compared depending on the number of dimension involved, the nature of the items and latent variables relation (between or within-item multidimensionality) and the number of model parameter (one or two parameter model).

The results and findings lead us to discuss some implications (gaining accuracy, modeling performance in complex task, etc.) and limitations (dealing with complex loading structure, etc.) of using IRT, over and above classical test theory, in the specific case of SJTs applicants performance. The originality of this communication also lies in the way we addressed SJTs reliability and construct validity issues with a measurement based model that differs from the one usually used in the field.