A small overview of available computer software to support computerized adaptive testing

Computerized adaptive testing (CAT) is becoming a central tool for testing and assessment. It offers many advantages over fixed (“paper-and-pencil”) methods, such as individualized assessment, reduction of fraud, and straightforward estimation of proficiency levels. CAT has been studied for decades and remains an up-to-date research field in psychometrics and educational science. Practical CAT administration, however, is less frequently considered in such studies. Assigning CAT to respondents requires both the sufficient availability of computer machines, and the use of a powerful and easy-to-use CAT software. With the fast increase of computer resources at moderate cost, the availability of computer machines is becoming a less central, yet important, issue in the practical assessment of CAT tests.
The choice of an accurate CAT software, on the other hand, should be guided by its flexibility, its underlying statistical modeling, and its user-friendly potential. According to the type of research or data analysis, some CAT software might be preferred to another. It is therefore important for the researcher or the clinician to know about the current availability of such software, in line with current research and practice in the CAT framework. Moreover, these software should allow enough flexibility to incorporate updates and new theoretical developments, such as e.g., new rules for next item selection.
This talk proposes a simple and user-oriented presentation of several CAT software that are currently available. The software to be presented are: the Firestar software (Choi, 2009), the R package catR (Magis & Raîche, 2012), the R package catIrt (Nydick, 2012) and the CAT web-platform Concerto (Kosinski & Rust, 2011). The first three are non-commercial software, while Concerto is a web interface between end users (willing to develop computerized assessment tests) and catR (as underlying routine software). Both R packages are written to be most useful for researchers, without end-user interface, and are therefore less appealing for applied researchers who are not familiar with R. Yet, they offer flexible solutions by means of many options to optimize the design of the test and generate many response patterns for further analyses. Also, they can be easily integrated as sub-routines for more sophisticated CAT software. Firestar provides a user interface and makes all necessary computations with underlying R code.
This talk aims at focusing on freely available CAT software. For this reason, only the four aforementioned programs will be presented, although it exists other, commercial CAT software such as e.g., the CATSim software (Assessment Systems Corporation, 2012).
The different CAT software are briefly presented and their advantages and drawbacks, flexibility and usefulness are compared, mostly from the point of view of the applied researcher and clinician. The following criteria were retained for objective comparison: (a) their main goal of application; (b) the type of data and IRT modeling they can deal with; (c) the type of users they are focusing on; (d) their operating options; (e) their availability and flexibility for further improvements.
A small demonstration of the R package catR will be proposed optionally, depending on time limitation.

References
Assessment Systems Corporation (2012). CATSim: Comprehensive simulation of computerized adaptive testing. St. Paul, MN. URL: http://www.assess.com/.
Choi , S. W. (2009). Firestar: Computerized adaptive testing simulation program for polytomous item response theory models. Applied Psychological Measurement, 33, 644-645.
Kosinski, M., & Rust, J. (2011). The development of Concerto: An open source online adaptive testing platform. Paper presented at the International Association for Computerized and Adaptive Testing (IACAT), Pacific Grove, CA.
Magis, D., & Raîche, G. (2012). Random generation of response patterns under computerized adaptive testing with the R package catR. Journal of Statistical Software, 48, 1-31.
Nydick, S. W. (2012). catIrt: An R package for simulating IRT-based computerized adaptive tests. R package version 0.3-0.