Impact of operator expertise on collection of the APACHE II score and on the derived risk of death and standardized mortality ratio

We assessed the impact of operator expertise on collection of the APACHE II score, the derived risk of death and standardized mortality ratio in 465 consecutive patients admitted to a multi-disciplinary tertiary hospital ICU. Research coordinators and junior clinical staff independently collected the APACHE II variables; experts (senior clinical staff) rescored 20% of the records. Agreement was moderate between junior clinical staff and research coordinators or senior clinical staff for most variables of the acute physiology score (weighted kappa<0.6); agreement between research coordinators and senior clinical staff data collectors was good (weighted kappa >0.75). The APACHE II score and its derived risk of death (ROD) were significantly lower using the junior clinical staff dataset compared to research coordinators and senior clinical staff (APACHE II score: 13.4+/-9.2 vs 16.8+/-8.5 vs 17.1+/-7 7, P<0.001; ROD: 14.7%+/-22.4% vs 21.6%+/-22.6% vs 20.8%+/-22.4%, P<0.01 respectively). The discriminative capacity was not altered by the lack of agreement (area under Receiver Operator Characteristic curve >0.8) but calibration of ROD from the junior clinical staff dataset was poor (Goodness-of-fit: P= 0.001). The standardized mortality ratio (SMR) was higher with the junior clinical staff dataset (SMR: 1.22, 95% CI: 0.96-1.52 vs 0.87, 95% CI: 0.70-1.06 vs 0.76, 95% CI: 0.40-1.3 calculated from junior clinical staff research coordinators and senior clinical staff datasets respectively). We conclude that the expertise of data collectors significantly influences the APACHE II score, the derived risk of death and the standardized mortality ratio. Given the importance of such scores, ICUs should be provided with sufficient resources to train and employ dedicated data collectors.