Looking for a reference for large datasets: relative reliability of visual and automatic sleep scoring

Study Objectives: New challenges in sleep science require to describe fine grain phenomena or to
deal with large datasets. Beside the human resource challenge of scoring huge datasets, the interand
intra-expert variability may also reduce the sensitivity of such studies. Searching for a way to
disentangle the variability induced by the scoring method from the actual variability in the data,
visual and automatic sleep scorings of healthy individuals were examined.
Methods: A first dataset (DS1, 4 recordings) scored by 6 experts plus an autoscoring algorithm was
used to characterize inter-scoring variability. A second dataset (DS2, 88 recordings) scored a few
weeks later was used to investigate intra-expert variability. Percentage agreements and Conger’s
kappa were derived from epoch-by-epoch comparisons on pairwise, consensus and majority scorings.
Results: On DS1 the number of epochs of agreement decreased when the number of expert
increased, in both majority and consensus scoring, where agreement ranged from 86% (pairwise) to
69% (all experts). Adding autoscoring to visual scorings changed the kappa value from 0.81 to 0.79.
Agreement between expert consensus and autoscoring was 93%. On DS2 intra-expert variability was
evidenced by the kappa systematic decrease between autoscoring and each single expert between
datasets (0.75 to 0.70).
Conclusions: Visual scoring induces inter- and intra-expert variability, which is difficult to address
especially in big data studies. When proven to be reliable and if perfectly reproducible, autoscoring
methods can cope with intra-scorer variability making them a sensible option when dealing with
large datasets.