kappa coefficient; nominal scale; ordinal scale; expert group
Résumé :
[en] The agreement between two raters judging items on a categorical scale is traditionally assessed by Cohen’s kappa coefficient. We introduce a new coefficient for quantifying the degree of agreement between an isolated rater and a group of raters on a nominal or ordinal scale. The group of raters is regarded as a whole, a reference or gold-standard group with its own heterogeneity. The coefficient, defined on a population-based model, requires a specific definition of the concept of perfect agreement. It has the same properties as Cohen’s kappa coefficient and reduces to the latter when there is only one rater in the group. The new approach overcomes the problem of consensus within the group of raters and generalizes Schouten’s index. The method is illustrated on published syphilis data and on data collected from a study assessing the ability of medical students in diagnostic reasoning when compared with expert knowledge.
Disciplines :
Mathématiques
Auteur, co-auteur :
Vanbelle, Sophie ; Université de Liège - ULiège > Département des sciences de la santé publique > Département des sciences de la santé publique
Albert, Adelin ; Université de Liège - ULiège > Département des sciences de la santé publique > Informatique médicale et biostatistique - Département de mathématique
Langue du document :
Anglais
Titre :
Agreement between an isolated rater and a group of raters
Date de publication/diffusion :
janvier 2009
Titre du périodique :
Statistica Neerlandica
ISSN :
0039-0402
Maison d'édition :
Wiley, Oxford, Royaume-Uni
Volume/Tome :
63
Fascicule/Saison :
1
Pagination :
82-100
Peer reviewed :
Peer reviewed vérifié par ORBi
Commentaire :
The definitive version is available at www.blackwell-synergy.com
Charlin, B., R. Gagnon, L. Sibert C. Van der Veluten (2002 Le test de concordance de script : un instrument d'évaluation du raisonnement clinique, Pédagogie Médicale 3, 135 144.
Cicchetti, D. V. T. Allison (1971 A new procedure for assessing reliability of scoring EEG sleep recordings, American Journal of EEG Technology 11, 101 109.
Cohen, J. (1960 A coefficient of agreement for nominal scales, Educational and Psychological Measurement 20, 37 46.
Cohen, J. (1968 Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit, Psychological Bulletin 70, 213 220.
Eckstein, M. P., T. D. Wickens, G. Aharonov, G. Ruan, C. A. Morioka J. S. Whiting (1998 Quantifying the limitations of the use of consensus expert commitees in ROC studies, Proceedings SPIE: Medical Imaging 1998: Image Perception 3340, 128 134.
Efron, B. R. J. Tibshirani (1993 An introduction to the bootstrap, Chapman and Hall, New York.
Fleiss, J. L. (1981 Statistical methods for rates and proportions, 2nd edn, John Wiley, New York.
Fleiss, J. L. J. Cohen (1973 The equivalence of weighted kappa and the intraclass correlation coefficient as measure of reliability, Educational and Psychological Measurement 33, 613 619.
Kalant, N., M. Berlinguet, J. G. Diodati, L. Dragatakis F. Marcotte (2000 How valid are utilization review tools in assessing appropriate use of acute care beds? Canadian Medical Association Journal 162, 1809 1813.
Kraemer, H. C. (1979 Ramifications of a population model for κ as a coefficient of reliability, Psychometrika 44, 461 472.
Kraemer, H. C., V. S. Periyakoil A. Noda (2002 Kappa coefficient in medical research Tutorials in Biostatistics 1, 85 105.
Landis, J. R. G. G. Koch (1977 An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers, Biometrics 33, 363 374.
Light, R. J. (1971 Measures of response agreement for qualitative data: some generalizations and alternatives, Psychological Bulletin 76, 365 377.
Miller, D. P., K. F. O'Shaughnessy, S. A. Wood R. A. Castellino (2004 Gold standards and expert panels: a pulmonary nodule case study with challenges and solutions, Proceedings SPIE: Medical Imaging 1998: Image Perception, Observer Performance and Technology Assessment 5372, 173 184.
Ruperto, N., A. Ravelli, S. Oliveira, M. Alessio, D. Mihaylova, S. Pasic, E. Cortis, M. Apaz, R. Burgos-Vargas, F. Kanakoudi-Tsakalidou, X. Norambuena, F. Corona, V. Gerloni, S. Hagelberg, A. Aggarwal, P. Dolezalova, C. M. Saad, S. C. Bae, R. Vesely, T. Avcin, H. Foster, C. Duarte, T. Herlin, G. Horneff, L. Lepore, M. van Rossum, L. Trail, A. Pistorio, B. Andersson-Gare, E. H. Giannini A. Martini (2006 Pediatric Rheumatology International Trials Organization. The Pediatric Rheumatology International Trials Organization/American College of Response to Therapy in Juvenile Systemic Lupus Erythematosus: prospective validation of the definition of improvement, Arthritis and Rheumatism (Arthritis Care and Research) 55, 355 363.
Salerno, S. M., P. C. Alguire S. W. Waxman (2003 Competency in interpretation of 12-lead electrocardiograms: a summary and appraisal of published evidence, Annals of Internal Medicine 138, 751 760.
Schouten, H. J. A. (1982 Measuring pairwise interobserver agreement when all subjects are judged by the same observers, Statistica Neerlandica 36, 45 61.
Smith, R., A. J. Copas, M. Prince, B. George, A. S. Walker S. T. Sadiq (2003 Poor sensitivity and consistency of microscopy in the diagnosis of low grade non-gonococcal urethritis, Sexually Transmitted Infections 79, 487 490.
Soeken, K. L. P. A. Prescott (1986 Issues in the use of kappa to estimate reliability, Medical care 24, 733 741.
Vanbelle, S., V. Massart, D. Giet A. Albert (2007 Test de concordance de script: un nouveau mode d'établissement des scores limitant l'effet du hasard, Pédagogie Médicale 8, 71 81.
Williams, G. W. (1976 Comparing the joint agreement of several raters with another rater, Biometrics 32, 619 627.
