kappa coefficient; nominal scale; ordinal scale; expert group
Abstract :
[en] The agreement between two raters judging items on a categorical scale is traditionally assessed by Cohen’s kappa coefficient. We introduce a new coefficient for quantifying the degree of agreement between an isolated rater and a group of raters on a nominal or ordinal scale. The group of raters is regarded as a whole, a reference or gold-standard group with its own heterogeneity. The coefficient, defined on a population-based model, requires a specific definition of the concept of perfect agreement. It has the same properties as Cohen’s kappa coefficient and reduces to the latter when there is only one rater in the group. The new approach overcomes the problem of consensus within the group of raters and generalizes Schouten’s index. The method is illustrated on published syphilis data and on data collected from a study assessing the ability of medical students in diagnostic reasoning when compared with expert knowledge.
Disciplines :
Mathematics
Author, co-author :
Vanbelle, Sophie ; Université de Liège - ULiège > Département des sciences de la santé publique > Département des sciences de la santé publique
Albert, Adelin ; Université de Liège - ULiège > Département des sciences de la santé publique > Informatique médicale et biostatistique - Département de mathématique
Language :
English
Title :
Agreement between an isolated rater and a group of raters
Publication date :
January 2009
Journal title :
Statistica Neerlandica
ISSN :
0039-0402
Publisher :
Blackwell Publishing, Oxford, United Kingdom
Volume :
63
Issue :
1
Pages :
82-100
Peer reviewed :
Peer Reviewed verified by ORBi
Commentary :
The definitive version is available at www.blackwell-synergy.com
scite shows how a scientific paper has been cited by providing the context of the citation, a classification describing whether it supports, mentions, or contrasts the cited claim, and a label indicating in which section the citation was made.
Bibliography
Charlin, B., R. Gagnon, L. Sibert C. Van der Veluten (2002 Le test de concordance de script : un instrument d'évaluation du raisonnement clinique, Pédagogie Médicale 3, 135 144.
Cicchetti, D. V. T. Allison (1971 A new procedure for assessing reliability of scoring EEG sleep recordings, American Journal of EEG Technology 11, 101 109.
Cohen, J. (1960 A coefficient of agreement for nominal scales, Educational and Psychological Measurement 20, 37 46.
Cohen, J. (1968 Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit, Psychological Bulletin 70, 213 220.
Eckstein, M. P., T. D. Wickens, G. Aharonov, G. Ruan, C. A. Morioka J. S. Whiting (1998 Quantifying the limitations of the use of consensus expert commitees in ROC studies, Proceedings SPIE: Medical Imaging 1998: Image Perception 3340, 128 134.
Efron, B. R. J. Tibshirani (1993 An introduction to the bootstrap, Chapman and Hall, New York.
Fleiss, J. L. (1981 Statistical methods for rates and proportions, 2nd edn, John Wiley, New York.
Fleiss, J. L. J. Cohen (1973 The equivalence of weighted kappa and the intraclass correlation coefficient as measure of reliability, Educational and Psychological Measurement 33, 613 619.
Kalant, N., M. Berlinguet, J. G. Diodati, L. Dragatakis F. Marcotte (2000 How valid are utilization review tools in assessing appropriate use of acute care beds? Canadian Medical Association Journal 162, 1809 1813.
Kraemer, H. C. (1979 Ramifications of a population model for κ as a coefficient of reliability, Psychometrika 44, 461 472.
Kraemer, H. C., V. S. Periyakoil A. Noda (2002 Kappa coefficient in medical research Tutorials in Biostatistics 1, 85 105.
Landis, J. R. G. G. Koch (1977 An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers, Biometrics 33, 363 374.
Light, R. J. (1971 Measures of response agreement for qualitative data: some generalizations and alternatives, Psychological Bulletin 76, 365 377.
Miller, D. P., K. F. O'Shaughnessy, S. A. Wood R. A. Castellino (2004 Gold standards and expert panels: a pulmonary nodule case study with challenges and solutions, Proceedings SPIE: Medical Imaging 1998: Image Perception, Observer Performance and Technology Assessment 5372, 173 184.
Ruperto, N., A. Ravelli, S. Oliveira, M. Alessio, D. Mihaylova, S. Pasic, E. Cortis, M. Apaz, R. Burgos-Vargas, F. Kanakoudi-Tsakalidou, X. Norambuena, F. Corona, V. Gerloni, S. Hagelberg, A. Aggarwal, P. Dolezalova, C. M. Saad, S. C. Bae, R. Vesely, T. Avcin, H. Foster, C. Duarte, T. Herlin, G. Horneff, L. Lepore, M. van Rossum, L. Trail, A. Pistorio, B. Andersson-Gare, E. H. Giannini A. Martini (2006 Pediatric Rheumatology International Trials Organization. The Pediatric Rheumatology International Trials Organization/American College of Response to Therapy in Juvenile Systemic Lupus Erythematosus: prospective validation of the definition of improvement, Arthritis and Rheumatism (Arthritis Care and Research) 55, 355 363.
Salerno, S. M., P. C. Alguire S. W. Waxman (2003 Competency in interpretation of 12-lead electrocardiograms: a summary and appraisal of published evidence, Annals of Internal Medicine 138, 751 760.
Schouten, H. J. A. (1982 Measuring pairwise interobserver agreement when all subjects are judged by the same observers, Statistica Neerlandica 36, 45 61.
Smith, R., A. J. Copas, M. Prince, B. George, A. S. Walker S. T. Sadiq (2003 Poor sensitivity and consistency of microscopy in the diagnosis of low grade non-gonococcal urethritis, Sexually Transmitted Infections 79, 487 490.
Soeken, K. L. P. A. Prescott (1986 Issues in the use of kappa to estimate reliability, Medical care 24, 733 741.
Vanbelle, S., V. Massart, D. Giet A. Albert (2007 Test de concordance de script: un nouveau mode d'établissement des scores limitant l'effet du hasard, Pédagogie Médicale 8, 71 81.
Williams, G. W. (1976 Comparing the joint agreement of several raters with another rater, Biometrics 32, 619 627.
This website uses cookies to improve user experience. Read more
Save & Close
Accept all
Decline all
Show detailsHide details
Cookie declaration
About cookies
Strictly necessary
Performance
Strictly necessary cookies allow core website functionality such as user login and account management. The website cannot be used properly without strictly necessary cookies.
This cookie is used by Cookie-Script.com service to remember visitor cookie consent preferences. It is necessary for Cookie-Script.com cookie banner to work properly.
Performance cookies are used to see how visitors use the website, eg. analytics cookies. Those cookies cannot be used to directly identify a certain visitor.
Used to store the attribution information, the referrer initially used to visit the website
Cookies are small text files that are placed on your computer by websites that you visit. Websites use cookies to help users navigate efficiently and perform certain functions. Cookies that are required for the website to operate properly are allowed to be set without your permission. All other cookies need to be approved before they can be set in the browser.
You can change your consent to cookie usage at any time on our Privacy Policy page.