The evaluation of singing voice accuracy: Are we good judges even if we are not musicians?

A previous study highlighted that objective and subjective measurements of vocal accuracy are highly correlated when the rating is done by musicians and that 81% of their variance was explained by the pitch interval deviation and the number of tonality modulations (Larrouy-Maestri, Lévêque, Schön, Giovanni, & Morsomme, 2013). However, the general audience is not expert in music. This study aims to observe the vocal accuracy assessment done by non musicians.
Eighteen non musicians were asked to rate on a 9-point scale the global pitch accuracy of 166 vocal performances sung by untrained singers. This database was presented two times (T1 and T2) in a random order. The sung performances were objectively analyzed regarding three criteria: pitch interval deviation, number of tonality modulations and number of contour errors.
The results show an intraclass correlation coefficient of .89 (p < .01) among the non experts and a mean intra-judges Spearman correlation of .66 (SD = .06) between T1 and T2. There is also a significant correlation between the acoustic measurements and the mean ratings (r(166) = .809; p < .01), with higher scores for accurate performances. Confronted with the music experts of the previous study, the judgment of the non experts correlated with the musicians ratings (r(166) = .840; p < .01). However, a Mann Whitney test showed that the rating of non experts was globally more severe than the experts one (p = 009). In addition, the regression analysis showed that only the pitch interval deviation criterion was considered in their judgment, explaining 66% of the variance of the judges.
This study highlights the reliability and the objectivity of non musicians in the vocal accuracy assessment. Moreover, an effect of music expertise is observed concerning the severity of the rating and the vocal accuracy assessment process.