Using Class-probability Models instead of Hard Classifiers as Base Learners in the Ranking by Pairwise Comparison Algorithm

In the field of Preference Learning, the Ranking by Pairwise Comparison algorithm (RPC) consists of using the learning sample to derive pairwise comparators for each possible pair of class labels, and then aggregating the predictions of the whole set of pairwise comparators for a given object in order to produce a global ranking of the class labels. In its standard form, RPC uses hard binary classifiers assigning an integer (0/1) score to each class concerned by a pairwise comparison. In the present work, we compare this setting with a modified version of RPC, where soft binary class-probability models replace the binary classifiers. To this end, we compare ensembles of extremely randomized classprobability estimation trees with ensembles of extremely randomized classification trees. We empirically show that both approaches lead to equivalent results in terms of Spearman’s rho value when using the optimal settings of their metaparameters. However, we also show that in the context of small and noisy datasets (e.g. with partial ranking information) the use of class-probability models is more robust with respect to variations of its meta-parameter values than the hard classifier ensembles. This suggests that using (soft) class-probability comparators is a sensible option in the context of RPC approaches.