A Quantitative Evaluation of Trademark Search Engines’ Performances through Large-Scale Statistical Analysis

Intellectual Property Offices now offer their users trademark search engines to help them identify earlier trademarks in their register. Such tools have proven to be extremely useful given the growing number of trademarks registered but have never been subjected to thorough evaluation, despite the necessity for openness and accountability in justice systems. Additionally, their performance is unknown, in particular the reliability of their results pertaining to applicable legal rules. In fact, their "black box nature" makes automatic and at-scale evaluation hard to perform directly, which is why we propose a novel method for evaluating their performance using settled case-law for ground truth, and at-scale analysis. Based on this methodology, we evidence the performance for two such systems, the Benelux Office of Intellectual Property (BOIP) and European Union Intellectual Property Office (EUIPO), using 8 126 opposition division decisions from the EUIPO. We show important disparities between the two systems, along with surprisingly good results for EUIPO’s system.