A Lens Model Based Judge-Participant Benchmark Repository and Its Application to Select Ranking Methods

In a judging system where multiple participants are evaluated by multiple judges, selecting a suitable ranking method to rank participants is a challenge for the organizer(s). More so in cases where the participant pool is large, the number of judges evaluating them is limited and the outcome is expected to be fair. Adding to the already existing complexities, each judging system has its own unique features and discrepancies that are required to be considered. This work introduces a multi-label classification approach that considers several features inherent to the score matrix in the judging system to recommend suitable ranking methods. The performance of existing ranking methods has been compared using a true rank condition as benchmark using a simulated bed. The simulation bed, called the judge-participant benchmark repository, has been constructed using the Lens Model framework to mimic real world score matrices and is further used to train the proposed method recommender. The repository of score matrices has been made available for researchers to be used as a benchmark for newly devised ranking methods in decision making problems.