Inconsistency thresholds and Benchmark Matrices in Pairwise Comparison

We studied the problem of determining justified thresholds for inconsistency indices of a pairwise comparison matrix A. The first and best-known index was introduced by T. Saaty and is based on the principal eigenvalue of A. Saaty suggested the purely heuristic 10% law of acceptance threshold for his index, which has been heavily criticised by many authors. Thereafter, many other different definitions of inconsistency indices have been proposed, but, in our view, little attention has been paid to the problem of determining reliable and well-justified numerical thresholds for identifying satisfactorily consistent judgements. We investigated, by means of numerical simulations, the behaviour of various known inconsistency indices with respect to what we call ‘Benchmark Matrices’. By ‘Benchmark Matrices’, we mean simple pairwise comparison matrices which inconsistency level is easily recognizable and comprehensible. The obtained results allow us to compare numerically the different inconsistency indices and to determine equivalent inconsistency thresholds corresponding to the same inconsistency levels for different indices. We used different classes of Benchmark Matrices. Some examples are: (1) Pairwise comparison matrices that can be made consistent by a minimal change of a single comparison [1], (2) Pairwise comparison matrices of order 3, where the unique consistency condition is violated by a fixed extent, (3) The so-called ‘corner matrices’. Our analysis is supported by several plots that point out the role of the parameters characterising each class of Benchmark Matrices.