Project name: SODA – Security of Decision Analysis

Full title: “Safety of decision-making methods based on pairwise comparisons”

NCN project number: 2021/41/B/HS4/03475

Project leader: Prof. Konrad Kułakowski

“Can we force politicians to tell the truth?” This is the title of an article by David Mountain in "Open Democracy" from January 19th, 2019. The title, or rather the question, is not purely rhetorical. On the one hand, one can imagine a law that compels decision-makers to be honest, but on the other hand, there are many “buts”. Aside from questions of personal freedom, including the freedom “to be dishonest”, there is the question of when a decision-maker is being dishonest versus merely imprecise or “misunderstood”. The answer to this question becomes more complicated the more closely we examine the matter. Decision-makers not only express opinions directly, but also participate in various decision-making bodies where the final opinion depends on the answers to many detailed questions. This provides ample room for interpretation, meaning that they may not be “entirely honest”. The question arises whether we (as a society, voters, people) are completely helpless in the face of such dishonesty. It turns out that we are not entirely powerless.

Decision-making, especially when it comes to significant and important matters, is usually a certain process. First, different options are selected, then they are analysed and compared by experts (decision-makers), and finally, a recommendation is formulated. The opinions developed in this way create a set of data. We can collect this data, compare it, and create recommendations, etc. Importantly, it seems that analyzing this data can provide answers to our questions. However, we still need to define how to do it.

A popular (but not the only) method of decision-making is AHP (Analytic Hierarchy Process). AHP was proposed by Thomas Saaty in 1977. This proposal is comprehensive, containing a method for calculating rankings, allowing for multiple criteria to be handled. It includes built-in verification mechanisms, such as the inconsistency index, as well as ready-to-use commercial software supporting this method. All these factors contributed to its undeniable success. AHP or more generally, the pairwise comparison method, provides well-defined frames for the decision-making process. This makes it easier to notice "strange" and "non-obvious" behavior of decision-making data. We would like these "irregularities" to help us get closer to the answer to the question of the honesty of decision-makers. After all, wouldn't it be convenient to have an algorithm (preferably simple, something like an oracle) that would decide who is honest and who is not?

Of course, the question of the honesty of decision-makers (even if we limit the considerations to AHP) is not so simple that an easy-to-write algorithm is enough for an answer. That's why in our research we want to:

• analyse various ways in which decision-makers (experts) can be dishonest,
• examine the properties of the AHP method that facilitate unfairness (unfortunately, there are probably no perfect decision-making methods).

Based on these analyses, we would like to:

• eliminate situations where dishonest behavior is easier (the type of problem, the nature of the data facilitates dishonest behavior), and on the other hand,
• identify suspicious situations.

To achieve this, we will create a series of simulation models corresponding to various types of fraud. We believe that observing these models will allow us to construct mechanisms for assessing the threat of attacks and methods for detecting dishonest behavior. This will provide users with tools to defend themselves against dishonest decision-makers, at least those who use the AHP method and pairwise comparison of alternatives. We believe that many of our proposed ideas will also find application in other decision-making methods.

Returning to the question from Mountain's quoted article: “Can we force politicians to tell the truth?” Our research will not help answer this question. However, we believe that through these studies, we will facilitate the answer to the question of who and when doesn't tell the truth, which may be the first step towards living in a more honest and predictable world.