Potentially All Pairwise RanKings of all possible Alternatives (PAPRIKA) is a method for multi-criteria decision making (MCDM) or conjoint analysis, as implemented by decision-making software and conjoint analysis products 1000minds and MeenyMo.
The PAPRIKA method is based on users expressing their preferences with respect to the relative importance of the criteria or attributes of interest for the decision or choice at hand by pairwise comparing (ranking) alternatives.
In MCDM applications, PAPRIKA is used by decision-makers to determine weights on the criteria for the decision being made, representing their relative importance. Depending on the application, these weights are used to rank, prioritize or choose between alternatives.
In conjoint analysis applications, PAPRIKA is used with consumers or other stakeholders to estimate 'part-worth utilities' (i.e. weights) representing the relative importance of the attributes characterizing products or other objects of interest (i.e., choice modelling, conjoint analysis and discrete choice).
Applications The PAPRIKA method is implemented by decision-making software and conjoint analysis products 1000minds and MeenyMo.
Examples of areas in which the method is used for multi-criteria decision making or conjoint analysis include (see also 1000minds applications): Patient and health technology prioritization Disease diagnosis and classification Clinical guidelines development Disease R&D prioritization Marketing research (conjoint analysis) Environmental resources management and climate change research Animal and plant breeding Urban planning and waste management Information and communications technology (ICT) Research into monetary policy, retirement income policies and charitable giving
Additive multi-attribute value models The PAPRIKA method specifically applies to additive multi-attribute value models with performance categoriesBelton, V and Stewart, TJ, Multiple Criteria Decision Analysis: An Integrated Approach, Kluwer: Boston, 2002, pp. 166–168. – also known as 'points', 'scoring', 'point-count' or 'linear' systems or models. The following explanations are mostly couched in terms of multi-criteria decision making. Analogous explanations in terms of conjoint analysis are possible but not presented here.
As the name implies, additive multi-attribute value models with performance categories – hereinafter referred to simply as 'value models' – consist of multiple criteria (or 'attributes'), with two or more performance categories (or 'levels') within each criterion, that are combined additively.
Each category within each criterion is worth a certain number of points that is intended to reflect both the relative importance ('weight') of the criterion and its degree of achievement. For each alternative being considered, the point values are summed across the criteria to get a total score – hence, these are additive value models – by which the alternatives are prioritized or ranked (or otherwise classified) relative to each other.
Thus, a value model (or 'points system') is simply a schedule of criteria (and categories) and point values for the decision problem at hand; for an example, see Table 1 in the sub-section below. This 'points system' representation is equivalent to a more traditional approach involving normalized criterion weights and 'single-criterion value functions' to represent the relative importance of the criteria and to combine values overall (see weighted sum model). The unweighted points system representation is easier to use and helps inform the explanation of the PAPRIKA method below.
An example application of a points system An example application of a points system is ranking candidates applying for a job.
Imagine that 'Maartje', 'Michelle' and 'Paulien' are three job candidates to be ranked using the value model in Table 1 below. Suppose they are assessed on the five criteria (see Table 1) like this: Maartje's education is excellent, she has > 5 years of experience, and her references, social skills and enthusiasm are all poor. Michelle's education is poor, she has 2–5 years of experience, and her references, social skills and enthusiasm are all good. Paulien's education is good, she has < 2 years of experience, and her references, social skills and enthusiasm are all good.
Table 1: Example of a value model (points system) for ranking job candidates Criterion Category Points Education poor 0 good 8 very good 20 excellent 40 Experience < 2 years 0 2–5 years 3 > 5 years 10 References poor 0 good 27 Social skills poor 0 good 10 Enthusiasm poor 0 good 13
Summing the point values in Table 1 corresponding to the descriptions for Maartje, Michelle and Paulien gives their total scores: Maartje's total score = 40 + 10 + 0 + 0 + 0 = 50 points Michelle's total score = 0 + 3 + 27 + 10 + 13 = 53 points Paulien's total score = 8 + 0 + 27 + 10 + 13 = 58 points
Clearly, Paulien has the highest total score. Therefore, according to the value model (and how Maartje, Michelle and Paulien were assessed) Paulien is the best candidate for the job. (Though, clearly, relative to other candidates who could potentially have applied, Paulien is not as good as the best hypothetically-possible candidate – who would score a 'perfect' 40 + 10 + 27 + 10 + 13 = 100 points.)
In general terms, having specified the criteria and categories for a given value model, the challenge is to derive point values that accurately reflect the relative importance of the criteria and categories to the decision-maker. Deriving valid and reliable point values is arguably the most difficult task when creating a value model. The PAPRIKA method does this based on decision-makers' preferences as expressed using pairwise rankings of alternatives.
Overview of the PAPRIKA method As mentioned at the start of the article, PAPRIKA is a (partial) acronym for 'Potentially All Pairwise RanKings of all possible Alternatives'. The following explanation should make clear this name's derivation.
The PAPRIKA method pertains to value models for ranking particular alternatives that are known to decision-makers (e.g. as in the job candidates example above) and also to models for ranking potentially all hypothetically possible alternatives in a pool that is changing over time (e.g. patients presenting for medical care). The following explanation is centered on this second type of application because it is more general.
PAPRIKA is based on the fundamental principle that an overall ranking of all possible alternatives representable by a given value model – i.e. all possible combinations of the categories on the criteria – is defined when all pairwise rankings of the alternatives vis-à-vis each other are known (and provided the rankings are consistent).
(As an analogy, suppose you wanted to rank everyone living in a given city from the youngest to the oldest. If you knew how each person was pairwise ranked relative to everyone else with respect to their ages – i.e. for each possible pair of individuals, you identified who is the younger of the two individuals or that they're the same age – then you could produce an overall ranking of the city's population from the youngest to the oldest.)
However, depending on the number of criteria and categories included in the value model, the number of pairwise rankings of all possible alternatives is potentially in the millions or even billions. Of course, though, many of these pairwise rankings are automatically resolved due to one alternative in the pair having a higher category for at least one criterion and none lower for the other criteria than for the other alternative – known as 'dominated pairs'.
But this still leaves potentially millions or billions of undominated pairs' – pairs of alternatives where one has a higher ranked category for at least one criterion and a lower ranked category for at least one other criterion than the other alternative – and hence a judgment is required for the alternatives to be pairwise ranked. With reference to the example of ranking job candidates in the previous section, an example of an undominated pair (of candidates) would be where one person in the pair is, say, highly educated but inexperienced whereas the other person is uneducated but highly experienced, and so a judgement is required to pairwise rank this (undominated) pair.
For n possible alternatives, there are n(n−1)/2 pairwise rankings. For example, for a value model with eight criteria and four categories within each criterion, and hence 48 = 65,536 possible alternatives, there are 65,536 x 65,535 / 2 = 2,147,450,880 pairwise rankings. Even after eliminating the 99,934,464 dominated pairs, there are still 2,047,516,416 undominated pairs to be ranked. Clearly, performing anywhere near this number of pairwise rankings – more than two billion! – is humanly impossible without a special method.
The PAPRIKA method resolves this 'impossibility' problem by ensuring that the number of pairwise rankings that decision-makers need to perform is kept to a minimum – i.e. only a small fraction of the potentially millions or billions of undominated pairs – so that the burden on decision-makers is minimized and the method is practicable. PAPRIKA keeps the number of pairwise rankings performed by decision-makers to a minimum by, for each undominated pair explicitly ranked by decision-makers, identifying (and eliminating) all undominated pairs implicitly ranked as corollaries of this and other explicitly ranked pairs. Fundamental to the efficiency of the method is application of the transitivity property of additive value models, as illustrated in the simple demonstration later below.
The PAPRIKA method begins with the decision-maker pairwise ranking undominated pairs defined on just two criteria at-a-time (where, in effect, all other criteria's categories are pairwise identical). Again with reference to the example of ranking job candidates, an example of such a pairwise-ranking question is: "Who would you prefer to hire, someone whose education is poor but he or she has 5 years or more experience or another person whose education is excellent but he or she has less than 2 years experience, all else being equal?" (see Figure 1).
Figure 1: Example of a pairwise-ranking question (a screenshot from 1000minds)
Each time the decision-maker ranks a pair (such as the example above), all undominated pairs implicitly ranked as corollaries are identified and discarded. After having completed ranking undominated pairs defined on just two criteria at-a-time, this is followed, if the decision-maker chooses to continue (she can stop at any time), by pairs with successively more criteria (i.e. three criteria, then four, then five, etc.), until potentially all undominated pairs are ranked.
Thus, Potentially All Pairwise RanKings of all possible A'lternatives (hence the PAPRIKA acronym) are identified as either: (1) dominated pairs (given), or (2) undominated pairs explicitly ranked by the decision-maker, or (3) undominated pairs implicitly ranked as corollaries. From the explicitly ranked pairs, point values (weights) are obtained via linear programming; although multiple solutions to the linear program are possible, the resulting point values all reproduce the same overall ranking of alternatives.
Simulations of PAPRIKA's use reveal that if the decision-maker stops after having ranked undominated pairs defined on just two criteria at-a-time, the resulting overall ranking of all possible alternatives is very highly correlated with the decision-maker's 'true' overall ranking obtained if all undominated pairs (involving more than two criteria) were ranked.