DEVELOPMENT OF A TECHNOLOGY OF STRUCTURING GROUP EXPERT JUDGMENTS UNDER VARIOUS TYPES OF UNCERTAINTY

The study considers the problem of structuring expert judgments formed under conditions of uncertainty of different nature and in presence of conflicting expert evidence. The method of aggregating group expert judgments that are formed under conditions of various types of uncertainty helps synthesize the group opinion, taking into account various forms of representing the preferences of experts (interval, fuzzy and crisp expert judgments). The proposed procedure makes it possible to synthesize a group decision in the event that there is a group or several groups of experts in a group of experts who express their preferences using different forms of expert judgments. This approach allows reflecting accurately the expert preferences regarding the object being analysed, without restricting the experts to a rigid form of presenting assessments. In order to analyse the obtained expert information and to get individual expert ratings of the analysed objects, the method of pairwise comparison and its modification were used in the study. It has been established that for the aggregation of crisp expert estimates, more precise combined results can be obtained by applying rules for redistributing conflicts of the theory of plausible and paradoxical reasoning. To aggregate interval expert judgments, one of the combination rules of the theory of evidence is recommended. It has been determined that in order to improve the quality of the aggregate results, it is advisable to establish a procedure for combining expert inputs, for example, taking into account the degree of dissimilarity and the structure of expert evidence. The obtained results are intended to help improve the quality and efficiency of the processes of preparing and making decisions while solving the problems of analysing and structuring expert judgments.


Introduction
In general terms, the technology of structuring group expert judgments can be presented in the form of successive steps, Fig. 1.
A panel of experts { | 1, } j E j t Ε = = is given one and the same set of options (objects of expertise, alternatives) = and the same instruction containing information as to what type of scorecard in which the experts will express their preferences will be used.It depends on the type of information received from the experts (words, conditional gradations, numbers, rankings, breakdowns or other types of objects of non-numeric nature).

As a result, a plurality of individual expert judgments
= is formed.The established set of expert judgments (EJs) enters the block of structuring procedures to perform operations of ranking, clusterization, and others.The obtained data fall into the block of evaluating the results of structuring, which contains a set of conditions that determines the correctness of the structurization block.
When solving problems of analysing group expert judgments and choosing appropriate methods (Fig. 2), two important circumstances should be taken into account: -availability of diverse scales of expert measurements and a large number of different forms of representing expert
Currently, the most widely used methods of analysing expert judgments obtained in the scale of relations are the method of pairwise comparison and the analytic hierarchy process (AHP) as well as its modification.
The idea of a pairwise comparison method consists in comparing the elements (objects) proposed by experts to one another in order to obtain an individual or collective ranking or to choose the best option.The experts pair each two objects together and evaluate the significance of one object relative to the other.As a result, strict or non-rigorous individual ranks of objects can be formed, for example, if their equivalence is recorded.
Despite the ease of implementation and the wide spread, it should be noted that the method is not devoid of a number of shortcomings, of which the following should be noted: -in case of insignificant differences between the objects to achieve a satisfactory ranking, it is not always possible; -with an increase in the number of comparable pairwise elements (n≥6), it is often difficult to achieve a high level of consistency between experts; -with a large number of objects being compared, it is necessary to construct a large number of inverse-symmetric matrices; -in the classical method of pairwise comparison, an expert only works with rigorous assessments of objects and avoids uncertainty in judgments.
The main disadvantage of the methods based on the procedures of pairwise comparison is that they can be used only for a small number of comparable elements.
At present, a rather large class of modern methods has been formed to overcome these shortcomings.The proposed modifications of the paired comparison method help simulate inaccuracy and uncertainty in experts' assessments, both at the stage of identifying expert preferences and at the stage of obtaining local priority vectors.

Literature review and problem statement
By the form of presenting expert opinions, the modification of the dual comparison method can be classified as follows: on the basis of crisp expert evaluations; based on fuzzy expert assessments; based on interval estimates of experts.
In [1], a review and a classification of methods for obtaining the priority vector from fuzzy matrices of pairwise comparisons are presented, and a modified AHP based on fuzzy expert information is suggested.In order to find the values of the priority vector from the fuzzy matrix of pairwise comparisons (MPC), it is proposed in [2] to carry out the procedure of dephasing the elements of the fuzzy MPC by the gravity method; in [3], it is suggested to use the metric of the Euclidean distance.As a disadvantage of this approach, the complexity of mathematical calculations should be noted.When applying the gravity method, it should be taken into account that the range of values of the output variable will be narrower than the interval at which it is defined.In [4], a method for obtaining a priority vector from the fuzzy MPC based on an evolutionary algorithm is proposed.The method makes it possible to process the fuzzy MPC the elements of which are presented as triangular and trapezoidal numbers, as well as to obtain crisp values of the priority vector and to assess the consistency of expert information.However, a disadvantage is the limited number of the compared items.In [5], the classification and analysis of methods for obtaining the priority vector from interval matrices of pairwise comparisons is performed.In order to obtain the values of the vector of local priorities in the interval MPC, the study considers modifications of the method of pairwise comparison on the basis of the goal programming method (GPM), linear programming method (LPM), and nonlinear programming method (NLPM).Examples of practical implementation are given.
To overcome the limit on the number of comparable objects, several modifications of the paired comparison method are proposed.In [6], it is suggested to apply a shortened procedure for the formation of the MPC and to divide the received expert information into groups, for each of which and for the results of the examination as a whole the vectors of local priorities are calculated.In [7], two versions of the modified AHP are considered.The first version corresponds to the case where information about the patterns of distortion of the values of the empirical matrix of pairwise comparisons γ ij is absent; it is proposed to find the weighting of the priority vectors based on the optimization models.In [8], it is suggested to construct truncated matrices of pairwise comparisons; the expert is allowed to allocate and evaluate certain subsets of alternative variants within the notation of the theory of evidence.This approach allows taking into account the uncertainty and inaccuracy in experts' assessments.Among the shortcomings, it is necessary to note the lack of consistency of expert assessments and the possibility of obtaining zero values of the priority vector.
Convolutional methods have been widely used to solve the task of structuring the expert judgments generated by the methods of pairwise comparison and to synthesize the group decision on the order of the preferences of the options considered in the case where the value of the weight of the priority vector is represented by crisp estimates.Among them there are additive, multiplicative and nonlinear convolutions.The limitations of such an approach include the absence of criteria for a reasonable choice of the convolution type, the necessity to form weight coefficients, as well as the possibility of compensating small values for one indicator and large values for other indicators [9].
For synthesis of a group decision, in the case where the value of the weight of the priority vector is represented by interval numbers, the linear programming method (LPM) is used.The main disadvantage is the computational complexity of the method.If the result of aggregating the interval values of the weights of the priorities of the experts' vectors is a priority vector whose values are represented by interval numbers, then there is a problem of comparing interval-given numbers.
The aforementioned methods of aggregating expert information do not allow taking into account the form of representing expert preferences and processing conflicting expert assessments; they are not able to operate with expert evidence having a different structure, in particular, to combine and intersect.

The aim and objects of the study
The aim of the research is to study the problem of structuring group expert judgments that are formed under various types of uncertainty and to develop a mathematical model of structuring (ranking) group expert judgments taking into account various forms of representing expert preferences.
To achieve the aim, the following tasks are set and done: -to propose a method of aggregating group expert judgments, which allows taking into account the form of submitting expert assessments; -to choose an effective combination algorithm for obtaining aggregated expert evaluations; -to conduct a computational experiment and an analysis of the obtained results.

Materials and methods of studying the problem of structuring group expert judgments
Methods that are based on the procedures of pairwise comparison make it possible to evaluate the significance of one object in relation to the other within a given scale of preferences.
For example, if there are two objects O 1 and O 2 , then only three variants of the result are possible when comparing these objects pairwise: The profile B j that has been formed by the expert E j reflects its preferences and presents its assessment in the form of a reciprocally symmetric matrix of the type where 1/ , a ij means expert preconditions formed within a given type of the scale.
To evaluate m objects of expert examination, it is necessary to perform m(m-1)/2 pairwise comparisons.
The basis of analysing matrix (1) is the procedure for finding the priority vectors, which, provided that a ij is expressed by crisp expert evaluation, is realized by the scheme of determining the geometric mean [10].Such an estimate is most characteristic of the scale of relations [10]: The verification of the coherence of the elements of matrix ( 1) is carried out on the basis of calculating the consistency ratio: where CI=(λ max -m)/(m-1) is the consistency index; m is the number of items to be compared; λ max is the maximum actual number of the matrix of pairwise comparisons; RI is the random index [10].
The calculated eigenvector from the matrix of paired comparisons is acceptable in the case if CR≤0.10.
If a ij is represented by a fuzzy number -triangular ( , , , ), then, as a result of the procedure of pairwise comparisons, a fuzzy matrix of pairwise comparisons of type ( 1) is formed.
The verification of the consistency of elements of the fuzzy matrix of pairwise comparisons whose elements are represented by triangular fuzzy numbers (TFNs) can be performed according to the following scheme [11,12]: where ( , , ) j j j j w w w w = is the vector of local priorities; m is the dimension of the matrix.
If CCI=0, the matrix is considered to be absolutely consistent.For the matrix of order m=3, the threshold values are CCI=0.3147;for m=4, they are CCI=0.3526;and for m>4, they are CCI=0.370.
To obtain the values of the vector of local priorities in the fuzzy matrix of pairwise comparisons presented by the TFNs, Chang's method [13] can be applied, which helps obtain crisp estimates of the values of the priority vector.The essence of the method is as follows.
1. Find the sum of the elements (ratings) of each line and normalize the obtained value: ; where is the arithmetic operation of TFN multiplication.
4. Calculate the value of the priority vector: If a ij is represented by an interval number, then, as a result of the procedure of pairwise comparisons, an interval matrix of pairwise comparisons (IMPC) of type ( 1) is formed, where The IMPC is coordinated if the next admissible area is not empty [14]: The IMPC is coordinated if its elements satisfy the inequalities In order to obtain the values of the vector of local priorities in the IMPC, the following methods have become widely used [5,14,15]: the linear goal programming method (LGPM), the lower and upper approximation method (LUAM), and the two stage linear goal programming method (TSLGPM).
Let us consider the situation in which a group of experts includes such experts or subgroups of experts as E⟹{{Gr 1 }, {Gr 2 },…,{Gr p }}, (Gr p ⊆ E, {Gr p }={E 1 ,…, E r }, t≥r≥1), t=|E|, expressing their preferences using different forms of expressing expert judgments.
For example, according to the results of an expert survey, a group of experts is divided into two subgroups E⟹{{Gr 1 }, {Gr 2 }}.Experts from the group Gr 1 , performing the procedure of pairwise comparison of alternatives, have expressed crisp expert opinions; Gr 2 group experts have formed fuzzy expert judgments.
The profile B i =<A>, which is formed by the expert 1 i

E Gr
Î , reflects its preferences and presents its estimates in the form of a matrix of paired comparisons of type (1) with the crisp values of expert judgments formed within a given verbal scale.
The profile B j =<A>, which is formed by the expert 2 j

E Gr
Î , reflects its preferences and presents its estimates in the form of a matrix of paired comparisons of type (1), where expert assessments a ij are presented in the form of triangular or trapezoidal fuzzy numbers.The task is to develop a group decision that allows taking into account the form of submitting assessments by all experts.
Formally, the procedure for finding a group expert opinion can be presented in the form of the following consecutive steps: 1. Determining the set of objects of expertise (alternatives).
2. Performing a procedure for identifying the priorities of alternatives.In the framework of this stage, expert preferences are determined and the matrices of pairwise comparisons of alternatives are formed.
3. Calculating the vector of matrix priorities of pairwise comparisons, taking into account the form of representing expert assessments.
As a result, the established set is = in which each element is a vector of local priorities, calculated on the basis of expert judgments The choice of the method is based on the form of submitting expert assessments: crisp expert evaluations, fuzzy expert assessments, and interval expert assessments.
4. Verifying the consistency of expert assessments.

Aggregating individual expert assessments into a collective set. The aggregation procedure is performed by combining the obtained values
The aggregation procedure is carried out in two stages: in the first stage, there is an aggregation of W i and W j , the elements of which are given in the same presentation form.For example, the elements of the vectors W i and W j are represented by interval numbers.In the second stage, if necessary, the aggregation of W k and W p is performed, the elements of which have different representation forms -crisp and interval.
For the aggregated estimates, it is recommended to use one of the rules for redistributing conflicts.The resulting combined probability masses are calculated by adding parts of the total conflict mass or local conflict mass to the corresponding value of the basic confidence mass m(х).In this case, the resulting subsets correspond to the output, and new subsets are not formed.
The composite bulk of confidence m PCR5 (C), according to the rule of redistribution of conflicts PCR5 ( where m 12 (C) is the basic belief assignment for the subsets C=X∩Y, which is calculated on the basis of a conjunctive consensus.
For the aggregation of interval expert assessments, it is recommended to use one of the combination rules in the theory of evidence [17][18][19].When choosing a combination rule, it is necessary to prioritize a number of criteria for which one or another combination rule will be evaluated.As criteria of choice, the following combination rules may be considered: information on the sources of data (experts), their compe-tence, and the nature of the data analysed (local conflicts, the structure of expert judgments, etc.).
To obtain more effective combination results, it is proposed to determine the order of combining on the basis of the metrics of the theory of evidence [20][21][22][23].The value of the metric d(W 1 , W 2 )∈[0, 1] is the distance representing the difference and expressing the degree of conflict between W 1 and W 2 .For the aggregation of the corresponding values of W 1 ,…, W k , (k≤t), the elements of which have the same representation, at each stage, the W i and W j are selected, for which the fulfilled condition is min(d(W i , W j )), , 1, .i j r ∀ = The result of the combination is the vector of local priorities reflecting the group assessment.

Results of the study of the problem of structuring group expert judgments
Let us consider examples illustrating the proposed method of aggregation.
Example 1. Suppose that the expert E 1 evaluates the significance of one alternative with respect to another by the triangular fuzzy number then the set of expert assessments of the expert E 1 can be represented in the form of a fuzzy matrix of pairwise comparisons of the following type: are formed within the verbal scale that expresses the degree of superiority of one element over another, Table 1.
As a result, we will form a priority vector derived on the basis of assessments by the expert E 1 : W 1 =(0.42;0.36; 0.22; 0).
The expert E 2 , evaluating the significance of one alternative in relation to other crisp estimates, has formed the matrix of pairwise comparisons of the type the estimates b ij are the positive integers formed within the verbal scale [10]: the same significance is 1, the weak significance is 3, and so on, the absolute significance is 9; 2, 4, 6, and 8 are intermediate values of degrees of preference between each gradation.
To find the vector of local priorities, we use the geometric mean method [10]: As a result, we will form a priority vector derived on the basis of assessments by the expert E 2 : W 2 =(0.56; 0.27; 0.13; 0.04).
To get a group decision, we will use the operation of combining the expert assessments.
Conflict rate: Taking into account the rather high level of conflict, we will use the combination rule PCR5 (11) to aggregate expert assessments, which allows redistributing the conflicting basic masses of assertiveness to subsets involved in local conflicts [16,17].
The resulting subsets and the existing local conflicts are given in Table 2.

Table 2
The degree of intersection of subsections determined by the experts

∅
As can be seen from Table 2, there are 9 local conflicts in the model: The first local conflict is proportional to the choice between A 1 and A 2 according to the expression: As a result of the performed calculations, we obtain a vector of local priorities, the coefficients of which reflect the group opinion W=(0.572; 0.301; 0.123; 0.004).
Example 2. Assume that the expert E 1 evaluates the significance of one alternative in relation to another interval number: then the set of expert assessments by the expert E 1 can be represented in the form of an interval matrix of paired comparisons of the type The matrix C, in accordance with (10), is consistent.To find the vector of local priorities, we use the linear programming method GPM [15]: ( , , ), formed the fuzzy matrix of pairwise comparisons A  (example 1), on the basis of which, according to Chang's method [13], a vector of local priorities was obtained: W 2 =(0.42; 0.36; 0.22; 0).The Chang method helps obtain crisp frames of a fuzzy MPC represented by triangular fuzzy numbers, while the linear programming method GPM allows obtaining interval frames from an interval MPC.To combine the elements of the W 1 and W 2 vectors, we need to bring the values of the W 1 and W 2 vectors to one form -either pointwise or interval.
To obtain the crisp estimates w i ∈W 1 , we apply a pessimism coefficient: (1 ) , , ; As a result, we will form a priority vector derived on the basis of estimates of the expert E 1 , at γ=0.7, taking into account the valuation that W 1 =(0.46;0.22; 0.12; 0.2).
The values of the main masses of confidence are the following: Expert 1: m 1 (A To aggregate expert estimates, we apply the rule of combination PCR5 (11).The resulting major masses of confidence in accordance with the PCR5 rule: As a result of the calculations, we obtain a vector of local priorities the coefficients of which reflect the group opinion W=(0.51; 0.29; 0.13; 0.07).

Discussion of the results of studying the problem of structuring group expert judgments
The proposed technology of structuring expert information helps process expert judgments presented in various forms and synthesize collective ranking, taking into account various types of "ignorance" (contradictory, incomplete, obscurity, etc.) under the influence of which expert judgments are formed.The given numerical calculations demonstrate the effectiveness of the proposed method of aggregating expert evaluations under conditions of incomplete processing (an expert may refuse to evaluate an object) and contradictory (inconsistent) expert data.
To aggregate expert judgments obtained in the process of pairwise comparison, the widely used methods are of congestion, estimation of the average geometric or weighted meanings, and construction of a generalized matrix of pairwise comparisons.Unlike the existing methods of aggregating expert judgments, the proposed aggregation procedure does not depend on the form of presenting expert information, does not require additional information on the qualification (weight) of experts and allows processing conflicting (contradictory, inconsistent) expert judgments.Such advantages are achieved through the use of the combination mechanism for aggregating expert evidence on the basis of the mathematical apparatus of the theory of evidence and the theory of plausible and paradoxical reasoning.They make it possible to handle various forms of interaction of expert evidence (their union and intersection) and to take into account such factors as uncertainty, inaccuracy and incompleteness of expert information.Combining expert evidence on the basis of the rules of redistribution of conflicts, despite the complexity of mathematical calculations, give more effective combinations and help handle conflicting expert judgments.Effective results of the combination, when constructing aggregate estimates, can be obtained by establishing the optimal order for the combination of expert evidence, for example, taking into account the degree of dissimilarity and the structure of expert evidence.This, in turn, allows using the expert information received in full, without losing it when combining contradictory expert evidence.
Further research may be aimed at developing methods for improving the quality of the expert information received and studying the dynamics of the level of uncertainty in relation to the structure of expert evidence.

Conclusions
1.The method of aggregating group expert judgments is suggested to help synthesize group decisions, taking into account various forms of representing judgments of experts (interval, fuzzy and crisp expert evaluations).This approach allows modelling uncertainty in expert judgments due to the presentation of inaccuracies in expert estimates in the form of fuzzy and interval numbers.The expert independently chooses the form of presenting preferences in constructing matrices of pairwise comparisons and also can refuse the evaluation of certain objects of expertise, in which case truncated matrices of pairwise comparisons are constructed.The absence of a restriction on the form of presenting expert preferences gives the expert an opportunity to express an opinion (estimate) with respect to the analysed object as precisely as possible.This approach can increase the efficiency of the expert, which will improve the quality, reliability and consistency of expert information.
2. To aggregate individual expert judgments, it is proposed to use a combination mechanism based on one of the rules of the theory of evidence or the theory of plausible and paradoxical reasoning.It has been determined that more effective combined results are achieved when using rules for redistributing conflicts.In order to improve the quality of the aggregate results, it is proposed to determine the procedure for combining expert evidence (the values of the vector of local alternatives priorities), taking into account the degree of conflict between them and the structure of expert evidence.This allows for the full use of expert information and the elimination of situations when some of the expert information may be lost during the process of combining.For example, when trying to integrate non-coincident, contradictory expert evidence.
3. Examples of practical implementation of the proposed method of synthesizing a group decision are presented for conditions of uncertainty of different nature.The obtained practical results are intended to help increase the efficiency of the processes of preparing and making optimal decisions for analysing and structuring expert evaluations.Their application can significantly improve the quality of the received expert data by eliminating the restrictions on the form of submitting expert judgments and the need for evaluating each object of expertise.