A COMPARATIVE ANALYSIS OF THE ASSESSMENT RESULTS OF THE COMPETENCE OF TECHNICAL EXPERTS BY DIFFERENT METHODS

4 The well-known forms of collection of ideas are divided into individual, collective and mixed, each having a number of varieties. All are based on different algorithms of realization and have advantages and defects. In many cases, each of these varieties is used together, which gives quite often a greater effect and objectivity. However, mostly a questionnaire is used in practice of expert evaluation. It allows with the least effort to collect opinions of experts, though takes more time than some other kinds [1]. For reliable estimations of group expert evaluation in any field of activity, it is necessary foremost to choose the most optimal method. It is important from the large variety of expert methods to distinguish and to perfect the most prevailing for certain necessities in the certain field. To that end, it is necessary to carry out a comparative analysis for the evaluation of fitness of methods. Exactly the same methods were applied by the authors during the realization of researches on improving the efficiency of complex systems in the sphere of technical regulation, in particular in standardization and metrology [1].

The well-known forms of collection of ideas are divided into individual, collective and mixed, each having a number of varieties.All are based on different algorithms of realization and have advantages and defects.In many cases, each of these varieties is used together, which gives quite often a greater effect and objectivity.However, mostly a questionnaire is used in practice of expert evaluation.It allows with the least effort to collect opinions of experts, though takes more time than some other kinds [1].
For reliable estimations of group expert evaluation in any field of activity, it is necessary foremost to choose the most optimal method.It is important from the large variety of expert methods to distinguish and to perfect the most prevailing for certain necessities in the certain field.To that end, it is necessary to carry out a comparative analysis for the evaluation of fitness of methods.Exactly the same methods were applied by the authors during the realization of researches on improving the efficiency of complex systems in the sphere of technical regulation, in particular in standardization and metrology [1].

Introduction
For reasonable decision-making in any spheres of activity, it is necessary to lean against experience, knowledge and intuition of specialists.To that end, group expert evaluations -procedures of estimation of a certain problem based on the opinions of specialists (experts) for further decision-making are conducted.For such estimations, it is necessary foremost to correctly approach the selection of experts in the field of certain activity.
As practice shows, reduction of subjectivity, and, accordingly, increase of objectivity of the results of using the methods of group expert evaluation substantially depend on compliance with the rules of organization, preparation and realization of such evaluation.It depends, first of all, on the set procedures of expert evaluation, appointment of the responsible for the organization and realization of expert evaluation, and also formation of expert groups.

Literature review and problem statement
Introduction of the legislation of the European Union (EU) and compliance with the requirements of the World Trade Organization (WTO), and also realization of institutional changes in accordance with European practice is the basis of reforms in the field of technical regulation.For an identification and further resolution of problematic issues in this field, it is expedient to widely use the group expert evaluation.This needs the involvement of highly skilled specialists.
In the field of technical regulation, there are specialists in different areas, particularly standardization, metrology, conformity assessment, certification, etc. Evaluations of experts in the field of standardization were the subject of previous researches [1].Another not less challenging direction in the field of technical regulation is the metrology assurance (MA).
МA is defined as an establishment and application of metrology norms and rules, and also development, production and application of technical equipments necessary for the achievement of unity and required accuracy of a certain measurement.Important is reliable knowledge about the real condition of МA for measurement of a certain physical quantity.One of the useful means for solving this issue may be the group expert evaluation involving experts in metrology, i. e. highly skilled specialists-metrologists.
Expert methodologies of evaluation are based on the use of knowledge of skilled specialists -experts in the investigated subject field.For the estimation of experts, there are a number of characteristics used for the selection of both certain experts and groups of experts: coefficient of competence (CC); coefficient of concordance; coefficient of reliability of expert estimations, etc.
Most expert methods can be divided into classes of individual and collective (group) expert estimation.Each of the methods has a lot in common, and the difference is mainly that the evaluation of the investigated objects or spheres is carried out in different ways.
Resources and capabilities that can influence the choice of methods include: -competence, experience, capabilities and possibilities of the expert evaluation group; -limitation in relation to time and other resources of the organization; -available budget in case if external resources will be needed.
Most widely used expert methods are quite simple and have the imperfections: -the method of ranking -does not provide sufficient accuracy of ranking more than 15-20 objects [2][3][4][5]; -the method of direct evaluation -cannot be used in case of incomplete knowledge of an expert about the investigated properties of the object [2,3,6]; -the method of comparison, including two varieties: -the method of successive comparison [5] -the most labour-intensive and complex; -the method of pairwise comparison [5,[7][8][9] -quite simple in comparison with other methods, characterized by the highest level of reliability of the estimation results and allows investigating plenty of objects with great accuracy.
In [10], the method of evaluation of expert's competence on the basis of fuzzy set theory has been proposed.The disadvantages of this method are divergences between the finite set of competences that characterize the states of the object, and characteristics suggested by a certain expert.This narrows the application scope of this approach.
No less known methods of scenario analysis: analysis of the root cause, scenario, influence on activity, cause-andeffect relations do not provide for numeral estimates [6,11,12]; the basic method of analytic hierarchy process [13][14][15] and its modifications [5,7,12,16] are applicable only in case of a small number of the set of alternatives and do not give an opportunity to combine different opinions of expert groups.
For the research of complex objects or systems, it is expedient to use the method of Analytic Hierarchy Process (АНР).АНР is a mathematical tool for a systematic approach to complex problems of decision-making.It allows an interactive finding of such a variant (alternative) that best fits both with its understanding of the problem and with the requirements to its solution.
This method is widely used due to the works [13,14], which have more fully revealed the possibilities of the procedure, and since then АНР has been actively developed and widely used in practice.It allows a clear and rational way to structure a complex problem of decision-making as a hierarchy, comparing and performing the quantitative estimation of alternative solutions.Many works on the application of expert methods describe the usage of special software.The software considerably increases the productivity of the applied methods of expert evaluation and eliminate errors in the calculations of the results.

The aim and objectives of the study
The research was aimed at developing the most effective method and means for the competence evaluation of experts using scientifically-reasonable criteria.
For the achievement of the aim, the following objectives were accomplished: -to choose the most effective methods of expert evaluation for certain tasks in the field of technical regulation, which would let to take into account the level of competence of the experts involved; -to conduct the comparative analysis of the results obtained by means of the selected methods of competence evaluation of experts, and estimate the fitness.

1. A method of evaluation of expert's competence taking into account the data uncertainties
Being based on previous researches of the authors, questioning of metrology specialists for the evaluation of expert's competence (EEC), who have experience in these spheres and work on different types of measurement was carried out.Such types of measurement are covered: electrical power, electrical capacitance and inductance, phase shift angle, time and frequency, alternating-current (AC) voltage, high direct-current (DC) voltage, high AC voltage, high AC [17,18].
For the evaluation of expert's competence in the field of technical regulation, a special questionnaire was worked out with the aim of carrying out the appropriate questioning.
In the ideal case, the available data about the experts will be the average score of all criteria of the EEC.In other cases, such data can be an exhaustive list of the total scores of all criteria of the EEC.In any case, there will be some data uncertainties concerning the experts.The range of uncertainties in these data can be limited by the use of independent methods or by checking for consistency.
For the realization of the method with taking into account characteristics of data uncertainties (DU), appropriate criteria for the score EEC in a certain field are established [18].
For the realization of this method (DU method), it is necessary to calculate [1]: -average score i x for each of the M experts for all criteria of the EEC N ( j from 1 tо N): -relative average score ri x for each of the M experts: -rationed average score ni x for each of the M experts: -total standard uncertainties u ci for each of the M (i-th) experts: ( ) Calculation of the reference value x ref and its total standard uncertainty u ref for the overall evaluation of experts is carried out according to the formulas: The consistency check of the data about the experts is performed on the basis of analysis and calculation of the value of χ 2 : If the value of the χ 2 criterion, calculated from data of the i-th expert, does not exceed a critical value χ 2 for the 0.95 confidence interval and the number of freedom degrees of M-1: ) , then the data on the formed group from M experts (minimum of 4 experts, maximum -up to 33) are recognized as agreed.
The remaining experts are ranked in descending order .ni x The selection of the most competent experts is expedi-ent to carry out by the analysis of the results obtained with application of the Pareto principle [19].
The algorithm of evaluation of expert's competence by the DU method is given in [1].This algorithm can be easily implemented with the use of common mathematical software packages.
The DU method is expedient to apply as a useful tool for the comparative EEC on the basis of objective data according to the set of criteria of the EEC for different spheres of activity.It allows selecting the most competent experts and rejecting (if necessary) the experts, whose data do not correspond to a certain level of the set requirements.

2. A method of evaluation of expert's competence on the basis of Analytic Hierarchy Process
АНР is well-known [13] and used in case of a small number of the set of alternatives, when the decision-maker's efforts (DM) are aimed at comparing only the set of alternatives.The task for the АНР method consists in that, with the well-known general aim (or subtask) of the problem solution, N criteria of estimation of alternatives and п alternatives, it is necessary to choose the best alternative.АНР is a mathematical tool of systems approach to complex decision-making problems.
АНР is intended for the determination of the best option (among several) taking into account many criteria of different nature and allows using various criteria for comparison (quantitative, qualitative, numerical with different dimensions, etc.).Its maximum efficiency shows up during the search of solutions to complex problems that require systems approach and involvement of plenty of experts [1].
Basic phases for implementation of the АНР method are: -to structure a task as a hierarchical structure with several levels (aims-criteria-alternatives); -to execute pairwise comparisons of elements of every level and transform the results of comparisons in numbers by means of the special scale of relative importance (SRI); -to calculate the weight coefficients (w i is the weight of the i-th of criterion) for the elements of every level and check up the consistency of the DM's judgements; -to compute the quantitative indicator of the quality of each of the alternatives and determine the best alternative.
SRI is a well-organized set of gradations that are determined numerically, for expression of the results of pairwise comparisons [13][14][15].This scale allows determining the relative importance of the compared elements exactly, so its division is dimensionless quantities.
The synthesis of the obtained weight coefficients is carried out by the expression: where S j is the index of the quality of the j-th alternative; w i is the weight of the i-th criterion of the EEC; V ji is the weight of the j-th of alternative on the i-th criterion of the EEC.
There may be errors in the compilation of pairwise comparison matrices (PCM).So one of the possible errors may be a violation of transitivity: a ij >a is (a ij -elements of the PCM) may not follow from a ij >a jk , a jk >a is .Violations of consistency of numeral judgements are also possible: a ij ×a jk ¹a jk .
For the detection of inconsistency, the calculation of the index of consistency of comparisons according to the PCM is carried out in the following sequence [13,14]: -to sum up the elements of every column in the PCM; -to multiply the sum of the elements of every column by the corresponding normalized components of the weight vector determined from the PCM; -to sum up the resulting numbers and denote the values of the sum as l max ; -to find the consistency index L=(l max -n)/(n-l), where n is the number of the compared elements (matrix size), l³n for a skew-symmetric matrix; -to calculate the mean value of the consistency index R for skew-symmetric matrices filled randomly (for n=7,•R=l.32,аnd for n=8,•R=l.41); -to calculate the consistency ratio C d =L/R, which must be within C d ≤0.1, and if the value of Т exceeds this level, the comparison is carried out again.
The consistency ratio C d for the entire hierarchy less than 0.1 is a good result.
Pairwise comparisons set priorities of elements for some level of hierarchy in relation to one element of the next level.If there are more than two levels, then different vectors of priorities can be united in the matrix of priorities, from which one eventual vector of priorities for the lower level is determined.
For the realization of the АНР method [1], the PCM of indexes A -a table of numeral values of the results of the pairwise comparisons obtained by the SRI is built: where N is the number of the generalized indexes.The numeral values of the PCM elements A a ij (i, j= =1, 2, …, N) are set by an expert way or group of experts (by an expert), or directly in each specific estimation of the compared objects.The relative importance of the index i compared to the index j is expressed by a natural number from 1 to 9 or reverse number from 1 to 1/9.For i=j, a ij =1, and for i≠j, a ij >0, a ij =1/a ij .The normalized eigenvector A i for the PCM of indicators A is determined by the expression: The consistency of local priorities, i. e. the quality of the output data included in the PCM A is checked, for which the consistency ratio C d is determined by the expression: where I c is the consistency index of the output data; R c is the casual consistency index.The consistency ratio for the system as a whole characterizes its weighted mean value for all the comparison matrices.
The consistency index I c of the output data included in the A matrix is determined by the formula: max , 1 where λ max is the largest eigenvalue for the A matrix, which, in turn, is determined by the formula: The consistency index I c is a quantitative index (positive number) of noncontradiction of the comparison results (contradictions in comparisons arise due to subjective errors in the output data).The consistency index that is randomly generated on a scale from 1 to 9 of the back-symmetric matrix with the corresponding reciprocals of the elements is the casual consistency index R c .
The value of the random consistency index of the output data depends on the dimension of the PCM A (L=N -for the A matrix) used and is given in [13,14,20].If C d ≤0.1, then the initial information is considered consistent, and if C d >0.1, then the output data are considered inadmissibly distorted.In the second case, it is necessary to revise the output data with the introduction of additional or corrective information.
Next, the PCM B k (k=1, 2, ..., M) is built for each of the compared by each generalized index: where The normalized eigenvectors B k i for each built PCM B k , which determine local priorities are defined by the formula: The index k as a natural number is used to indicate the number of the indicator, to which the value with this index belongs.
The consistency of local priorities, i.e. the quality of the output data, included in the PCM B k is checked, for which consistency ratio k d C is determined by the formula: where k c I is the consistency index of the output data included in a specific PCM ; B k R c is the random consistency index, which depends on the dimension of the matrix used (L=M -for the PCM B k ).
The consistency index of the output data included in the PCM B k is determined by the formula: max , 1 where max l k are the largest eigenvalues for the PCM B k determined by the formula: then the initial data is considered consistent, and if 0.1 -inadmissibly distorted.In the second case, it is also necessary to revise the output data with the introduction of additional or corrective information.
In the case k=1, a comparison of one expert with competence indexes, i. e. checking for accordance to the set of criteria of the EEC is carried out.Next, the global priorities G n for each of the M compared experts determined by the formula: where 0 ,..., i i n B B are the components of the normalized eigenvectors of local priorities determined by using the expression (16).
The obtained values of global priorities for each expert are ranked in ascending order of the G n values.The expert who received the maximum value of G n is considered the most competent [20].

Evaluation of competence of technical experts by the methods of data uncertainties and Analytic Hierarchy Process
For the realization of the marked and described DU and АНР methods, the following criteria of the EEC for metrologists and standardizers are applied [1,20,21]: K 1 -education and scientific level in the field of metrology (standardization); K 2 -overall experience; K 3 -experience in the field of technical regulation (metrology, standardization); K 4 -experience as expert in the field of technical regulation (metrology, standardization); For the realization of the АНР method, the values of the PCM of A criteria with normalized eigenvectors K ivectors of priorities for the selected criteria of the EEC and weight coefficients for the selected criteria of the EEC are defined [20].
For the proposed criteria of the EEC, the largest eigenvalue of the PCM of A criteria was λ max =5.35.Verification of consistency of the output data used for the construction of the A matrix on the obtained consistency index I c =0.09 and consistency ratio C d =0.07 showed that the consistency ratio satisfied the requirements of consistency (C d ≤0.1).This demonstrates the consistency of the set criteria of the EEC [1,20].
For comparison of the results, obtained with the use of DU and АНР methods, for the АНР method, recalculation of the received global priorities for the experts in the CC (k AHP ) was carried out by the formula: In the comparative analysis of the results obtained by the DU and АНР methods, vari-ation (dispersion) of the CC for the experts was calculated by the formula: where k max is the maximum of CC (equals 1.00); k min is the minimum of CC, obtained for a certain expert.

An example of competence evaluation of time and frequency measurement expert's
Competence evaluation of 21 time and frequency measurement expert's was made.Among 21 involved experts, 16 (76 %) represented the state enterprises of the technical regulation system, 5 (24 %) -other enterprises.The questionnaire data about the experts was processed by specialized software "Competence DU 1.1" and "Competence АНР 1.1" (Ukraine) intended exactly for the EEC and based on the above methods [20].
The values of the CC used for all time and frequency measurement experts are shown in Table 1 [20].The CCs were obtained by the use of DU (k DU ) and АНР (k AHP ) methods described in [18].Comparison of CC of time and frequency measurement experts obtained by the use of DU and АНР methods was carried out and the results are shown in Fig. 1.
As can be seen from Fig. 1, comparisons of CC of the experts obtained by the use of DU and АНР methods have a clear correlation.Thus, the АНР method is characterized by a greater dispersion of the CC (k AHP ) values: the difference between the highest and the lowest CC is R AHP =0.70 (minimum CC -0.30).For the DU method, the dispersion of the CC (k DU ) values is R DU =0.42 (minimum CC -0.58), that is by 40 % less than for the АНР method.

Discussion of the results of evaluation of competence of technical experts
The results of the evaluation of CC of technical experts by the use of DU and АНР methods are shown in Table 2. Within the framework of the conducted questioning, there was also a questioning about the experience in the field of technical regulation, necessary for an achievement of the expert level.For all types of activity, an average value is 7, and the mode -5.The results for the DU and АНР methods show a large correlation.
With the application of the universal software Microsoft Excel (USA), the ratio of averages for the criteria used for the EEC in the field of technical regulation was estimated (Fig. 2).The results show a small dispersion of average values for the criteria of the EEC (from 5.2 to 7.9), which testifies to a good balance.Average values of CC of technical experts were obtained using the DU method (the average for all fields is 0.81, the maximum for all fields is 0.88, the minimum for all fields is 0.76).Average maximum eigenvalues for technical ex-perts, were obtained using the АНР method (the average for all fields is 17.6, the maximum for all fields is 32.4, the minimum for all fields is 11.1).
A general comparison of CC of metrology and standardization experts, obtained using the DU and АНР methods, shows a clear correlation between the values on the experts.Thus, the АНР method is characterized by a greater dispersion of the CC (R AHP ) values, and for the DU method, the dispersion of the CC (R DU ) values is less than for the АНР method.The dispersion of the values obtained using the DU and АНР methods is shown in Table 3. Dispersion of the values of CC for experts in the field of technical regulation, obtained using: -the DU method: the average for all fields is 0.46; the maximum for all fields is 0.60; the minimum for all fields is 0.36; -the АНР method: the average for all fields is 0.80; the maximum for all fields is 0.88; the minimum for all fields is 0.67.The results in Table 3 show a large correlation for the DU and АНР methods.Therefore, these methods are expedient to apply as a useful tool for the comparative evaluation of expert's competence on the basis of objective data according to the set of criteria for different fields (spheres) of activity.However, the АНР method, to a lesser extent, allows taking into account opinions of less competent experts than the DU method.This is evidenced by a lower CC for this method than for the DU method.Thus, the DU method allows the selection of the most competent experts and discard the experts the data on whom disagree with a certain level of the set requirements.

Conclusions
1.The most effective methods of expert evaluation, suitable for the evaluation of expert's competence in the field of technical regulation, are considered in detail.The DU and АНР methods as the most suitable are selected.Using the selected methods, the evaluation of expert's competence in the field of technical regulation is conducted (time and frequency measurement) according to the set criteria.The results were processed by special methodologies, and also universal and specialized software "Competence DU 1.1" and "Competence АНР 1.1" (Ukraine).
2. The comparative analysis of the results obtained using the DU and АНР methods showed the convergence, fitness and correlation of the obtained values for the experts.The estimated ratio of the averages for the criteria used for the EEC in the field of technical regulation, showed a small dispersion of the average values for the criteria of the EEC (from 5.2 to 7.9), which testifies to a good balance.
However, the АНР method, to a lesser extent, allows taking into account opinions of less competent experts, than the DU method.This is evidenced by a lower CC for this method than for the DU method.Therefore, these methods are expedient to apply as a useful tool for the comparative evaluation of expert's competence on the basis of objective data according to the set of criteria for different fields (spheres) of activity.Thus, the DU method allows the selection of the most competent experts and discard the experts the data on whom disagree with a certain level of the set requirements.

ijb
is the result of pairwise comparison of the i-th and j-th object relatively the k-th index; M is the number of the compared objects.Numeral values of the elements of the PCM B k -k ij b (i, j=1, 2, ..., M) are obtained with the use of available information from those or other accessible sources.If i=j, b ij =1,

Fig. 1 .
Fig. 1.Comparison of competence coefficients of time and frequency measurement expert's obtained using the methods of data uncertainties and Analytic Hierarchy Process

Fig. 2 .
Fig. 2. The average for the criteria of evaluation of expert's competence in the field of technical regulation

Table 1
Coefficients of competence of time and frequency measurement experts obtained using the methods of data uncertainties and Analytic Hierarchy Process

Table 2
The results obtained using the methods of data uncertainties and Analytic Hierarchy Process

Table 3
The dispersion of the values obtained using the methods of data uncertainties and Analytic Hierarchy Process