DOI: https://doi.org/10.15587/1729-4061.2019.163964

Development of the method of distances for processing expert estimates in information systems

Tetyana Otradska, Natalia Shybaieva, Oleksandr Shyrshkov, Mykola Rudnichenko, Ashot Halustian

Abstract


Current research focuses on expert estimation of indicators by employing a limited number of qualified experts, which involves considerable time and cost. The advent of modern information technology allows rapid and efficient assessment of indicators that characterize performance of enterprises. The main tool for solving the issue related to the credibility of such an estimation is the development of new methods for processing its results.

We have proposed a method of distances to treat the results from expert estimations of indicators and examined the results of its work.

To this end, theoretical substantiation of the method has been performed, based on the concept of proximity (distances) among estimates relative to the average value, coefficients of experts' competence, and normalization of the point scale of assessment.

We have explored three variants of expert point- and verbal-based estimates of independent and dependent indicators in line with the method of distances, which showed that the convergence rate of iteration process for all cases varies from 1 to 4 orders of magnitude at each iteration. That makes it possible to draw a conclusion about a very small number of computations by the information system and sufficient speed of processing expert estimates.

A comparative analysis of the method of distances with a similar method of square deviations has revealed almost the same rate of convergence up to 3‒5 iterative steps, but the proposed method yields an estimate close to the average estimate of each indicator. It also enables the processing of results in information systems by 99 % faster, from dozens, even hundreds of times, more estimates than that in the methods of expert selection, which is important under current competitive environment. Average efficiency of the method compared to the method of expert selection is 5.8 %

Keywords


expert estimates; estimation of indicators; on-line assessment; processing of expert estimate; analysis of expert estimation; information technology

References


Blikhar, T. (2013). Monitorynh navchalnykh dosiahnen u vyshchykh navchalnykh zakladakh: pidkhody, zdobutky. Monitorynh yakosti vyshchoi osvity: dosvid roboty, innovatsiyi, problemy: materialy I Vseukrainskoi naukovo-praktychnoi internet-konferentsiyi. Irpin: NU DPSU, 129–130.

Odainyk, S. (2010). Monitorynh yakosti pedahohichnoi osvity: teoretychnyi aspekt. Molod i rynok, 9, 64–67.

Zakharova, A. A., Grigorjeva, A. A., Tseplit, A. P., Ozgogov, E. V. (2016). Models Used to Select Strategic Planning Experts for High Technology Productions. IOP Conference Series: Materials Science and Engineering, 127, 012029. doi: https://doi.org/10.1088/1757-899x/127/1/012029

Liljelind, I. E., Rappaport, S., Levin, J., Pettersson Strömbäck, A. E., Sunesson, A.-L. K., Järvholm, B. G. (2001). Comparison of self-assessment and expert assessment of occupational exposure to chemicals. Scandinavian Journal of Work, Environment & Health, 27 (5), 311–317. doi: https://doi.org/10.5271/sjweh.619

Wieck, M. M., McLaughlin, C., Chang, T. P., Rake, A., Park, C., Lane, C. et. al. (2018). Self-assessment of team performance using T-NOTECHS in simulated pediatric trauma resuscitation is not consistent with expert assessment. The American Journal of Surgery, 216 (3), 630–635. doi: https://doi.org/10.1016/j.amjsurg.2018.01.010

Zakharova, A. A., Ostanin, V. V. (2015). Formalization model of expert knowledge about a technical index level of engineering products. IOP Conference Series: Materials Science and Engineering, 91, 012070. doi: https://doi.org/10.1088/1757-899x/91/1/012070

Dugarova, D. T., Starostina, S. E., Cherepanova, L. V. (2016). Expert Training as a Factor of Independent Assessment of Quality of Social Services. Scholarly Notes of Transbaikal State University. Series Philosophy. Cultural Studies. Sociology. Social Work, 11 (2), 170–179.

Cooke, R. M., Goossens, L. L. H. J. (2008). TU Delft expert judgment data base. Reliability Engineering & System Safety, 93 (5), 657–674. doi: https://doi.org/10.1016/j.ress.2007.03.005

Mach, K. J., Mastrandrea, M. D., Freeman, P. T., Field, C. B. (2017). Unleashing expert judgment in assessment. Global Environmental Change, 44, 1–14. doi: https://doi.org/10.1016/j.gloenvcha.2017.02.005

Kozierkiewicz-Hetmańska, A. (2017). The analysis of expert opinions’ consensus quality. Information Fusion, 34, 80–86. doi: https://doi.org/10.1016/j.inffus.2016.06.005

Christoforaki, M., Ipeirotis, P. G. (2015). A system for scalable and reliable technical-skill testing in online labor markets. Computer Networks, 90, 110–120. doi: https://doi.org/10.1016/j.comnet.2015.05.020

Cain, M., McLeay, S. (2016). Statistical Auditing of Non-transparent Expert Assessments. Sankhya B, 78 (2), 362–385. doi: https://doi.org/10.1007/s13571-016-0124-8

Ageev, A. S. (2011). Technique of carrying out of expert estimations of activity of aviation enterprise on safety of flights. Nauchniy vestnik Moskovskogo gosudarstvennogo tekhnicheskogo universiteta grazhdanskoy aviacii, 174, 69–72.

Kolpakova, T. A. (2011). Determination of experts competence in group decision-making. Radioelektronika, informatika, upravlenie, 1, 40–43.

Roberts, T. S. (2004). A ham sandwich is better than nothing: Some thoughts about transitivity. Australian Senior Mathematics Journal, 18 (2), 60–64.

Podd'yakov, A. N. (2006). Intransitive character of superiority relations and decision-making. Psihologiya. Zhurnal Vysshey shkoly ekonomiki, 3 (3), 88–111.

Otradskaya, T., Gogunskii, V., Antoshchuk, S., Kolesnikov, O. (2016). Development of parametric model of prediction and evaluation of the quality level of educational institutions. Eastern-European Journal of Enterprise Technologies, 5 (3 (83)), 12–21. doi: https://doi.org/10.15587/1729-4061.2016.80790

Gül, E., Çokluk, Ö., Gül, Ç. D. (2015). Development of an Attitudes Scale toward Online Assessment. Procedia – Social and Behavioral Sciences, 174, 529–536. doi: https://doi.org/10.1016/j.sbspro.2015.01.699

Formanek, M., Wenger, M. C., Buxner, S. R., Impey, C. D., Sonam, T. (2017). Insights about large-scale online peer assessment from an analysis of an astronomy MOOC. Computers & Education, 113, 243–262. doi: https://doi.org/10.1016/j.compedu.2017.05.019


GOST Style Citations


Blikhar T. Monitorynh navchalnykh dosiahnen u vyshchykh navchalnykh zakladakh: pidkhody, zdobutky // Monitorynh yakosti vyshchoi osvity: dosvid roboty, innovatsiyi, problemy: materialy I Vseukrainskoi naukovo-praktychnoi internet-konferentsiyi. Irpin: NU DPSU, 2013. P. 129–130.

Odainyk S. Monitorynh yakosti pedahohichnoi osvity: teoretychnyi aspekt // Molod i rynok. 2010. Issue 9. P. 64–67.

Models Used to Select Strategic Planning Experts for High Technology Productions / Zakharova A. A., Grigorjeva A. A., Tseplit A. P., Ozgogov E. V. // IOP Conference Series: Materials Science and Engineering. 2016. Vol. 127. P. 012029. doi: https://doi.org/10.1088/1757-899x/127/1/012029 

Comparison of self-assessment and expert assessment of occupational exposure to chemicals / Liljelind I. E., Rappaport S., Levin J., Pettersson Strömbäck A. E., Sunesson A.-L. K., Järvholm B. G. // Scandinavian Journal of Work, Environment & Health. 2001. Vol. 27, Issue 5. P. 311–317. doi: https://doi.org/10.5271/sjweh.619 

Self-assessment of team performance using T-NOTECHS in simulated pediatric trauma resuscitation is not consistent with expert assessment / Wieck M. M., McLaughlin C., Chang T. P., Rake A., Park C., Lane C. et. al. // The American Journal of Surgery. 2018. Vol. 216, Issue 3. P. 630–635. doi: https://doi.org/10.1016/j.amjsurg.2018.01.010 

Zakharova A. A., Ostanin V. V. Formalization model of expert knowledge about a technical index level of engineering products // IOP Conference Series: Materials Science and Engineering. 2015. Vol. 91. P. 012070. doi: https://doi.org/10.1088/1757-899x/91/1/012070 

Dugarova D. T., Starostina S. E., Cherepanova L. V. Expert Training as a Factor of Independent Assessment of Quality of Social Services // Scholarly Notes of Transbaikal State University. Series Philosophy. Cultural Studies. Sociology. Social Work. 2016. Vol. 11, Issue 2. P. 170–179.

Cooke R. M., Goossens L. L. H. J. TU Delft expert judgment data base // Reliability Engineering & System Safety. 2008. Vol. 93, Issue 5. P. 657–674. doi: https://doi.org/10.1016/j.ress.2007.03.005 

Unleashing expert judgment in assessment / Mach K. J., Mastrandrea M. D., Freeman P. T., Field C. B. // Global Environmental Change. 2017. Vol. 44. P. 1–14. doi: https://doi.org/10.1016/j.gloenvcha.2017.02.005 

Kozierkiewicz-Hetmańska A. The analysis of expert opinions’ consensus quality // Information Fusion. 2017. Vol. 34. P. 80–86. doi: https://doi.org/10.1016/j.inffus.2016.06.005 

Christoforaki M., Ipeirotis P. G. A system for scalable and reliable technical-skill testing in online labor markets // Computer Networks. 2015. Vol. 90. P. 110–120. doi: https://doi.org/10.1016/j.comnet.2015.05.020 

Cain M., McLeay S. Statistical Auditing of Non-transparent Expert Assessments // Sankhya B. 2016. Vol. 78, Issue 2. P. 362–385. doi: https://doi.org/10.1007/s13571-016-0124-8 

Ageev A. S. Technique of carrying out of expert estimations of activity of aviation enterprise on safety of flights // Nauchniy vestnik Moskovskogo gosudarstvennogo tekhnicheskogo universiteta grazhdanskoy aviacii. 2011. Issue 174. P. 69–72.

Kolpakova T. A. Determination of experts competence in group decision-making // Radioelektronika, informatika, upravlenie. 2011. Issue 1. P. 40–43.

Roberts T. S. A ham sandwich is better than nothing: Some thoughts about transitivity // Australian Senior Mathematics Journal. 2004. Issue 18 (2). Р. 60–64.

Podd'yakov A. N. Intransitive character of superiority relations and decision-making // Psihologiya. Zhurnal Vysshey shkoly ekonomiki. 2006. Vol. 3, Issue 3. P. 88–111.

Development of parametric model of prediction and evaluation of the quality level of educational institutions / Otradskaya T., Gogunskii V., Antoshchuk S., Kolesnikov O. // Eastern-European Journal of Enterprise Technologies. 2016. Vol. 5, Issue 3 (83). P. 12–21. doi: https://doi.org/10.15587/1729-4061.2016.80790 

Gül E., Çokluk Ö., Gül Ç. D. Development of an Attitudes Scale toward Online Assessment // Procedia – Social and Behavioral Sciences. 2015. Vol. 174. P. 529–536. doi: https://doi.org/10.1016/j.sbspro.2015.01.699 

Insights about large-scale online peer assessment from an analysis of an astronomy MOOC / Formanek M., Wenger M. C., Buxner S. R., Impey C. D., Sonam T. // Computers & Education. 2017. Vol. 113. P. 243–262. doi: https://doi.org/10.1016/j.compedu.2017.05.019 







Copyright (c) 2019 Tetyana Otradska, Natalia Shybaieva, Oleksandr Shyrshkov, Mykola Rudnichenko, Ashot Halustian

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.

ISSN (print) 1729-3774, ISSN (on-line) 1729-4061