DEVELOPMENT OF COMPREHENSIVE DECISION SUPPORT TOOLS IN DISTANCE LEARNING QUALITY MANAGEMENT PROCESSES

The object of this study is the quality of dis­ learning. The need for procedures to assess the quality of this form of education was manifes­ ted most clearly in connection with the COVID­19 pandemic, wars, and other global problems, which predetermine the relevance of the study. The study considers the construction of a deci­ sion support model for assessing the quality of dis­ tance learning. Underlying the method is a com­ bination of the method of expert assessments and the criterion model of data analysis, the basic method for analyzing the data obtained is the method of hierarchy analysis. and of the lity and on the the of indi­ cators within the criterion. Due to the combination of a criteria­based approach to data analysis in combination with the method of expert assessments, the model can be easily adapted for a point assessment of indi­ vidual components and finding problem areas in the implementation of distance learning and management decision­making. The results of the study reported here may be of interest to both heads of educational institu­ tions and employees of services involved in pro­ cessing information about the organization and reporting for strategic decision­making


Introduction
Higher education systems in today's world face quality assurance challenges. Quality is not a static state and needs to be regularly evaluated and improved. Quality assurance is in most cases encouraged by ongoing self-assessment and the search for some form of external verification.
Modern realities -the COVID-19 pandemic and possible crises -dictate a change in distance education from classical teaching towards a combined or even unpredictable transition from one form of education to another. Issues of the quality of education, and especially the quality of the provision and organization of the educational process with distance learning technology, are becoming increasingly important throughout the modern world, and therefore research into the area of the effectiveness of the organization of the distance learning process is relevant.
In today's globalized world, it is a fact that economic growth is determined not so much by the growth of physical capital and labor but by the stock of knowledge and the rate of its growth. At the same time, the latest information and communication technologies have made it possible to create knowledge societies in which higher education institutions (universities) play a key role. According to UNESCO, the development of the higher education sector should become a priority in both the most developed and developing countries, which is spelled out in the recommendations [1,2]. Therefore, research on the development of decision support models for assessing quality in education is relevant.
The study of quality management of distance learning should take into consideration the monitoring, control, and analysis of the effectiveness of the organization of learning processes from the standpoint of integrity and consistency. This will give the quality management system of distance learning new opportunities for realizing the educational potential of universities. The devised methods and models for analyzing the quality of distance learning are able to form an objective assessment of the distance learning system of the university in an automated mode.
The development of an automated system capable of assessing the quality of services provided and identifying problem areas is impossible without building a mathematical model of decision support. The introduction of this model in the educational process will contribute to improving the quality of distance learning.

Literature review and problem statement
Information plays an important role in the development of the organization as a commodity and as a resource. All processes, without exception, depend on information when making management decisions and developing a strategy. Competent organization of interaction within the information space of the university will simplify the work and organize interaction at the operational, tactical, and strategic levels. One of the directions of such research may be associated with the creation of an automated system for assessing the quality of distance learning. Such a system can be considered an important component of the general information and analytical system of the university [3], which makes it possible to make decisions aimed at improving the quality of education. Any quality control system should have criteria against which it will be evaluated. The criteria and guidelines for quality assurance in the European Higher Education Area (ESG) were adopted as early as 2005, but even now the quality management of education and its evaluation is a problem that no educational institution can neglect, which, in turn, requires the development of appropriate tools. At the same time, the concepts of the quality of educational processes and systems remain very diverse and problematic [4]. This fully applies to distance learning. The authors of [5] note that the assessment of the quality of university education meets two main goals: 1) identification of weaknesses and strengths of the educational process at the university, which will make it possible to develop and apply strategies aimed at improving quality; 2) determining the compliance of the achieved level of quality of educational activities in the institution with the established standards.
The authors of [6] consider the features of the processes of external quality assurance in the system of higher education management. It is noted that the quality of educational programs at higher education institutions is controlled by quality assurance agencies using the accreditation system from the point of view of expert assessment and highlights the problem of managing the accreditation process, namely the subjectivity and inconsistency of expert decisions. However, no specific methodology was proposed.
The ability to acquire operational and reliable data makes it possible to take operational and predictive decisions to improve the quality of education at the university. The authors of [7] substantiate the increase in the efficiency of this process with the construction of the information environment of the university.
Noting that distance (e-learning) is not just another way to implement traditional educational services, but a completely new approach to education, the authors of [8] point out that the quality assessment system should take this fact into consideration. Quality assessment criteria should take into consideration the unique characteristics of distance learning that distinguish it from traditional education. It is no coincidence that the analysis of scientific literature shows a growing interest in the problem of the quality of distance (e-learning).
The analytical study [9] summarizes new approaches to assessing the quality of higher education, which take into consideration the interests of various stakeholder groups (teachers, students, university administration) and are aimed at establishing a balance of accountability and improving processes. This makes it possible to form an assessment of educational programs that is fairer and more trustworthy for stakeholders. However, the authors ignored the issue of involving such an important group of stakeholders as students in the expert evaluation. This information gap is to some extent filled by paper [10], which gives a wide range of functions (from advisory to full-fledged experts) and describes the role of students in external quality assurance processes. It should be noted that models for the formation of an expert group, taking into consideration the specificity of the academic experience of students, have not yet been developed.
The authors of the recommendations from the International Council for Open and Distance Education (ICDE) [11] indicate that the quality assurance system of distance learning, first of all, should guarantee the predefined level of quality, and not serve as a tool for promoting innovation.
Evaluation of such a complex phenomenon as the quality of distance education and its management requires taking into consideration many different heterogeneous indicators with varying degrees of expression in the overall structure of the assessment. According to the authors of works [12,13], one of the most effective methods for determining the relative weight of each indicator and building a balanced system is the method of analyzing hierarchies. Establishing the weight of the elements and building a hierarchy on this basis is carried out using the Saati pairwise comparison scale [14].
In works [15,16] it is reported that managing the quality of distance education and predicting its results is an important and complex task. In their study, the authors showed that the quality of the courses offered, and the organization of the learning process depend on the number of successfully completed courses of students.
As follows from our review, the quality of distance learning must be constantly monitored, taking into consideration the peculiarities of this form of education. All this suggests that it is appropriate to conduct a study on the development of a decision support model to assess the quality of distance learning.

The aim and objectives of the study
The purpose of this study is to develop a comprehensive toolkit for the formation and adoption of a collegial management decision by an expert group to assess the quality of distance learning. The proposed decision-making tools will identify problem areas to which the management influence of the university administration will be applied.
To accomplish the aim, the following tasks have been set: -to develop and describe structural and functional schemes for managing the quality of distance learning; -to analyze the organizational features of distance learning and develop a database of criteria and indicators to assess the quality of distance learning; -to conduct an expert assessment of the criteria and indicators of the distance learning quality assessment system and develop a scheme of the criteria-based decision support model for assessing the quality of distance learning.

The study materials and methods
The object of this study is the quality of distance learning. The hypothesis of the study assumes that the joint application of methods of hierarchy analysis, expert assessments, and the criteria approach will make it possible to develop an effective model capable of identifying problem areas and recommending a control effect in the implementation of distance learning.
For the study, we adopted the following assumptions and simplifications. When implementing the method of hierarchy analysis, the arithmetic mean of expert assessments is used since with a large number of experts, the probability of the influence of the human factor is leveled. The method of hierarchy analysis itself can be applied if there is a rank coefficient for each expert but this can be important only with a small number of experts.
The methods used in the research process corresponded to the stages of designing and building a decision support model to assess the quality of distance learning.
At the first stage, the methods of analysis and synthesis were used to build a structural and functional scheme of the quality management system for distance learning. It defines the components of the system and their interaction.
At the second stage, the method of theoretical and methodological analysis was applied. The analysis of sources made it possible to select the most important criteria that are paid attention to in scientific papers from the Scopus, WoS databases, and conference proceedings that addressed distance learning. As a result of the review of 97 sources, criteria and indicators for assessing the quality of distance learning were identified, which are reported by different authors. Systematization of the results of the analysis of scientific sources, taking into consideration the purpose and key idea of the current study, was carried out by us through discursive reflection. That has made it possible to reduce the list of criteria and indicators detailing them.
At the third stage, the method of expert evaluation was implemented. Matrices have been developed for the analysis of hierarchies. Questionnaire sheets for experts have been prepared. A mathematical model has been built. The survey was conducted using the Likert scale: from 1 «the indicator is not important at all», 2 «the indicator is of insignificant importance», 3 «the indicator is of average importance», 4 «the indicator is important», to 5 «the indicator is very important». Subsequently, for ranking indicators, the method of hierarchy analysis proposed in [14] was used, with the help of which the weights of the criterion indicators were defined. Based on the data obtained, the importance of each indicator for the construction of the criterion model was assessed.
The fourth stage was a briefing and a survey of experts; the results of the survey were recorded in the prepared matrices; an analysis of the results was carried out and priorities were highlighted. The modeling method made it possible to build a mathematical model of decision support for assessing the quality of distance learning based on the data obtained.
The model was tested at the Non-Profit Joint Stock Company «North Kazakhstan University named after M. Kozybayev» (Republic of Kazakhstan).

Results of the development of a decision support model
for assessing the quality of distance learning

1. Building a functional scheme of the distance learning quality management system
To build a functional scheme of the distance learning quality management system, it is necessary to determine the place of the automated information system in the distance learning management system. To do this, it is necessary to build a structural scheme for managing the quality of distance learning. The proposed structural scheme of quality management of distance learning is shown in Fig. 1.
The implementation of distance learning through information systems in universities is strongly influenced by the external environment. It includes requirements for distance learning standards, educational programs, teacher qualifications, etc.
The information system of distance learning is information tools, software, the way a distance learning system is implemented at the university. A quality assessment system is a system capable of assessing quality relative to pre-entered criteria and parameters.
The management system is representatives of the university management: rector, vice-rector, dean, etc. Based on the data received from the automated system, a decision is made on the control effect. For example: improve the skills of trainers, change training materials, or other activity, depending on the data obtained.
Below, in Fig. 2, the proposed functional scheme of the quality management system of distance learning is shown.
The parameters by which the quality of distance learning will be assessed are entered into the database. In the parameter analysis block, they are processed, and certain values are assigned to the indicators. When comparing parameters, inconsistencies are revealed. For example, the parameter «Speed of veri fication of the student's work (DE2)» is assigned a value of 4 by experts on the Likert scale when assessing «as it should», and when assessing «as is» the value of 1. This indicates that the work of students is checked very slowly, or partially not checked. In this case, the university employee responsible for monitoring the inspection of work should strengthen control and regularly report to management about how quickly the check began to be performed. If there is a change in the requirements for the implementation of distance learning, then an adjustment is made, and new quality parameters are entered into the database.

2. Results of the development of a database of criteria for assessing the quality of distance learning
The purpose of this stage is to develop a database for assessing the quality of distance learning. To do this, it was necessary to exclude overlaps in the criteria and indicators used by different authors to evaluate distance learning. In this regard, 18 criteria (categories) were identified, which were detailed by 147 indicators. As a result, 10 criteria were defined. The number of indicators for each criterion ranged from 3 to 9 indicators.
In total, 52 indicators were obtained, which are presented below: 1. Regulatory documents on educational programs implemented with the use of distance educational technologies (DOT) (RD). Criterion parameters. General requirements for the provision of distance learning (RD1). Requirements for educational programs implemented using distance learning (RD2). Technical requirements for the provision of distance learning (RD3). The curriculum of the educational program (RD4). Balanced course schedule (RD5).
2. The structure of an educational program implemented partially or completely with the use of distance learning (SP). Criterion parameters. Course Map of the Educational Program (SP1). The correspondence of the set of goals for each of the courses to the goals of the educational program (SP2). Educational Program Competency Map (SP3).
3. Professionalism of trainers (PT). Criterion parameters. Experience in the use of remote technologies (PT1). ICT Competence Level (PT2). Course teaching experience (PT3). Availability of a certificate of completion of courses on the study of distance technologies in education (PT4). 4. IT infrastructure of distance learning (IT). Criterion parameters. Availability of own server infrastructure (IT1) in the university. Availability of algorithms for integrating data into the information system of the university (IT2). Availability of a mail server (IT3). Availability of own server for conferences (IT4) at the university. Availability of an educational portal (IT5).
5. Quality of educational services provided (QE). Criterion parameters. Adaptation of training methods to software for the implementation of distance technologies (QE1). The ability to use a variety of methods and forms of distance learning and assessment (QE2). Ensuring the interaction of individual components of the distance learning system (QE3). Quality control systems for learning students (QE4).
6. Quality of content training materials (QM). Criterion parameters. Matching content to course objectives (QM1). Optimality of the volume of educational material (QM2). Availability of learning materials for learners to understand (QM3). Relevance of submissions (QM4). Content matching to the learning activities required to master it (QM5). The focus of the content is on the cognitive activity of students (QM6). Adaptation of training materials to individual needs of students (QM7). Accounting for pre-and post-course details (QM8). Links to additional materials and resources (QM9).
7. Remote interaction between the trainer and student (DE). Criterion parameters. Feedback on the questions of the studied course (DE1). The efficiency of verification of the student's work (DE2). Assistance in preparing for checkpoints and exams (DE3). Possibility of re-examination of the material (DE4). Feedback when taking the course externally (DE5).
8. Advisory and Support Service (CS). Criterion parameters. Providing trainers with consulting support on working in the distance learning system (CS1). Providing students with consulting support on working in the distance learning system (CS2). The processing speed of student applications (CS3). Availability of prepared comments on frequently asked questions (CS4). Availability of assistant bots (CS5). The speed of the message processing procedure (CS6).
9. Evaluation of Learning Outcomes (EL). Opportunity to discuss learning outcomes and peer review (EL1). Didactic support for intermediate or examination (EL2) testing. The focus of assessment on identifying the level of mastery of competencies of students (EL3). Availability of information about the results of the study of courses (EL4). Availability of an identification system during the exam (EL5). Optimality of intermediate control points, the presence of formative assessment (EL6). Self-assessment of the student within the course (EL7). 10. Course design (CD). Criterion parameters. Availability of methodological recommendations for the development of the course (CD1). Availability of a convenient template for posting course materials (CD2). Visibility of course materials (CD3). Multimedia of the course materials (CD4).Thus, these criteria and their detailed indicators will form the basis of a database of parameters for assessing the quality of distance learning. The database will be used to create an automated quality management system for distance education.

3. Development of a criteria-based decision support model scheme for assessing the quality of distance learning
This stage was implemented on the basis of the method of expert evaluation. Experts were asked to assess the importance of each indicator on the Likert scale. To calculate the total data and calculate the priority vector, the resulting matrices are compiled according to the following algorithm. To begin with, the data of expert assessment surveys were consolidated into a single system. Identical matrices have been compiled. For each indicator, expert averages were found (1): where e ij is the arithmetic mean of each indicator, N is the number of experts interviewed, a ij is the value of the expert assessment of each expert of the i-th row and j-th column in the matrix of options. The indicators were compared in pairs. A matrix of pairwise comparisons for each criterion was then constructed. In the matrix, the average values of expert assessments were placed above the main diagonal. The values below the principal diagonal of the matrix are equal to the corresponding symmetric values relative to the principal diagonal. The main diagonal is filled with the maximum importance value; in our case, it is 5.
To calculate the priority vector, an additional column was created, in each cell of which the geometric mean for each of their rows of the criterion matrix (2) was pre-calculated: where y i is the geometric mean of the i-th row of the criterion, e is the arithmetic mean of each indicator of expert assessments.
The elements of the priority vector are calculated by element-by-element division by the sum of the values of the column vector using the formula (3) where y i is the weight of the priority vector value.
Since the sum of the values of the priority vector should be equal to 1, a verification formula was drawn up that determines the value of the sum of the elements of the priority vector. After the priority vector has been calculated, a formula is used to check the priority by identifying the maximum value in the priority vector column. Fig. 3 shows an example of one of the comparison matrices according to the IT criterion. Experts were invited to rank the indicators in terms of their importance and relevance relative to each other within each of the criteria. Since the experts are made up of different social groups, a joint opinion will make it possible to comprehensively assess the importance of each of the indicators. Despite the fact that the method of Saati makes it possible to add weight for each of the experts, we decided to consider the opinion of the experts equivalent since the quality of the implementation of distance learning equally depends on the involvement of the student, the professionalism of the trainer, the coordinated work of methodologists and representatives of educational and support staff. Figure 3 shows a summary table for one of the criteria, obtained by averaging the data of the expert group. Since the survey was carried out using remote technologies, we can argue that there was no psychological impact or pressure from administrative workers on the teaching staff and students. The results of the calculation of the weight values of all indicators are given in Table 1. Table 1 Results of calculation of weight values of indicators for 10 criteria for assessing the quality of distance learning The sum of the weight values of the indicators for each criterion is 1. This approach was chosen in accordance with the agreed opinion of experts that each of the indicated criteria is of equal importance for assessing the quality of distance learning.
Identifying priorities based on expert assessments, it is possible to assess the existing distance learning system at the university and choose one or another management impact.
Calculations at the previous stage made it possible to build a scheme of a criteria-based decision support model for assessing the quality of distance learning (Fig. 4).
A model is a set of criteria and a hierarchy of indicators related to each of the criteria.
Indicators that have large weight values are located closer to the center of the model.
Weights in the criteria decrease when their location is removed from the center.

Discussion of results of the development of a decision support model for assessing the quality of distance learning
The proposed model makes it possible to combine the criterion approach, the method of expert assessments, and the method of hierarchy analysis.
Depending on the needs of the organization, criteria and indicators for data analysis can be selected according to the priority directions for the development of distance learning.
The presented structural scheme of quality management ( Fig. 1) identified the components of the information environment of the university involved in the process of assessing the quality of distance learning. The functional scheme (Fig. 2) describes the process of assessing the quality of distance learning.
It is important to determine strategic directions in the area of distance learning before starting the analysis. For this purpose, a database of criteria for assessing the quality of distance learning was developed. The database includes 10 criteria and 52 indicators. The selection of criteria was carried out by us through discursive reflection. To analyze the importance of each criterion, the method of expert assessments was applied. To simplify the understanding of experts of the principles of the criterion approach, the jackal Likert was used, which made it possible to rank the criteria by degree of importance. Expert assessments, calculated according to formulas (1) to (3), make it possible to rank the indicators by the degree of importance. Using the method of hierarchy analysis, a matrix is drawn up, the main task of which is to identify priority indicators within each of the criteria (Fig. 3).
The priorities interpreted with the help of the final matrix will allow researchers to identify strategic directions of development within the framework of the study. Based on the data obtained, a scheme of the criteria-based decision support model for assessing the quality of distance learning was built (Fig. 4). The scheme clearly showed the distribution of priorities. The greater the weight of the indicator, the closer to the center of the model it is located, the greater the impact it has on the quality of distance learning.
The proposed methods make it possible to identify priority areas for the development of the organization within the framework of those indicators and criteria that have been previously described. As shown in Fig. 1, this process can be cyclical, such studies can be repeated after a certain period of time. The universality of this method of research is that if the criteria and indicators change depending on the directions of development of the organization, then the algorithms for data processing do not change.
Thus, a feature of the criterion model and the application of the hierarchy analysis method is the ability to take into consideration expert opinion within the framework of the study. Data processing can reveal non-obvious priorities, which can be especially important if the organization is experiencing difficulties and needs to be reformed. Well-designed criteria will identify the main directions of development for making important management decisions. The model reported in this paper can be used not only in educational organizations but also for organizations of the production cycle. The criteria proposed in the work are far from complete but only demonstrate the current state of distance learning at the university. Depending on the changing environmental requirements, the criteria may change but the functional scheme of the distance learning quality management system will remain unchanged.
The negative aspect is that with a small number of experts, there is a high probability of error due to the human factor. It follows that the number of experts should be as large as possible. It should be borne in mind that when working with any social systems, one needs to make an adjustment for the human factor. It is important to have as many specialists of various fields of activity as possible in the team of experts.
We see further development of our research in the study of the results obtained and the controlling effects on the organization of distance learning at the university. The issue of adapting such a methodology for making managerial decisions of the university as a whole and creating an analytical decision support system is being studied.

1.
A universal model of decision support based on the expert assessment of indicators and criteria, the processing of which is carried out using the method of hierarchy analysis, has been proposed. The universality of the method is in the simple processing of the acquired expert assessments.
2. As an example of analysis, a study taking place at the North Kazakhstan University named after M. Kozybayev, Petropavlovsk, Kazakhstan, was considered. 10 criteria and 52 indicators were identified to assess the quality of distance learning. The number of indicators for each criterion ranged from 3 to 9. These criteria and indicators formed the basis of a database for assessing the quality of distance learning.
3. During the expert evaluation of the criteria and indicators of the distance learning quality assessment system, it was determined that the weight of indicators within the criterion ranges from 0.09953 to 0.34262. Based on the weight values obtained, a scheme of the criterion model of decision support for assessing the quality of distance learning was built. This model is dynamic because depending on the overall strategy for the development of distance education, indicators and criteria may change. The model also takes into consideration the change in the importance of a particular indicator depending on the implementation at the university. The proposed methodology will make it possible to find problems in the implementation of distance learning and designate the zone of application of the control influence. This algorithm for conducting research can be scaled to any organization.