Devising An Automated Method to Form the Content of Educational Specialization Disciplines of Maximum Utility for Implementation in the Professional Field

The rapid growth and emergence of breakthrough technologies in the information and communication field requires annual adjustment and modernization of curricula for specialization disciplines in accordance with the current market requirements. A method of prompt response of the educational environment to the requirements of the external environment, based on the creation of a pool of demanded learning outcomes, is proposed. The object of the research is the process of formation of the required content of specialization disciplines. A formalized method for forming the content of educational disciplines of specialization of maximum utility for implementation in the professional field is described based on expert assessments of representatives of the job market. An algorithm for finding a set of learning outcomes for the identified conditions was devised. A software tool that includes a database and two components: a web interface for collecting and primary processing of information and a C++ program module for automating the main algorithm was developed. The result of the software tool is a pool of learning outcomes recommended for implementation in the content of the educational program. The results of an experimental study based on test data, which proved the accuracy of the calculations of the software component, were presented. Verification of data in the area of web technologies and web programming made it possible to identify a list of learning outcomes for implementation in the educational program «Information Systems». The algorithm developed as a result of research can be used to form the demanded content of elements of educational programs in any industry. It is supposed to develop the proposed method when creating interactive training programs on the technological processes of aircraft repair at the enterprises of the aviation industry.


Introduction
Within the framework of higher education, the term «competency» is presented in the form of a complex of abilities, interpersonal and intellectual knowledge, instrumental skills, and ethical attitudes. The whole complex is generated in the process of all forms of educational activity of a student in various sets of incoming elements. In this regard, it is quite difficult to correlate competencies with parts of the curriculum of educational programs. Within the framework of the European Higher Education Space, the term «competency» increasingly refers to the professional activity of a graduate after graduation, directly at the workplace. In the course of completing an educational program, the teacher deals with planning and control of specific learning outcomes for separate components of the curriculum, rather than with the formation of competencies themselves [1]. The same presentation structure is proposed in the materials of the international project «Tuning Educational Structures in Europe» [2].
Consequently, there is a clear distinction between learning outcomes throughout the educational program, which are expressed in the form of competencies, and learning outcomes in separate components of the curriculum, which are expressed in the traditional format of knowledge, habits, and skills [3]. In Kazakhstan (National Chamber of Entrepreneurs «Atameken» [4]) and Russia (Association of Classical Universities of Russia, ACUR [5]), competency maps that contain characteristics of competencies and tables for correlating certain levels of this competency with specific habits, knowledge, and skills, are devised Competencies can be adequately assessed only in the process of State final certification. However, completion of the elements of educational programs by specialization is best represented in the form of learning outcomes.
The existing guidelines for designing educational programs recommend conducting a preliminary survey and analysis of the wishes of potential employers, as, for example, it is presented in paper [6]. However, the work in accordance with the published recommendations is a long and time-consuming process, the successful completion of which depends on the professional skills of an analyst-developer of educational programs.
In addition, in the modern job market, there is a sufficient number of industries and spheres characterized by a fairly frequent update of the required professional knowledge, habits, and skills. Basically, these industries are associated with new technologies, equipment, and software. A special place here is occupied by the IT field. The requirements of the job market in the IT industry are changing dynamically due to the development and emergence of new information and communication technologies. New requirements are a prerequisite for increased competition and work intensification. Annual adjustment/modernization of curricula in the disciplines of specialization in accordance with the current requirements of the market is required.
Thus, there appeared a need to develop a formalized method for the formation of the content of academic specialization disciplines of maximum utility for the implementation in the professional field. By «utility», we will imply the choice for educational programs of the pool of learning outcomes that are most demanded among various participants of the job market of the same specialization. Automation of the process of collecting data on learning outcomes demanded in the external environment, data processing and analysis will help to control the relevance of specialization disciplines. Such a solution will provide an opportunity for timely management of the content of academic disciplines.

Literature review and problem statement
Paper [7] deals with the problem that technological universities are lagging behind the level of development of the IT industry. It was shown that it is possible to ensure effective training of future IT specialists only with the active cooperation of IT companies and IT faculties. The proposed solution includes a model for integrating Microsoft resources and services into the academic process. The drawback of the proposed model is that it is focused on the specific Microsoft company, there is no possibility of using the model to organize cooperation with other IT companies.
Scientists in many countries are concerned about the adaptation of the higher education system to the changing economic environment. Collection [8] contains the most extensive overview of the studied area. It presents essays by 17 authors, which include approaches and studies of the problem of graduates' employment in a changing job market. On the whole, these studies prove the fact that now there is an increased interest in the degree of so-called «readiness for work» and point out the importance of sustainable access to the job market. However, these works do not present any formalized approaches to the organization of the relationship of educational programs with the job market.
According to the authors [9], career prospects and «alignment of education and the future job market» have become more uncertain, individual career paths have become less stable, and graduates should be prepared for this new environment. This means that people must be able to continuously perform, acquire or create work, using their competencies in an optimal way [10] in order to enter the labor market sustainably. The indicator for measuring employment opportunities was proposed in article [10]. This indicator is based on a five-dimensional conceptualization of employment opportunities, in which professional knowledge is complemented by general competencies. The tool may be of interest in the matter of personnel hiring. The disadvantage of the proposed tool is its one-way orientation, there is no feedback from an educational institution.
Educational programs need to be constantly reviewed due to the rapidly changing job market and assess their relevance, as is stated in research [11]. In this regard, there appeared the need to make changes to educational programs. The proof of this statement is presented in paper [12], which also considers the approach to planning changes in educational programs, but this approach is not formalized.
Educational institutions should analyze the job market in order to monitor its changes and requirements. To ensure sustainable entry into this job market, it is necessary to conti nuously invest in improving the alignment between the ne cessary skills of students and higher education programs [13]. In addition, study [13] mentions that an important para meter of the quality of higher education is the quality of the achieved results, which makes it possible to use the results of learning as the main tool in the formation of the content of educational programs. Papers [14][15][16] showed the formation and assessment of competencies in the process of completing educational programs. The works are similar conceptually and in the methods of solutions, they are based on the competency model. However, the competency model was only partially able to assess the level of quality of their higher education by graduates. In addition, the proposed solutions are poorly formalized and time-consuming.
Modeling of the educational process based on learning outcomes is presented in paper [17]. The model uses the distinction between predictable and unpredictable learning outcomes, as well as the distinction between desirable and undesirable ones. The resulting report is intended to help understand the nature and proper use of learning outcomes in teaching and learning. However, the proposed model does not trace the connection with the requirements of the external environment.
Paper [18] solved the problem of increasing the functional efficiency of machine learning of the information and analytical system for assessing compliance with modern requirements of the content of academic disciplines of the Bachelor's level using an example of the speciality «Cybersecurity». The devised method allows the graduating department of a higher education institution to assess the compliance of educational content with modern requirements by machine analysis of the results of a survey of respondents in the monitoring mode. However, this method does not consider the method for changing the content of the educational program in the event of an unsatisfactory external assessment.
There are various approaches to developing the best content of educational programs, for example: -paper [19] proposed the guidelines on the improvement of programs of social and technical colleges; -the authors of paper [20] consider effective methods of developing educational programs; -article [21] described the system of development and evaluation of educational programs.
All the considered approaches were presented in the form of weakly formalized recommendations. This complicates the process of developing the content of educational programs and does not make it possible to respond promptly to the requirements of the external environment. In the existing guidelines for designing educational programs, it is recommended to conduct a preliminary survey and analysis of the wishes of potential employers. However, working in accordance with the published recommendations is a long and time-consuming process, the successful completion of which depends on the professional skills of an analyst-developer of educational programs.
Thus, the analysis of literary data showed that currently: -here is no formalized method for developing an educational program relevant to the modern job market; -there is no automated way to generate the optimal set of learning outcomes that are most useful and demanded in the professional field All this makes it possible to assert that it is appropriate to conduct research into the formalization and automation of the method for forming the content of educational disciplines of specialization of maximum utility.

The aim and objectives of the study
The purpose of this study is to develop an automated method for the formation of the content of academic disciplines of specialization of maximum utility in terms of current market requirements. This will allow an educational institution to respond promptly to the requirements of the external environment in the field of the content of educational programs in the areas of specialization.
To achieve the aim, the following tasks were set: -to perform structural and logical modeling of the process of formation of the content of specialization disciplines as a result of cooperation between IT faculties and IT companies, based on learning outcomes; -to formalize the main provisions and develop an algorithm for an automated method for forming the most demanded content of specialization disciplines; -to verify the method on the example of specialization of IT specialists in the field of web technologies and web programming.

The study materials and methods
This paper is applied research that has a social and scientific-technical aspect. The social aspect involves the estab-lishment of an operational link between the production base and an educational institution. The scientific and technical aspect requires the development of a formalized method for the formation of the demanded content of the elements of the educational program.
The object of the study is the process of forming the content of academic disciplines of specialization of maximum utility from the point of view of the current requirements of the market. The speed and volume of the information flow in education and in any professional activity are rapidly increasing, existing educational materials need to be supplemented with real-time information to prepare students to solve current practical problems. The main hypothesis of the study is that it is possible to automate the process of selection of the most relevant areas in the professional field for the content of educational programs.
The best method for conveying the concept of solving the problem is modeling. Process modeling is an integral part of the study, which makes it possible to decompose the subject area, show the implementation of the stages of the process from the point of view of an external observer. The theoretical significance of process modeling lies in the explanation of key concepts used for further formalization and development. To implement this task, a structural and logical scheme was chosen as a tool.
A structural and logical scheme is a graphical model that displays key concepts located in a certain logical sequence that makes it possible to present the object under study in an integrated form [22].
The second task of the study was solved by presenting the problem in the form of a formal system. To formalize the main provisions, it was decided to use the apparatus of matrix algebra, which can most successfully show the pairwise interaction of all learning outcomes. To visualize the method of formation of the content of educational disciplines of specialization of maximum utility, it was decided to use elements of the graph theory.
To solve the third problem of the study, a two-component software was developed. This is due to the fact that preliminary remote data collection from experts can be indefinitely stretched over time. Since the data from the job market are collected remotely, the corresponding component was developed in the form of a client-server application written in the php programming language. Data storage is implemented in the MySQL relational database. The best solution for data analysis is to develop a component in the C ++ programming language that will ensure the most efficient performance of the algorithm. It was decided to carry out preliminary approbation of the method at the Department of Information and Communication Technologies of the North Kazakhstan University named after M. Kozybayev (Republic of Kazakhstan) on the example of specialization of IT specialists in the field of web-technologies and web-programming.

1. Structural and logical modeling of the process of formation of the content of disciplines of specialization of maximum utility for market requirements
The process of forming the content of educational disciplines of specialization of maximum utility from the point of view of the current requirements of the market involves the implementation of the following obligatory stages: 1) to form a list of the most useful learning outcomes for disciplines of specialization in the result of cooperation of the IT faculty and IT-company; 2) to form the content of educational specialization disciplines of maximum utility for potential employers taking into account the temporal limitations of the educational program.
The preparatory stage of modeling involves making agreements on the used terms and designations. The key concept of the model is learning outcomes. Learning outcomes are acquired knowledge, habits, skills, and formed competencies; what a student will know, understand and be able to do after successful completion of the learning process. The term «learning outcome» was officially used for the first time in the documents of the Bologna Process in 2003 in the communiqu of the Berlin Conference of Ministers of Education [23].
Modeling of the process of formation of the demanded content of disciplines of specialization is realized in the form of a structural and logical scheme shown in Fig. 1.
The subjects interacting with the key concept of the «learning outcome» will be defined by the terms «expert» and «moderator». An expert is a subject who has reliable information about the professional skills of a graduate of an educational institution demanded at an enterprise. It is assumed that the number of experts is a finite, limited set. The term «moderator» defines an experienced employee of an educational institution, an expert in the content of specialization disciplines.
The set of learning outcomes in the specialization area is replenished by experts from the established set. Each expert established a utility score for each outcome. A moderator assigns the optimal time required to achieve a particular learning outcome and also establishes prerequisite links between different results.
As a result of processing all the presented input parameters, we form a group of useful learning outcomes, for which the time spent to achieve each of them and the causal relationships between them are known.

2. Formalization of the main provisions and development of an algorithm for finding a pool of useful outcomes
For a formalized description of the method for the formation of the content of academic disciplines of specialization of maximum utility in the professional field, the following designations were introduced: -n is the number of learning outcomes in the list evaluated by experts; is the vector of scores of the utility of each learning outcome based on processing the experts' scores; - is the optimal time to achieve the learning outcome established by a moderator ∀ = Ts is the total academic hours allocated to study a specialization discipline; m is the number of experts related to an educational institution; Id = (Id 1 ,Id 2 ,…,Id m ) contains the potential number of vacancies at the enterprise of an expert in the analyzed specialization in the long-term forecast and thus determines the significance of the scores of the relevant expert; (1) Thus, the system (2), reflecting the mathematical statement of the optimal choice of learning outcomes with the maximum total utility score was obtained. A limitation for solution in the form of the value of the sum of academic hours alloca ted to study specialization disciplines was imposed.
It should be noted that R res by coefficient a i = 1 determines the output list of learning outcomes of maximum utility included in the content of the educational program.
The following algorithm to find a set of learning outcomes R res according to the conditions determined by the system (2) was devised.  Step 1. Assessment of the utility of each learning outcome, calculated from formula (1), is replaced by the value of effectiveness of each learning outcome. Effectiveness shows the utility of a learning outcome per unit of an academic hour and is calculated from (3): Step 2. The set of graphs, the nodes of which are learning outcomes, and the arcs show the prerequisite connections specified by a moderator, is plotted.
Step 3. For each node from the set of the graph, the accumulated value for effectiveness for all prerequisites up to the element that has no prerequisite is assigned to the accumulated value of the time spent to study for the same elements: Step 4. In accordance with the established conditions (2), learning outcome R d with maximum effectiveness and the total time of which does not exceed assigned Ts for all specialization disciplines is chosen.
Step 5. For element R d , found at Step 4, establish the value of coefficient a d = 1.
Step 6. The accumulated sums of effectiveness and the time to study for each element, excluding the values of the elements included in the chain, are recalculated: Step 7.
Step 4, Step 5, Step 6 are repeated sequentially until there are elements R d , meeting conditions (1).
Visualization of the described method involves plotting a set of graphs, the nodes of which are learning outcomes, and the arcs show the prerequisite relations indicated by a moderator. A prerequisite relationship means that the achievement of a subsequent learning outcome is possible only if the previous result is achieved.
The rules for establishing causal relationships between learning outcomes were determined. The fundamental indicator for analysis is determining a prerequisite for each learning outcome. When forming relationships, it is possible to obtain various, and even unrelated, structures in the form of graphs. Examples of possible relationships between nodes are shown in Fig. 2.
Variant (а) describes the case of unrelated learning outcomes. In case (b), learning outcome R 3 is supposed to be achieved only after achieving outcome R 2 , which, in turn, is a consequence of achieving outcome R 1 . In case (c), the dependence of R 2 and R 3 on R 1 and independence on each other is demonstrated. Case (d) demonstrates the direct dependence of R 3 on R 1 and R 2 . Variant (e) demonstrates the dependence of a node on a previous node.
The resulting group of unrelated graphs will allow proving the reliability of the obtained results thanks to visualization.

3. Verification of the method using an example of specialization of IT specialists in the field of web-technologies and web-programming
The method for forming the content of educational disciplines of specialization of maximum utility will best contribute to the confirmation of the social «order» for a program, its relevance under modern conditions. Preliminary approbation of the method was made on the example of specialization of IT specialists in the field of web technologies and web programming. To organize the experiment, five experts in this field were selected in Petropavlovsk (Republic of Kazakhstan), where the university is located. Table 1 shows a fragment of the collected data and primary calculations (utility of each element is calculated according to (1), effectiveness is calculated according to (3)).
According to the experimental data, based on the established rules, two unrelated graphs, the scheme of which is shown in Fig. 3, were identified.
For each node from the set of the graph, we calculated the accumulated efficiency value for all prerequisites up to an element that does not have a prerequisite and the accumulated value of time spent to learn by the same elements in accordance with (4). It should be pointed out that in the excluding relations, the data of the preceding node with the value of maximum effectiveness are taken into account for calculation in the accumulated values. In Fig. 3, calculations are indicated by the designation {accumulated value of effectiveness; accumulated time value}.
A moderator set the time for the disciplines of specialization Ts = 345 academic hours excluding hours for independent work of students. Thus, as a result of the first iteration, a related chain of learning outcomes R 1 , R 2 , R 6 , R 12 , R 18 , R 19 , recommended for inclusion in the content of academic disciplines as the most useful, was obtained. The last element in the chain (R 19 ) shows the total academic hours required to achieve all the outcomes from the resulting chain.
After completion of the first iteration, the original Ts is replaced with a new value, which is calculated by reducing the original one by the maximum value of the total time from the found chain.
The result of the second iteration is implemented in Fig. 4. For each node of the graphs that is not included in the found chain R 1 , R 2 , R 6 , R 12 , R 18 , R 19 , the accumulated sum of effectiveness and time to study for each element is recalculated in accordance with (5). Since residue Ts is not significant, only one node R 24 is included in the recommended learning outcomes in the second iteration.   As a result of approbation of the method for the formation of the content of disciplines of maximum utility on experimental data, seven learning outcomes of maximum utility recommended for inclusion in the curriculum were obtained. It should be noted that six of them must be studied sequentially and one outcome can be included in the curriculum at any time. In this case, the outcome was obtained in two iterations using the full use of Ts time. However, at other original data, the number of iterations may be higher, and time may be not fully included.
The presented formalized description of the method for the formation of the content of educational disciplines of specialization of maximum utility for implementation in the professional field allowed automating it in the form of a two-component software tool.
The first component is a web interface for the remote interviewing of experts. The collected data are placed in the MySQL database. The structure of the database provides for the storage of information in all areas of specialization existing in an educational institution.
The first component of the software tool, implemented in the form of a web interface, performs the work of the preliminary stage, which includes the following actions: -a moderator enters the information on the experts, including the specified identifier of an expert; -experts edit the list of learning outcomes by the area of specialization; -experts give the utility scores of each learning outcome from the list; -automatic editing of the list of learning outcomes, deletion of the elements with zero scores given by all experts; -a moderator enters the optimal time to achieve the learning outcome for each from the list; -a moderator determines prerequisites for each learning outcome, indicating the type of relation (sequential, aggregating, or excluding); -automatic calculation of the utility score of each learning outcome from (1) and its saving in the database; -automatic calculation of the effectiveness of each learning outcome according to (3) and saving it in the database.
The second component of the software tool is designed to generate automatically a list of the most useful learning outcomes in accordance with the rules presented in the system (2). It is implemented using the C ++ programming language. The possibility of the existence of five different types of relations in the construction of graphs of relationships between the elements makes it difficult to use the known algorithms and data structures in the research. That is why to implement data processing and analysis, it was decided to use a square matrix, the size of which corresponds to the number of learning outcomes in the list. Each element of the matrix is a learning outcome, programmatically implemented in the form of a structure with the following areas: -identifier of the learning outcome; -name of the learning outcome; -optimal time to achieve the outcome; -the accumulated sum of time with the initial value equal to the previous element; -effectiveness of the learning outcome; -the accumulated sum of effectiveness with an initial value equal to the previous element; -a two-dimensional array, in the first line of which the identifiers of prerequisites are indicated, and in the second line -the type of relation with the prerequisite; -flag of including the learning outcome in the final list of the most useful elements, false by default.
Thus, the work of the second component of the software tool consists of sequential performing the following actions: 1) to request to read information from database tables; 2) to generate a matrix in which diagonal elements are filled with the data of the corresponding identifier of the learning outcome; 3) to establish the element of prerequisite, the identifier of which corresponds to the number of the column, for each row of the matrix corresponding to the basic learning outcome; 4) to recalculate the accumulated sums of time and effectiveness for each non-empty element of the matrix with the value of the inclusion flag false; 5) to choose the element with the highest effectiveness, close to the original by time; 6) in the entire chain of elements starting from the found one, to establish the truth of the inclusion flag true and to reduce the value of original time by the magnitude of the accumulated sum of time; 7) to perform actions 3-5 until the element within the original time is found.

Discussion of results of studying the method for the formation of the content of academic disciplines of specialization of maximum utility
At the first stage of the study, we performed structural and logical modeling of the process of forming the demanded content of specialization disciplines, which actually illus-  trates the stages of the implementation of the method of the same name (Fig. 1). The presented model demonstrates the interaction of an educational institution with representatives of enterprises within the set goal. Thus, it was determined that 2 types of subjects participate in the process: «a moderator», as a representative of an educational institution, and a set of «experts» -representatives of the job market. As a result of modeling, 3 main parameters that help to identify useful learning outcomes were obtained: experts set a utility score for each learning outcome, and a moderator sets prerequisites and minimum time to achieve each outcome. The structural and logical model served as the basis for the next task -formalization. At Stage 2, a formalized setting of the problem of choosing the most useful learning outcomes in the form of a system of conditions (2) was performed. The introduction of the methods for quantitative assessment can be considered a special achievement of this study: -utility of (demand for) learning outcomes in the external environment (1); -effectiveness of each outcome (3), i.e. its utility per unit of selected time.
To illustrate the «manual» calculation, the visualization method in the form of a set of graphs, the nodes of which are learning outcomes, was proposed. The prerequisite relations of 37 learning outcomes from the field of web technology and programming, 35 from the field of information systems design, 21 from the field of 3D modeling and virtual reality were analyzed. Analysis revealed 5 types of possible relations between learning outcomes, presented in Fig. 2. It is possible that in other specializations, there will be different combinations of presented relations. However, in general, this does not affect the calculations of the method under consideration, so the issue of studying combined relations was not considered in this article.
As a result of formalization, an algorithm, which is a good basis for further automation, was obtained. Software implementation of any algorithm is necessarily accompanied by making verification tests. The number of learning outcomes in the test is limited to 50 since in three specializations of the IT area, the maximum number of learning outcomes is 37. In addition, «manual» calculations for a large number of values are time-consuming. For each test, we made a table modeled by Table 1, where the scores of each expert, prerequisites, and learning time were randomly determined. The characteristics of compiled tests and the time to complete the program for each test are shown in Table 2.
«Manual» and automated calculations on the developed tests by 100 % proved the accuracy of the calculations of the software component. The program time of the execution of the algorithm implemented in C ++ showed a sequential increase depending on the amount of processed data. The values of algorithm execution time indicate that more research is needed to optimize the algorithm.
The experiment with actual data from experts in the field of specialization of web technologies and web programming enabled obtaining the list of 7 learning outcomes from 37 initial ones. Five selected experts formed a list of 37 learning outcomes and determined the scores for each outcome. The moderator indicated a list of prerequisites and the optimal study time for each learning outcome. The obtained demanded results in the North Kazakhstan region can be included as a specialization in the educational program «Information Systems». The main achievement of the study is the data analysis method, which allows, based on software processing of input parameters, obtaining the desired list of learning outcomes for educational programs. The specific feature of the proposed method is the possibility of automation, which will effectively control the process of tuning/modernizing the content of educational programs. Taking into consideration recent trends, the method is based on the relationship of learning outcomes. However, this is not a limitation or a significant drawback, since, with a little refinement, the algorithm can work with competency terms.
The limitation of the method for the formation of the most demanded content of specialization disciplines is the dependence on expert assessments. Methods of expert assessments have long been successfully adopted for decision-making in various fields, but there is a related task -the selection of experts. In the context of this study, the selection of the quantitative and qualitative group of experts should be made taking into consideration the scope of the study and the reliability of the estimates.
The scope determines the need to involve in the examination the specialists from one region or several regions of the country, depending on the supposed area of the job market for graduates of an educational institution. The best solution for an educational institution when choosing the number of experts can be considered compliance with the condition: value Id j j m = ∑ 1 must exceed the predicted number of graduates of a specialization. The reliability of the assessments of the study group depends on the level of knowledge of separate experts. Thus, an unsuccessful selection of an expert group may adversely affect the results of the proposed method.
The direction of further research is focused on optimizing the automated algorithm for finding useful learning outcomes, as well as the methods for automated distribution of the results obtained in specialization disciplines during the entire period of study at a university. The proposed method is supposed to be developed in the field of devising the elements of new educational programs, in particular, in the use of 3D modeling technologies and virtual reality. The research results will be tested when creating interactive training programs on technological processes of aircraft repair at enterprises of the aviation industry.

Conclusions
1. Modeling of the process of formation of the demanded content of disciplines of specialization was carried out. As a result of modeling, a structural and logical scheme was constructed. The scheme displays the relationships of the subjects of the process with the objects of control -«learning outcomes». Thanks to modeling, the main provisions of the studied process were determined. 3 parameters affecting the choice of the most useful learning outcomes were identified: expert assessments, prerequisites, and minimum time required to achieve each learning outcome.
2. The formalization of the method for the formation of the most demanded content of specialization disciplines was implemented. To formalize the main provisions, the apparatus of matrix algebra was used. The formula for assessing the utility of each learning outcome, defined as the arithmetic weighted mean of a set of expert scores with the weights that assign the significance of each expert, was obtained. The process of finding a set of the most demanded learning outcomes based on analysis of input parameters was implemented in the form of an algorithm. The algorithm reflects the optimal solution for the choice of learning outcomes with the maximum total utility score, limited by the total time allocated for specialization training. An indicator of the effectiveness of a learning outcome result for its inclusion in the educational program, which is a utility score per unit of an academic hour, was found. We proposed the method of visualizing the operation of the developed algorithm by plotting a set of related graphs of a special shape due to the existence of a non-trivial type of relationship -an excluding relation. In total, 5 possible types of relations between the nodes of graphs -learning outcomes -were determined.
3. An experimental study of the operability of the method for forming the content of academic disciplines on the example of specialization of IT specialists in the field of web technologies and web programming was carried out. The input parameters from the participating actors (experts and moderator) were selected online. For this purpose, a clientserver application, including interfaces for an expert and a moderator, implemented in the php programming language, was developed. All analyzed information is placed in the MySQL database. Data processing and analysis were performed in two ways. First, «manual» calculations were performed using visualization in the form of graphs. The method for plotting the graphs of learning outcomes and the mechanism for finding a solution for them were shown. Then the algorithm for finding the list of the most useful learning outcomes was automated by implementing the software component in the C ++ programming language. Verification of the accuracy of the program component execution was proved by performing 15 test data with a maximum number of learning outcomes of 50. Thus, a two-component software, which is an automation of the method stated in the article was obtained. The experiment with actual data from the experts in the field of specialization of web technologies and web programming made it possible to obtain a list of 7 learning outcomes from 37 initial ones. The resulting list is a recommendation for inclusion in the educational program. The conducted experiments on actual data showed the viability of the method, based on which it is possible to form flexibly the content of specialization disciplines in accordance with modern market requirements.