DEVISING AN INTEGRATED METHOD FOR EVALUATING THE EFFICIENCY OF SCRUM-BASED PROJECTS IN THE FIELD OF INFORMATION TECHNOLOGY

Modern projects in the field of information technology (IT) are characterized by complexity, innovative technologies and technical solutions, a close combination of organizational and technological components, their mutual integration. Managing such projects requires reducing the time of practical implementation of fundamentally new ideas and technologies, a significant change in the nature and dynamics of project and technological processes, flexibility, efficiency, and mobility of management decisions. Along with the use of various tools and software in project management (Microsoft Project, Atlassian JIRA, Redmine, Trac, Trello, etc.), it is relevant to evaluate the effectiveness of the IT project. This could ensure the evaluation of the design and technological process, objectively reflect the results of teamwork and project performance, as well as ensure the most effective project management in general. Given that IT companies use Agile software development methodologies, process management is implemented to improve the results of their activities. Today, the Scrum methodology [1–3] is one of the most popular, providing such opportunities. Therefore, the task of evaluating IT projects requires new approaches and solutions. Thus, there is a need to devise new methods for evaluating the effectiveness of the IT project in the context of a flexible Scrum methodology, which would ensure compliance with time limits, reduce resource overspending and project losses. Such methods could contribute to improving the efficiency of the IT project. This substantiates the relevance of our study.


Introduction
Modern projects in the field of information technology (IT) are characterized by complexity, innovative technologies and technical solutions, a close combination of organizational and technological components, their mutual integration. Managing such projects requires reducing the time of practical implementation of fundamentally new ideas and technologies, a significant change in the nature and dynamics of project and technological processes, flexibility, efficiency, and mobility of management decisions. Along with the use of various tools and software in project management (Microsoft Project, Atlassian JIRA, Redmine, Trac, Trello, etc.), it is relevant to evaluate the effectiveness of the IT project. This could ensure the evaluation of the design and technological process, objectively reflect the results of teamwork and project performance, as well as ensure the most effective project management in general. Given that IT companies use Agile software development methodologies, process management is implemented to improve the results of their activities. Today, the Scrum methodology [1][2][3] is one of the most popular, providing such opportunities. Therefore, the task of evaluating IT projects requires new approaches and solutions. Thus, there is a need to devise new methods for evaluating the effectiveness of the IT project in the context of a flexible Scrum methodology, which would ensure compliance with time limits, reduce resource overspending and project losses. Such methods could contribute to improving the efficiency of the IT project. This substantiates the relevance of our study. Therefore, to evaluate the effectiveness of an IT project, the use of traditional tools such as analysis of discounted cash flows [4], apparatus of statistical methods [5], economic and mathematical models [6] does not give the desired result.
Traditionally, in the process of researching the effectiveness of projects, scientists focus on the use of dynamic methods based mainly on the discounting of project cash flows. Paper [4] reports the results of studying the main performance indicators based on cash flows. It has been shown that this approach provides the possibility of forecasting positive and negative cash flows for the planned period, discounted at the appropriate rate. However, it is necessary to accept a series of assumptions that are quite difficult to implement in practice. Moreover, the issues related to the correctness of the assessment of both the amount of initial investments and current costs and revenues over the entire period of the project implementation remained unresolved. Work [5] proposes the use of a comprehensive indicator of integrated economic efficiency together with the condition of the optimal payback period of the project. However, all the conditionality of such data is obvious even in a stable economy with a predictable level and price structure and a high degree of market knowledge. In economics, the volume of assumptions that must be accepted when calculating cash flows is incomparably higher. In paper [6], the authors propose in the study of the effectiveness to move from criteria estimates to parametric data, which would take into consideration the uncertainty and subjectivity of assessments of financial parameters. However, for calculations using dynamic methods, a prerequisite for the stability of indicators in which cash flows are evaluated is used. That is, those methods do not make it possible to fully determine the value and effectiveness of the project.
Paper [7] considers approaches based on determining the usefulness of the project for stakeholders under specified conditions. However, for IT projects implemented in the context of Scrum application, it is important to promptly determine the effectiveness of the project for its direct participants and stakeholders in real time. Work [8] reports the results of research into the properties of projects in a rapidly changing environment. However, the issues related to the development of a tool for determining the effectiveness of projects under the conditions of dynamic changes remained unresolved. In [9], to evaluate the effectiveness of tasks in an IT project, it is proposed to use a burnout chart. However, this standard IT project performance metric provides the ability only to determine the dynamics of the project as the number of implemented and unrealized tasks. At the same time, the empirical relationship between the overall effectiveness of the IT project and its components is not investigated. Paper [10] investigates the Velocity metrics of an IT project, i.e. the speed of the team. However, the system of knowledge and evaluation of competence of project team members is not taken into consideration. A given metric can be considered as a partial indicator and does not fully reveal the essence of the project efficiency.
Our review of works [4][5][6][7][8][9][10] showed that existing mathematical models and methods do not fully evaluate the effectiveness of IT projects implemented under Scrum conditions. The dynamics of IT industry development requires devising new methods for the integrated evaluation of the effectiveness of IT projects in which efficiency criteria could characterize, in the interconnection and mutual agreement, both the technological and organizational components of an IT project. Therefore, the combined application of formalized, intelligent, and expert methods would provide opportunities to study the effectiveness of an IT project in order to make effective management decisions and promote the productivity of teamwork based on Scrum.

The aim and objectives of the study
The purpose of the original work was to devise an integrated method for evaluating the effectiveness of an IT project in Scrum based on the combined use of formalized, intelligent, and expert methods. This would provide an opportunity to evaluate the effectiveness of an IT project, taking into consideration the peculiarities of organizational and technological components in real-time to improve the productivity of teamwork.
To achieve the set aim, the following tasks have been solved: -to substantiate and investigate the procedure of integrated evaluation of project efficiency in terms of Scrum; -to build a model of the efficiency index, which is the basis of an integrated method for evaluating the effectiveness of an IT project in terms of Scrum.

The study materials and methods
An IT project includes closely interrelated organizational and technological components. An IT project is characterized by complexity and variable structure, the presence and change of many goals, activities, multifunctionality of elements, etc. This gives grounds to categorize a project in the class of organizational and technological systems [11]. The class of organizational and technological systems (OTS), according to [12], has the properties of both technological and organizational systems and consists of two fundamentally different subsystems, each of which performs its management functions. If f is the OTS control functions, and f∈F, where F is a set of all control functions implemented by the system, then fo are the control functions implemented by the organizational subsystem, ft are the control functions implemented by the technological subsystem. The introduced functions belong, respectively, to the sets: fo∈F and ft∈F. Then, we have ∪ = .
o t F F F Management decision-making is aimed at improving the efficiency of the organizational and technological system, therefore, to determine the expected effect, we introduce a criterion for the effectiveness ef, which is determined as follows: .
Performance indicators can be the time of decision making, its reliability, economic assessment of the consequences of the decision. Therefore, it is necessary to obtain such a value of the efficiency criterion that: Studies into the organizational and technological systems are also reported in [13][14][15][16][17], whose authors used various methods such as analytical modeling, system dynamics, agent modeling, etc. Works [12][13][14][15][16][17] show that the organizational and technological systems are nonlinear, dissipative, dynamic. The processes that take place in such systems are complex, the management of which contains, along with formalized and poorly structured tasks, a class of mixed-type tasks. Both analytical and heuristic models and methods are used to study such processes. In addition, the organizational and technological processes are hybrid in nature. This predetermines the fact that the use of only formalized methods of synthesis of management, which have proven proper for technical systems, or only heuristic methods characteristic of organizational objects, does not give the desired result.
A characteristic feature of modern IT projects is their complexity. The level of complexity is determined not only by a large number of interdependent components of the project but also by a high degree of interdependence of their characteristics, emergent properties, variety of functions, variety of efficiency criteria. These properties make it possible to consider an IT project as a system, while the presence of an organizational component, on the one hand, and a technological one, on the other hand, refer it to the class of organizational and technological systems. Therefore, devising an integrated method for evaluating the effectiveness of an IT project in terms of Scrum should be implemented in the class of organizational and technical systems. This will provide the possibility of integrated use of formalized methods and means of artificial intelligence, as well as expert methods.
The effectiveness of an IT project is an integrated indicator that characterizes the ultimate results and costs over a certain period. The main feature of efficiency may be the need to achieve an integrated goal of both organizational and project-technological activities at the lowest cost of resources or time. In this regard, a distinctive feature of those organizational and technological processes that affect the effectiveness of an IT project is their dependence on quantitative and qualitative indicators of various kinds and nature. In addition, it is worth noting the dependence on the timing of an IT project, the pace of development of the IT industry, the introduction of innovative technologies, the level of qualification of industry workers.
To evaluate the effectiveness of an IT project in terms of Scrum and obtain adequate forecast data, important is not the number of indicators that need to be evaluated but the proper choice of criteria used to preserve the value of the project. The total value of an IT project should exceed the sum of values of both the organizational and its design and technological component. To achieve this, it is necessary to implement an IT project management in terms of Scrum so that all its components can generate synergistic effects when interacting with each other within the project. This would ensure the creation of completely new and creative values by increasing its effectiveness.
Taking into consideration that both the organizational and technological components of the project are interdependent, the effectiveness of an IT project can be represented as a composition of the terminal criteria of each component that have a significant impact on the effectiveness of an IT project in general. From a position of value, this means that the effectiveness criteria of an IT project should be correlated not only with the content of its components but also in agreement with the overall value of the project.
Since an IT project is implemented under Scrum conditions, it ultimately must satisfy the interests of all stakeholders (Scrum Master, Scrum Team, Product owner Users, Stakeholders, Consulting Experts). Therefore, it is desirable to form balanced efficiency criteria that could provide an operational, objective, and integrated assessment of the prospects and capabilities of an IT project. The IT project in terms of Scrum strives to create new values as a result of the pleasant satisfaction of all participants. At the same time, it is important to highlight and analyze innovative approaches, approaches to the transformation of the organization of project goals. In this regard, there is a need for a comprehensive and integrated assessment of the effectiveness of an IT project in terms of Scrum based on the combined use of formalized, intelligent, and expert methods. Therefore, this task should be solved in the class of organizational and technological systems. The effectiveness of an IT project is formed as an integrated characteristic and can be reflected through an integrated criterion that includes both the organizational and technological components. Therefore, the evaluation of the effectiveness of an IT project can be objectively carried out using a set, a system of criteria. This would provide a detailed and complete description of both project and technological processes, as well as taking into consideration its organizational component at the level of competence of project team members. Determining the effectiveness of an IT project in terms of Scrum is important primarily for all stakeholders. On the one hand, these are Scrum Master, Scrum Team, Product owner, as well as representatives of the other party Users, Stakeholders, Consulting Experts interested in the project. Therefore, it is necessary to systematically, in detail and dynamics, analyze the effectiveness of an IT project since its economic prospect depends on improving the effectiveness of an IT project.
Traditionally, efficiency criteria are based on the difference in the results obtained and the costs incurred. At the same time, the obtained results and costs are measured in monetary units. When devising a system for evaluating the effectiveness of an IT project in terms of Scrum, we shall proceed from the fact that the effectiveness of an IT project is described by different criteria. On the one hand, the criteria characterize the organizational component of an IT project, and, on the other hand, technological. The effectiveness criteria of an IT project are in interconnection and interoperability. Since it is advisable to evaluate the effectiveness of an IT project in stages, based on a set of basic criteria, the basic procedure is based on the method of evaluating the effectiveness of an IT project (Fig. 1). A given procedure takes into consideration the complexity of an IT project structure, its organizational and technological components.
Below are brief explanations to Fig. 1. A. A set of criteria for evaluating the effectiveness of an IT project in terms of Scrum is determined, taking into consideration both the organizational and technological components. А1. A general (global) criterion of efficiency is formed. The global criterion, as a rule, decomposes into a set of criteria of the next level, the latter decomposes into their components, etc. until we receive the final (terminal) criteria. Thus, a hierarchy of criteria is formed -a criterion tree.
A2. To evaluate the effectiveness, it is necessary to rank the criteria. Ranking and selection are carried out according to weight coefficients based on the method of hierarchy analysis [61] with the involvement of experts.
A3. Efficiency criteria are interpreted as linguistic variables that accept qualitative values (established by experts and expressed by verbal formulations such as low, medium, high, etc.). The qualitative scale of the criterion is mapped onto a quantitative scale at which each linguistic estimate is matched with a point that expresses a certain number of points.
A4. The IT project performance index is being formed. B. The project situation is being investigated. B1. According to the specified criteria, recommendations for improving the situation are determined.
B2. Choosing the best solution based on the alternatives studied.
The methods used to analyze and evaluate the effectiveness of an IT project are usually based on economic indicators and take into consideration only the ratio of costs and results in view of the interests of the participating parties. However, a full and integrated assessment is not always possible if there is only quantitative information. Therefore, it is effective to use in this procedure a combined combination of intelligent and expert methods, which could ensure that qualitative information is acquired.
In the basic procedure for evaluating the effectiveness of an IT project in terms of Scrum, expert methods are used, in particular, the "intellectual assault" method. As experts, there may be representatives of all interested parties to the project, according to the Scrum methodology (Scrum Master, Scrum Team, Product Owner, Users, Stakeholders, Consulting Experts). Experts discuss and analyze the relevant IT project criteria and put up expert assessments on each criterion. In addition, they translate qualitative estimates into quantitative, and vice versa.

2. Building a project performance index model
To solve the task of evaluating the effectiveness of an IT project in terms of Scrum, it is proposed to build a criterion tree based on the methodology given in [18], taking into consideration its organizational and design and technological component (Fig. 2). Substantiation of IT project efficiency criteria under Scrum conditions belonging to the organizational component is investigated in [19,20]. The criteria for the effectiveness of the technological component are given in [21]. It should be emphasized that all criteria for the effectiveness of an IT project are in interconnection and interoperability. The content of the criteria is given in Table 1.
The weight coefficients of the tops of the tree have been determined, characterizing the importance (significance) of the relevant criteria. For this purpose, the following structures were constructed.
Two-level fragments consisting of the parent vertex and a set of incidental vertices -descendants -are highlighted in the tree (when traversing it from top to bottom). A fragment in which the root vertex of a tree is the parent is assigned a zero rank. The vertices of the lower level of this fragment are the parent for fragments of the first rank, etc. In the example in Fig. 1, a zero rank is assigned to fragment {k 0 , k 1 , k 2 }, the first rankfragments k 1 , k 1.   Table 1 Content of criteria in the criterion tree

Criterion designation
Criterion content k 0 Effectiveness of an IT project k 1 Productivity of an IT project team k 2 Complexity of an IT project k 1.1 Competence of project team members k 1.2 Velocity k 2. 1 Sprint backlog k 2.2 Product backlog k 2. 3 Protection profile k 1.1.1 Level of theoretical knowledge of project team members k 1.1.2 Gained practical experience of project team members k 1.1.3 Level of certification of project team members k 1 For each fragment of the criterion tree, starting from zero rank, a matrix of pairwise comparisons of the significance of criteria is constructed. For the fragment {k 0 , k 1 , k 2 }, this matrix is given in Table 2. Table 2 Matrix of pairwise comparisons for a zero-rank fragment The rows (columns) in the matrix meet the criteria of the vertices of the lower level of the tree fragment. The upper cell of the leftmost column contains the specified weight of the parent vertex of the fragment (for the root of the global criterion tree k0, the weight is w(k0)=1).
At the intersection of the row ki and the column kj, the rij value is specified kj, equal to: − 1, if ki=kj; − the score count bij on the scale of the comparative significance of criteria (Table 3), if ki is more important than kj; − the 1/bij value if ki is less important than kj. Table 3 Scale of comparative significance of criteria Linguistic value Numerical (point) value The same significance of the criteria ki and kj 1 Some (weak) advantage of the significance of ki over the significance of kj 3 Significant (strong) advantage of ki over kj 5 An obvious (very strong) advantage of ki over kj 7 The absolute advantage of ki over kj 9 Intermediate values between adjacent scale values 2, 4, 6, 8 The weight of criterion k i is determined as: where w(k p ) is the weight of the parent vertex of the fragment; n -the number of criteria of the lower level of the fragment of the tree. The weights of criteria for the vertices of the lower level are recorded in the extreme right column of the matrix.
By performing the described procedure sequentially for all fragments of the criterion tree (Fig. 2), the result was obtained, given in Table 4 (processing of matrices and calculation of weight coefficients was carried out using the program). Table 4 Weight coefficients of IT project effectiveness criteria  Table 4, asterisks mark the terminal criteria. After organizing them in descending order and accepting the limit value w(ki)>0.01, a set of the most significant terminal criteria was built: k1.1.2, k1.2.2, k2.2.1, k1.1.3 (marked with two asterisks - Table 4).
In Table 5, the selected criteria are reassigned accordingly k1, … k4. Terminal criteria are represented as fuzzy sets [22,23]. For each criterion ki, i=1, .., 4, its membership function µ(ki) is constructed. The plots of µ(ki) functions are shown in Fig. 3-6 The IT project effectiveness index (W) has been calculated, equal to the sum of the products of the ki criteria weight coefficients, i=1,..,4 ( Table 5)  .
The procedure for evaluating the effectiveness of an IT project in terms of Scrum based on the proposed IT project performance index makes it possible: -to quantify the effectiveness of an IT project; -to establish a logical connection among the criteria that characterize, on the one hand, the organizational component of an IT project, and, on the other hand, the design and technological one; -to conclude on the effectiveness of an IT project in terms of Scrum in real time.
In the original study, a simple academic example of an IT project was taken as a basis, which made it possible to illustrate the proposed procedure for evaluating the effectiveness of an IT project in terms of Scrum. Important is to study the mutually agreed relationship between the different components of an IT project. This determines the overall effectiveness of an IT project and the level of satisfaction of all stakeholders, which is in principle typical of any IT project.

Discussion of results of studying the integrated method for evaluating the effectiveness of a project in terms of the flexible Scrum methodology
The integrated method of evaluating the effectiveness of an IT project in terms of Scrum is based on the combined use of formalized, intelligent, and expert methods. Expert methods ensure the acquisition of information in situations where we have information of a qualitative nature and the use of other methods is impossible. However, the use of expert methods depends on the competence of experts and the techniques of expert surveys, which is a limitation of a given integrated method. Therefore, the improvement of the procedure for selecting experts and conducting an expert survey is a would-be advancement of this study. The use of statistical methods as an alternative to expert methods could ensure obtaining experimental data but would not provide an opportunity to take into consideration qualitative information.
The comprehensive application of formalized, intelligent, and expert methods in the class of organizational and technological systems for evaluating the effectiveness of IT projects in terms of Scrum has several advantages: -a reasonable reduction in the number of terminal criteria of efficiency (Table 5) provides opportunities for obtaining an IT project performance index (4) without going beyond the psychological capabilities of experts; -an IT project performance index (4) ensures prompt assessment of the project's effectiveness in terms of Scrum for participants and stakeholders in real time. Unlike [7], this avoids overspending and losses in the case of a negative value;  -the established link between the criteria that characterize, on the one hand, the organizational component of an IT project, and, on the other hand, the design and technological one, unlike [9,10], indicates the interdependence of efficiency criteria and their overall impact on the project efficiency in general.
The integrated method of evaluating the effectiveness of an IT project in terms of Scrum provides prompt management decision-making in IT projects implemented when Scrum is employed. The practical result of this method is the increased effectiveness of an IT project by improving the productivity of teamwork, reducing the time of implementation of an IT project, overspent resources and losses in the project, avoiding conflict situations. We assessed the qualitative and quantitative impact of each criterion on the efficiency index based on the calculations in which the argument under consideration was conditionally changed by 1 % of the average value at a fixed value of other arguments for the IT project under study when performing each task. This provided an opportunity to increase the efficiency of the project by 1.52 %.
In the future, the proposed procedure for evaluating the effectiveness of an IT project may be the basis of the information technology of IT project management and the appropriate decision support system. IT project management information technology could provide opportunities for collecting, preserving, editing, processing, presenting, and disseminating information about the state of implementation of an IT project and its effectiveness to all stakeholders; it is the likely subject of further research.

Conclusions
1. An integrated method for evaluating the effectiveness of IT projects developed in the class of organizational and technological systems based on the combined use of formalized, intelligent, and expert methods has been devised. A given method involves determining the IT project performance index based on which management decisions are made promptly, as well as determining the future strategy. The practical result of applying this integrated method of evaluating the effectiveness of IT projects is the increased efficiency of an IT project by improving the productivity of teamwork, as well as reducing project losses and overspending of financial resources.
2. The built model of an IT project efficiency index based on the method of hierarchy analysis, fuzzy sets theory, qualitative decision-making methods is the basis of an integrated method for evaluating the effectiveness of IT projects. A given model provides a detailed (structural) pattern of the global criterion and, given our study results, enables a reduction in the number of terminal criteria.