DEVISING AN IMAGE PROCESSING METHOD FOR TRANSPORT INFRASTRUCTURE MONITORING SYSTEMS

Identifying and categorizing local structures (such as contours and edges) in color images is important in many areas, such as image segmentation, image mapping, object recognition, visual tracking in image processing areas, and computer vision. Most of the images currently in use are color images. Therefore, increased attention is paid to the use of color information to identify and categorize the local characteristics of the image. Such a method of information acquisition as aerial photography from an unmanned aerial vehicle (UAV) is the most cost-effective, remote, and efficient way to provide consumers with the necessary information about any object of interest on the surface [1, 2]. There are a series of tasks solving which requires analysis of the acquired geospatial information that identifies the geographical location and various properties of natural or artificially created objects, as well as their contours [3, 4]. This information could be obtained by remote sensing, mapping, and various types of aerial imagery. Such tasks include operational tasks that are solved through the use of UAVs, for example, for dynamic monitoring (evaluation and analysis) of the state of transport infrastructure (road surface quality) [5]. The main feature of operational databases of geoinformation systems is the ability to represent data in the form of spatial-time layers of flight mapping images. Analysis of cartographic images makes it possible to recognize and predict the dynamics of processes, which, in turn, makes it possible to take timely and informed decisions in intelligent control systems [6]. In particular, cartographic images of UAV flights created in operational databases make it possible to monitor transport infrastructure; a separate case of such monitoring is control of the state of road markings. The condition of the road is a set of individual elements that affect the comfort of travel, health and safety of people who use the road. In addition, it affects the transportation costs of enterprises, the cost of production, and, consequently, the economic life of the regions. The unsatisfactory functional condition of roads increases fuel consumption, leads to delays of goods and passengers on the road, and sometimes to damage and loss of goods, damage to vehicles. To solve the task related to monitoring the transport infrastructure based on geospatial data acquired from UAVs, it is necessary to devise computer vision methods for the How to Cite: Volkov, O., Komar, M., Volosheniuk, D. (2021). Devising an image processing method for transport


Introduction
Identifying and categorizing local structures (such as contours and edges) in color images is important in many areas, such as image segmentation, image mapping, object recognition, visual tracking in image processing areas, and computer vision. Most of the images currently in use are color images. Therefore, increased attention is paid to the use of color information to identify and categorize the local characteristics of the image.
Such a method of information acquisition as aerial photography from an unmanned aerial vehicle (UAV) is the most cost-effective, remote, and efficient way to provide consumers with the necessary information about any object of interest on the surface [1,2].
There are a series of tasks solving which requires analysis of the acquired geospatial information that identifies the geographical location and various properties of natural or artificially created objects, as well as their contours [3,4]. This information could be obtained by remote sensing, mapping, and various types of aerial imagery. Such tasks include operational tasks that are solved through the use of UAVs, for example, for dynamic monitoring (evaluation and anal-ysis) of the state of transport infrastructure (road surface quality) [5]. The main feature of operational databases of geoinformation systems is the ability to represent data in the form of spatial-time layers of flight mapping images. Analysis of cartographic images makes it possible to recognize and predict the dynamics of processes, which, in turn, makes it possible to take timely and informed decisions in intelligent control systems [6].
In particular, cartographic images of UAV flights created in operational databases make it possible to monitor transport infrastructure; a separate case of such monitoring is control of the state of road markings. The condition of the road is a set of individual elements that affect the comfort of travel, health and safety of people who use the road. In addition, it affects the transportation costs of enterprises, the cost of production, and, consequently, the economic life of the regions. The unsatisfactory functional condition of roads increases fuel consumption, leads to delays of goods and passengers on the road, and sometimes to damage and loss of goods, damage to vehicles.
To solve the task related to monitoring the transport infrastructure based on geospatial data acquired from UAVs, it is necessary to devise computer vision methods for the extraction and use of environmental information. It is a relevant task to build a method for detecting contours that would meet the requirements for speed, accuracy, and ease of implementation on the hardware aboard UAVs. Because computer vision algorithms must be performed on mobile devices that can be installed on board UAVs. One of the requirements for such methods is that they must be implemented on almost any computing platform, both mobile and stationary type.

Literature review and problem statement
The main elements of road markings are lines of a certain thickness, for the segmentation of which one can use a series of methods that solve the task of contour detection. The authors of works [7,8] explore the application of a standard Hough transform to detect the shape and state of road markings. Those studies reveal the possibilities of applying the Hough transform to a reliable decision-making system to support technicians. The main advantages of that method are a fairly significant accuracy of detecting objects in an image, the ability to work with a high level of noise in an image. The main disadvantage of the method is the considerable complexity of computations, which increases along with the number of parameters that the desired geometric object is expressed with [9].
Another approach was proposed in work [10] by using the probabilistic Hough transform in the warning system for leaving the lane. When trying to apply the algorithm proposed by the authors to control the state of road markings, the results obtained using the probabilistic Hough transform method are unsatisfactory and of little use for further processing. Since the method itself is probabilistic, its results, therefore, may turn unpredictable. The resulting selected segments do not reflect the qualitative and quantitative indicators of the objects of interest in the image. A likely option to overcome such difficulties is to use the method proposed in [11]. However, the effectiveness of the method is ensured only with a clearly defined marking of the road.
An alternative to Hough transforms could be Kenny's contour detector. The authors of [12] used this very method to get the designation of the edge of the lane in the image of the road in the system of assistance to the driver of the car based on computer vision. The limitations of Kenny's operator use are that the markings should be clearly applied, which is not always possible because, in particular, the state of marking is the aim of monitoring it. Kenny's operator sometimes produces false results, which is due to the sensitivity of the method to noise in the image. For reliable use of Kenny's operator, significant preprocessing and filtering of the image are required [13]. The disadvantage of Kenny's operator is that the calculations are complex enough, which makes it almost impossible to use it for real-time calculations.
A less demanding method for computing power is the Laplace operator. In [14], the Laplace operator is used in the detection of roadway contours for road safety systems, as well as provide the hardware and software necessary for its implementation. The proposed method demonstrates greater efficiency from a computational point of view but produces slightly worse results compared to the Hough transform and the Kenny operator. This is due to the loss of part of the information related to the direction of the contours, which leads to intermittence detection of contours and sensitivity to noise [15].
A likely option for overcoming the corresponding difficulties is to use a Sobel operator. This approach was used in works [16,17] but, due to the features inherent in the Sobel operator, there is a loss of parts in the image, which negatively affects the accuracy of the method [18]. Interesting is the result reported by the authors of paper [19]. The Sobel operator is applied to the image, then, based on the Otsu method, an effective binary coding algorithm for the road lane image is proposed, to segment the edge of the road by the Hough transform method. Those studies make it possible to conclude that by using the Sobel operator and constructing image filtering algorithms, one can devise an image processing method that would eliminate problems that arise when solving the issue of selecting a contour in an image.
Thus, a significant body of research allows us to assert that it is expedient to conduct a study on devising a method of filtering and processing images based on Sobel gradients. The main requirements for which are performance, that is, the ability to work on a real-time scale, and to ensure the accuracy and reliability of operation.

The aim and objectives of the study
The purpose of this study is to devise an image processing method for vehicle infrastructure monitoring systems that would meet the requirements for speed, accuracy, and implementation on the hardware aboard the UAV. The results could make it possible to qualitatively and meaningfully distinguish the contours of objects of interest, as well as expand the range of tasks for which UAVs are used, in particular, to monitor the state of transport infrastructure.
To accomplish the aim, the following tasks have been set: -to design a combined noise removal filter in a binary image; -to devise a method for separating the contours of a cluster of points using histograms to eliminate the shortcomings of known methods and to qualitatively and meaningfully separate the boundaries of objects of interest.

The study materials and methods
One of the ways to develop imaging technologies is to improve the methods of contour analysis, processing, and filtration of binary images.
In general, in our study, we used such theoretical and methodological research tools as the theory of analysis, meaningful interpretation, and the use of large volumes of complex structured data acquired from onboard sensory networks. To validate the proposed method, we applied methods of linear algebra, mathematical statistics, algorithm theory, methods of multi-agent modeling, mathematical modeling, and computer experiment.
The research was conducted in the field of computer vision, using methods of contour analysis of images, as well as methods and algorithms that we devised aimed at intelligent processing of aerial photographs. That would make it possible to make informed decisions in intelligent control systems, which are associated with the segmentation and separation of objects of interest or contextual search, as well as the formation of a context-dependent description of the image, etc.
The methodological basis for devising the image processing method for transport infrastructure monitoring systems includes the following: -threshold methods of segmentation by brightness, which are to set the threshold value by brightness and, relative to it, segment image pixels by two sets belonging to the object and background; -statistical methods of segmentation, based on the use of optimal statistical classifiers. Compared to deterministic methods, such methods are more time-consuming but they make it possible to steadily segment images under conditions of uncertainty. The disadvantage of such methods is that they are most effective only for processing color images; -contour segmentation. Contour methods are considered the basis of segmentation methods since they are resistant to variations in the brightness and contrast levels of images. The main for constructing contour methods of segmentation are boundary detectors, designed to detect the maximum pixels of images by contrast based on the use of masks. This approach has become widespread due to its simple program implementation. After segmenting the boundary pixels, the next step is to link the contours and build the boundaries of the images. In general, when building a boundary, there are two key problems -the appearance of gaps and the thickening of the boundary. However, under difficult conditions of low contrast, the methods are characterized by the high complexity of eliminating false boundaries with a low threshold, as well as the difficulty of eliminating boundary tears with an increased threshold [20]; -the histogram segmentation is based on the idea that in order to build a bimodal histogram of image brightness in an area selection, select the threshold at the minimum point, and segment the image with that threshold.
It should also be noted that images to be processed may contain noise. This may be caused by distortions made to the image by objects that actively reflect light, uneven air transparency during shooting, dust particles, equipment quality, etc. Therefore, additional pre-filtering of the image is required, for example, by pre-filtering the image, in which processing occurs by applying some operator sequentially to each point in the image.
The proposed method is one of the components of the information technology of dynamic monitoring of the state of transport infrastructure, which is currently being created on the basis of Ukrainian unmanned aircraft systems Spectator-M1 and Bereginya. To confirm the correctness of the choice of theoretical and methodological tools, a complex of testing works was carried out to support the declared information technology at all stages of development and implementation, including flight tests. During those operations, the material and technical base of the enterprise-developer of unmanned aircraft systems was used, including research stands, simulators, and UAVs necessary for the project. During the set of research and testing operations, such theoretical and methodological tools were used as the methods of semi-natural modeling, software engineering methods, flight measurement methods, data processing and analysis methods, etc.

1. Combined noise removal filter in a binary image
A colored object can emit light or absorb light. In the first and second cases, the color of the object is described differently, that is, different color models are used to describe it. Color settings can be expressed using a variety of color models. Most often, graphics packages employ the RGB color model, a color model that describes how to encode to reproduce a color using three colors, which are called the basic colors [21].
At the initial stage, the image is in a PNG format in the form of a two-dimensional matrix of 1,920×1,080, each element of which is a 3-element vector. In RGB space, each three-element vector sets the relative brightness of the point.
To solve the task of selecting contours and subsequent operations with the image, our method converts the original image (Fig. 1) from the RGB model to the HLS model (Hue-Lightness-Saturation). HSL (Hue, Saturation, Lightness) is a color model in which any color is determined by three characteristics. Such characteristics are a color tone (hue); saturation, which is part of pure color, without admixture of black and white; lightness, which is proximity to white [21]. After conversion, at the output there is a two-dimensional matrix of dimensionality 1,920×1,080, each element of which is a 3-element vector -this is the input array of numeric parameters with which the proposed method works.
Subsequently, filtering procedures are carried out that apply to the original image converted to the HSL color model.
In accordance with the task set, our method highlights the road surface and road markings in the image. Therefore, a working hypothesis is put forward that the road surface and road markings are "gray", monochromatic, and thus they differ from the "colored" roadside and other objects. That is why it is possible to adopt the saturation channel and reject saturated pixels, leaving only those that are closer to gray.
Thus, at the first stage, filtering is carried out "by color". That is, points (pixels) are selected with a given range in the saturation channel (S). As a result of this filtering, the method filters out too bright and dark points (pixels) (Fig. 2). That is, from the input array of numeric parameters, the element of each 3-element vector corresponding to channel S is distinguished. The result is a two-dimensional matrix of 1,920×1,080, each element of which is a byte -the relative value of the channel S of the image. Filtering is based on the following principle: if the value of an array element is greater than or equal to the given lower threshold value and less than or equal to the upper threshold value, the byte takes the value of unity. Otherwise, it accepts zero. In a given example, 0 was selected for filtration in channel S as the lower threshold value, and 25 was selected as the upper one.
In the next step, the Sobel operator is applied to the image to obtain the luminosity gradient value [22]. A two-dimensional matrix is also found, each element of which is a byte -the relative value of the channel L of the image.
We select the luminosity channel L of the image and build a luminosity value gradient (the amount of light in pixels). As a result, those parts of the image are highlighted, where there is a sharp transition between light and dark fragments of the image. It is the well-defined (outlined) dark elements on a light background (or vice versa) that make up what is called a contour in our work. The result of applying the Sobel operator at each point in the image is a two-dimensional vector whose components are the image luminosity derivatives horizontally and vertically. At each point in the image, the gradient vector is oriented towards the greatest increase in luminosity, and its length (magnitude) corresponds to the magnitude of the change in luminosity.
The process of deriving a luminosity gradient in an image is based on moving the masks of the Sobel operators over the image points.
Based on a two-dimensional matrix, each element of which is a byte -the relative value of the channel L of the image, four new arrays are built, the elements of which are the floating-point numbers, according to the following rules: -the elements of the array sobel_X are the value of the Sobel gradient for the neighborhood of the corresponding element of the output array along the X-axis (the value of the projection of the Sobel gradient onto the X-axis of the array in the neighborhood of a given element); -the elements of the sobel_Y are the values of the Sobel gradient for the neighborhood of the corresponding element of the output array along the Y-axis (the value of the projection of the Sobel gradient onto the Y-axis of the array in the neighborhood of a given element); -the elements of array M are the absolute value (length, magnitude) of the gradient; -the elements of the MD array are the direction (angle, azimuth) of the gradient.
The size of the Sobel Kernel operator, in fact, the neighborhood size of the point in which the gradient is built, was selected in a given example equal to 5.
Next, based on four arrays of the numeric parameters sobel_X, sobel_Y, M, MD, a new array of dimensions 1,920×1,080 is built, each element of which contains a byte with a value of unity or zero. The new array is built according to the following rules: for each element of the source array, the corresponding elements of sobel_x, sobel_y, m, md of arrays sobel_X, sobel_Y, M, MD are considered. The corresponding elements occupy the same position (row and column), in other words, they have the same coordinates.
The sobel_x, sobel_y, m values are normalized to the range of values [0, 255]. To this end, we determine the maximum value of the element in the matrix sobel_X, in the matrix sobel_Y, as well as the maximum value of the element in the matrix M.
After using the Sobel operator, the method carries out the procedure of filtering on the magnitude and direction of the gradient built along the L-channel of the image, comparing them with the lower and upper threshold values (Fig. 3).
In a given case, for the program implementation, the thresholds were selected as follows: -  As a result of certain preprocessing procedures, the original image is oriented so that the roadbed and marking elements are oriented horizontally or vertically (or at angles close to them). Therefore, the method selects horizontal or vertical contours. To this end, such places are found in the image where the luminosity gradient is significant only along one of the axes. The magnitude makes it possible to estimate how sharp the difference in "luminosity" is.
The direction (angle) of the gradient and the gradient projection value on the image axis make it possible to estimate the direction of luminosity drop, that is, the orientation of the contour. The next step is the procedure of filtering by the value of the projections of the gradient built over the L-channel, in projections onto the image axis, comparing it with the threshold value (Fig. 4).
Thus, as a result of the above operations, there are two sets of arrays of numeric parameters: -S-matrix with a dimensionality of 1,920×1,080, with values of zero or unity, that is, a mask on the S channel; -L-matrix with a dimensionality of 1,920×1,080, with values of zero or unity, that is, a mask by channel L.
Next, a new set of numeric parameters is built, a matrix of dimensionality 1,920×1,080, with values of zero or unity, that is, an image mask on channels S and L. The source element accepts a unity value if the corresponding elements of the S and L masks are zero, otherwise, the source element accepts a value of zero. Thus, an image mask is built -a composite filter. Fig. 5 shows a composite overlay of the above filters (red color -filtering "by color", green -filtering by magnitude and direction of the gradient, blue -filtering by gradient projection values).
The built "composite" filter, combining all three filtering methods described above, is used to select only those points of the image that meet the requirements or filter by the values of the gradient or filter projections in the direction and magnitude of the gradient. And, additionally, these points must meet the requirements of the filter by color. Next, the method works with the results of combined filtration.
To improve the quality and accuracy of work, in the next step the noise is filtered in a binary image.
The essence of filtering is that for each white pixel of an image, a certain aperture or an area of a point (pixel) is specified -a set of pixels in an image with a dimensionality of n×n. Next, the number of white points that are included in the aperture of each white point is calculated. The rule of deciding on the value of the element (pixel) of the original image reads like this: if the number of white points in the selected aperture is less than the set threshold value, the base pixel is defined as noise and removed from the image. In a given case, for the programmable implementation, the aperture of the noise filter is 13, the threshold value of the noise filter is 19. Fig. 6 shows the results of applying a mask based on a combined filter. In addition, a noise removal filter was additionally applied to the image.
Filtering an image is an important aspect when you select contours. The presence of unnecessary data in the image may affect the performance of the method. In turn, the loss of image details negatively affects the accuracy of the method. The proposed combined noise removal filter in a binary image makes it possible to clean the image of existing noise qualitatively. Our study results have demonstrated that applying the proposed filter makes it possible to remove up to 80 % of the noise and excess parts in the image. Such data are obtained by counting and comparing the number of white pixels in a binary image taken for noise before and after filtering.

2. A method for separating point cluster contours using histograms
Next, we consider the work of the method directly for the detection of road surface markings for further analysis of its condition.
The constructed image (Fig. 6) is a raster one. To build quantitative and qualitative estimates, as a rule, it is necessary to generate vector information: segments of object contours, boundaries of individual elements of the image (objects).
At the next stage of separating the boundaries of a cluster of points using histograms, the axis on which the histogram will be built subsequently and further operations will be performed is specified. To do this, we build histograms on the axes OX and OY of the image, which determine the number of white dots (pixels) in each column and in each line of the image, respectively. Next, N maximum and minimum values are selected for each built-in histogram and the average minimum and minimum value for the histogram is calculated. As the selected axis, the method distinguishes the one that has the largest difference between the average maximum and minimum value, that is, the histogram with the largest range of values. The next step in separating the boundaries divides the image into strips along the selected axis using the l-wide window. For example, for the image in Fig. 7, we selected the axis on which the histogram is constructedthe OY axis. Along the OX axis, a window of width l is set, for this window, a histogram is built along the OY axis. When analyzing the resulting histogram, one can highlight positions on the OY axis where objects of interest begin and end. Then the window is shifted by a distance of l/2 and the steps are repeated, starting with the construction of the histogram. The method works until it reaches the entire image (reaches the right edge of the image). As a result, in each strip of the window, objects of width l are obtained. Combining all these objects determines the final object of interest in the image Thus, in a given case, for the program implementation, the value of the width of strip l in which the histogram is built, is 120. And the histogram threshold is 25. The algorithm scans the constructed histogram: when the histogram threshold is exceeded, the algorithm determines the beginning of the object of interest in a given strip. If the histogram value falls below the threshold value, the algorithm captures the final contour of the object in a given strip. This determines the outline of the position of the object in a given strip.
The data obtained can be further used for quantitative and qualitative descriptions of the object of interest.
The highlighted zones characterize the features of road surface markings and can be further used to take and make informed decisions to maintain a satisfactory state of road surface markings, etc.
To prove the effectiveness of the result, Fig. 8 shows the results obtained using the current probabilistic method of Hough transform to the image in Fig. 6.
The results obtained from the Hough method are unsatisfactory and of little use for further processing since the selected segments do not reflect the qualitative and quantitative indicators of the objects of interest in the image. Fig. 8 demonstrates that the method finds some lines that are unsatisfactory for their further analysis.
To build quantitative and qualitative estimates, as a rule, it is necessary to generate vector information: segments of object contours, boundaries of individual elements of the image (objects). It is to solve this issue that the devised method of image processing is directed.

Discussion of results of the operation and application of the method of image processing in transport infrastructure monitoring systems
To analyze the advantages of the proposed results, it is advisable to compare them with a detailed study [23]. The authors of work [23] also solve the task of detecting the marking of the lane of road transport. The difference is that the images are acquired from the camera, which is located on the dashboard of the car. To improve the results of algorithm performance, the authors consider the image "from the height of a bird's flight", which is similar to acquiring an image from aboard a UAV. In [23], Hough transform and Kenny's operator are used, whose advantages and disadvantages have already been analyzed. Although the system proposed in [23] detects lanes, there are certain limitations of that algorithm. In particular, the algorithm is highly dependent on preprocessing an image to achieve marking accuracy. Preprocessing is the costliest stage of the algorithm in computational terms.
The proposed method of image processing for transport infrastructure monitoring systems differs in that it contains an operational image processing tool -a combined filter. It reduces the preprocessing time by about 50 % of the data listed in [23] on systems of the same configuration, even when working with a higher resolution image. This collectively gives an increase of 15 % in performance compared to the use of known methods of pre-filtering and image processing. The proposed approach differs from study [23], where, when marking is identified, the authors do not obtain indicators for qualitative or quantitative assessment of the detected marking and apply the methods of contour detection sensitive to noise in the image.
Our results are components of the existing information technology (which includes a set of methods, algorithms, and programs) for dynamic monitoring of the state of transport infrastructure using unmanned aircraft systems. This information technology is a tool for solving problems related to one of the priority areas -monitoring the quality of road works.
The development and improvement of unmanned aviation technologies and remote sensing technologies could ensure high-quality monitoring of the state of transport infrastructure at the nationwide level. The latest technologies should serve as a safe and highly efficient function of the transport system. We are talking about the functioning of the most important transport sectors of the state: road, railroad, and water transport, oil and gas pipelines, electricity networks, etc.
Unmanned aircraft systems are being successfully designed in Ukraine. One of the most promising technological advancements is the tactical unmanned aircraft complex "Spectator-M1". Intelligent information technology of dynamic monitoring of the state of transport infrastructure is arranged on the basis of this system. The proposed method was tested at the Meridian Open Joint Stock Company (Kyiv, Ukraine) during tests of information technology for dynamic monitoring of the state of transport infrastructure.
Our results could be further used for scientific and technical research in the field of dynamic monitoring technologies. The findings would make it possible to build new high-quality means of air dynamic monitoring of ground facilities using hardware of relatively low cost. It is also The limitations of the proposed method are that the image filtering algorithm is not universal to solve the task of monitoring any infrastructure. Sometimes there are minor errors in the construction of a histogram, characterizing road markings, on rounded sections of the road with a deterioration in the level of illumination.
Further development of this study involves increasing the level of versatility of the devised method for the possibility of its use for monitoring infrastructure of various types (for example, power lines, pipelines, railroad tracks, etc.).
According to our research, the devised method has several advantages and eliminates the disadvantages of known methods when tackling the problem of highlighting the contours of road markings on images acquired from a UAV or drone.

1.
A combined noise removal filter on a binary image has been developed, which, unlike existing ones, runs in the HSL color space. That makes it possible to clean the image of existing noise qualitatively. The results of our simulation showed that applying the proposed filter makes it possible to remove up to 80 % of the noise in the image.
2. The method of separating the contours of a cluster of points using histograms differs in that to solve the problem of monitoring the state of road markings, the selected segments reflect the qualitative and quantitative indicators of the objects of interest in the image. That makes it possible to solve the tasks at the level of a human expert in a given area and ensure sufficient reliability and accuracy of decision-making. In addition, during the tests, an increase in the performance of the method was registered, by at least 15 %, compared to the use of known methods of computer vision libraries.