DEVISING A METHOD FOR PROCESSING THE IMAGE OF A VEHICLE'S LICENSE PLATE WHEN SHOOTING WITH A SMARTPHONE CAMERA

Experimental studies were conducted on the processing of images of a vehicle's license plate. During the experimental studies, the license plate of a military vehicle (Ukraine) was considered. The original image was the color image of a vehicle. The results of experimental studies are given. A comparison of the quality of character recognition in a license plate has been carried out. It was established that the improved method that uses the combined results from streaming video frames works out efficiently at the end of the sequence. The improved method that employs the combined results from streaming video frames operates with numerical probability vectors.


Introduction
High-quality image processing aimed at recognizing a vehicle's license plate is relevant in security and video surveillance systems. Automatic license plate recognition systems are used to automate traffic control. Such systems have a wide range of applications: -in paid parking lots. The license plate is read twice. The parking time and the amount of payment are automatically determined. In addition, such a system can identify vehicles for which payment is not made; -access control -automatic passage to the protected area; -on toll roads. Identification of the license plate is carried out to determine the fare or for additional verification of the travel permit; -border control -a license plate is registered when entering the country and leaving it;

This paper reports an improved method for processing the image of a vehicle's license plate when shooting with a smartphone camera. The method for processing the image of a vehicle's license plate includes the following stages:
-enter the source data; -split the video streaming into frames; -preliminary process the image of a vehicle's license plate; -find the area of a vehicle's license plate; -refine character recognition using the signature of a vehicle's license plate; -refine character recognition using the combined results from frames in the streaming video; -obtain the result of processing.

Experimental studies were conducted on the processing of images of a vehicle's license plate. During the experimental studies, the license plate of a military vehicle (Ukraine) was considered. The original image was the color image of a vehicle. The results of experimental studies are given. A comparison of the quality of character recognition in a license plate has been carried out. It was established that the improved method that uses the combined results from streaming video frames works out efficiently at the end of the sequence. The improved method that employs the combined results from streaming video frames operates with numerical probability vectors. The assessment of errors of the first and second kind in processing the image of a license plate was carried out. The total accuracy of finding the area of a license plate by known method is 61 % while the improved method's result is 76 %. It has been established that the minimization of errors of the first kind is more important than reducing errors of the second kind. If a license plate is incorrectly identified, these results would certainly be discarded
with the license plate. After that, the found area is cut from the original image and considered separately. Normalization of the image of a vehicle's license plate involves geometric transformations, suppressing noise, changing brightness and contrast. Segmenting the characters on a license plate involves dividing the image into places for characters (selecting areas of individual characters). After that, the procedure of optical character recognition is carried out. As a result of this procedure, a line of symbols is built. The syntactic analysis procedure is performed to take into consideration the elements of the line with the symbols of the license plate. These elements may differ according to the standards in the countries of registration of vehicles. The advantage of [3] is a detailed description of the stages of recognition of a vehicle's license plate. The disadvantage of [3] is that only a list of recognition stages, not experimental studies, is given.
In [4], it is noted that to localize the area of a license plate in the image, methods are used that are based on binarization, contour selection, morphological processing of images. The method of localization of the area of the license plate [4] includes the following sequence of actions: 1. Convert the original color image to a halftone. 2. Image processing by Sobel (or Prewitt) operators. Generation of an image in which the brightness of pixels is equal to the gradient module at the corresponding point of the image.
3. Conduct image binarization. 4. Carry out operations of morphological closure with a rectangle as a structural element. 5. Define a contour in the image. 6. Define areas that are limited by contours. 7. Determine the area whose parameters correspond to a vehicle's license plate.
The advantage of [4] is to solve the problem of localization of a vehicle's license plate in the image. The main disadvantages of the method reported in [4] are the sensitivity of Sobel, Prewitt operators, as well as morphological operators, to all sorts of noises in the image (Gauss, speckle, "salt and pepper").
To localize the area of a vehicle's license plate, the Viola-Jones method can be used [5]. The essence of the method is to find the features in the image using Haar signs. Haar's feature consists of adjacent rectangular regions that are in the image. Then the pixel brightness is summed up in the areas and the difference is calculated between the sums. At the stage of detection of the predefined area, a window of a certain size is used, which moves around the image. Haar's feature is calculated for each area of the image. The presence or absence of an object in the window is determined by the difference between the value of the sign and the threshold. The advantage of [5] is the use of a priori information regarding the attributes of a vehicle's license plate in the image. The main disadvantages of [5] are: -there may be many rectangular objects in the image that look like a vehicle's license plate; -high complexity of calculations; -it is not effective in the absence of pronounced boundaries in the image (for example, a dusty license plate).
To determine the places of license plates in the image, the support vector machine (SVM) method is used [6]. The main advantages of [6]: -the method's learning can be reduced to the problem of quadratic programming. Such a problem has a single solution that can be effectively defined on large samples; -search for stolen vehicles. The system is located on the side of the road and controls the license plates of all vehicles. In real time, a comparison of license plates with the list of license plates of stolen vehicles is carried out. In determining such a vehicle, the system issues a message to the road police to stop such a vehicle; -control of speeding and travel to the prohibitive traffic light; -control of movement of vehicles. The classification of vehicles is carried out and their direction to different lanes of movement for the purpose of possible inspection, etc.
Stationary video cameras are used in existing systems of automatic recognition of license plates. The plate recognition distance is usually known and is tens or hundreds of meters. Video cameras are properly placed and configured, which greatly facilitates the recognition of vehicle plates. It is possible to use various kinds of auxiliary functions to acquire the highest quality image in heterogeneous lighting. The above makes it possible to use a sufficient level of lighting, avoid tilting the license plate, ensure automatic focusing, avoid reflecting light and the presence of shadow. In addition, the possibility of deformation of the license plate, blurring and fuzziness of the image is reduced as much as possible.
There are situations when it is necessary to carry out prompt recognition of the license plate of a vehicle. Such an event arises when patrolling as part of military, police, peacekeeping missions, performing tasks as part of mobile outfits for the protection of the state border, during anti-terrorist activities, etc. (for example, [1]). In addition, prompt recognition of a vehicle's license plate is relevant when imposing a state of emergency, curfew in case of natural disasters, riots, etc. (for example, [2]). In such situations, there is no possibility of using stationary video surveillance systems and, accordingly, existing systems for automatic recognition of license plates. Therefore, mobile video surveillance tools are typically used, including standard smartphones.
The main features of shooting streaming video with a smartphone camera are: -insufficient detail of the image; -limited use of different types of camera lenses; -the presence of vibrations of the smartphone camera; -limited size of the smartphone camera matrix; -limited focus and zoom modes of the camera; -lack of tracking mode for the object, especially when the object is approaching (lack of tracking automatic focus); -lack of implementation of the "bokeh" effect; -limited use of additional options (sharpening, sepia, determining white balance and fixing it, color filters, blurring some parts, etc.).
Using existing methods for processing the images of a vehicle's license plate when shooting with a smartphone camera is not effective. Therefore, it is a relevant task to devise a method for processing the image of a vehicle's license plate when shooting with a smartphone camera.

Literature review and problem statement
Paper [3] states that the task of recognizing a car number plate includes the stages of plate detection, normalization of the license plate image, segmentation of license plate symbols, character recognition. Detection of a vehicle's license plate involves localization in the image of the area -the position of an optimal hyper plane depends only on the small share of objects used for training (support vectors); -by introducing the kernel function, the method can be summarized for the case of nonlinear surfaces that are not separated.
The main disadvantage of [6] is that the standard method of support vectors is used only for binary classification.
For the case of several classes, the following approaches are most used [7,8]. In [7], it is proposed to build a set of classifiers and select the class at which the object is farther from the separating surface. In [8], it is proposed to construct a set of binary classifiers, and then select the class proposed by most of these classifiers. The advantages of [7,8] are the use of two stages for the classification of a license plate in the image. This improves the quality of classification. The disadvantages of [7,8] are the difficulty of finding the boundary between classes and the complexity of computation with an increase in the number of classes.
In [9], a method of histogram analysis of regions is proposed. The method assumes that the characters on the plate, the background and frame are contrasting. The advantage of [9] is the ease of operation of the histogram method. The disadvantage of [9] is the instability of the method to noise and mud. In addition, in the case of a large perspective (a large angle of inclination of the camera when shooting) or a small image of the car in comparison with the surrounding objects, defining the maximum projection is difficult.
As a rule, the selected area with the license plate has a relatively small size, includes the boundaries of the license plate, horizontal stripes of the bumper, and the radiator grille. Therefore, to determine the angle of rotation of the license plate frame, one can apply a method based on Hough transformation for lines [10]. The method from [10] involves the following: -determine the lines whose length is more than half the width of the license plate area; -construct a straight line from the average values of all points along the lines obtained; -calculate the angle between the resulting straight line and the horizon line.
Paper [11] considers the features of the practical application of Hough transformation to determining elements of urban infrastructure. Additionally, at the first stage, the use of a Canny method is proposed. The advantages of [10,11] are the use of simple geometric primitives, reducing computing costs. The disadvantages of [10,11] are the expediency of using Hough transformation in determining simple geometric primitives and the presence of a large number of "junk" objects.
In [12], a method of identification and recognition of license plates for streaming video is proposed. The method is based on a neural network with feedback. The a priori values of direct and bidirectional matching between sequential video streaming frames are properly combined with layer structures that are specifically designed to identify license plates. Study [12] provides a set of video frames for deep learning of the proposed network. The advantage of [12] is that during network training, data augmentation is performed based on the rotation of the image. The disadvantage of the method reported in [12] is the need to perform a multicycle adjustment of the internal elements of the network and the connection between them.
In [13], a method is proposed that makes it possible to recognize the license plates of Chinese vehicles. Recognition is performed using a deep learning architecture based on the License Plate Recognition Neural Network (LPR-Net). LPR-Net is a hybrid deep architecture. The advantages of [13] are that the architecture consists of a network of residual errors to obtain basic features, a multi-scale network to obtain multi-scale features, a regression network to determine the location of license plates and symbols, and a classification network for recognition. The disadvantage of the method reported in [13] is the low speed of learning in the LPR-Net training procedure.
Paper [14] proposes a single deep neural network that localizes license plates and recognizes letters simultaneously in one pass through the sample. That is, the entire network is taught from beginning to end. The use of this method avoids intermediate accumulation of errors, speed up the speed of image processing. The disadvantage of the method reported in [14] is problems during the preparation of the training sample, which are associated with the difficulty of finding a sufficient number of training examples.
In [15], a method based on the Rhys fractional operator is proposed. This method is used to increase the detail of edge information in license plate images. The proposed approach performs the operation of convoluting the fractional Riesz derivative over each input image, strengthening the force of the edge in it. However, to solve the task of identifying and recognizing license plates, it is necessary to use additional methods for detecting and recognizing a text.
In [16], a neural network is proposed to detect small and inclined license plates. The method is based on the application of the analysis of the ratio of a vehicle to the license plate. It is proposed to evaluate the local area around the license plate using the ratio between the vehicle and license plate. That makes it possible to significantly reduce the search area and identify very small license plates. The quadrangular frame of the license plate is determined by regression of the four corners of the license plate. The disadvantage of the method reported in [16] is that a given method works effectively only for small and inclined license plates.
In [17], a method is proposed, which is a set of algorithms such as a sliding window, a histogram of a directional gradient, and a method of support vectors. The image is processed by a sliding window method to find the position of the license plate. Attributes are acquired at each movement of the sliding window by using the method of support vectors and the method of a histogram of directional gradients. The advantage of the method reported in [17] is the high accuracy of identification of license plates in the image even of poor quality. The disadvantage of [17] is the high time costs, and as a result, the inability to detect license plates in real time.
In [18], it is proposed to use a cascading classifier of the Haar type to localize license plates, which is widely used to solve the problem of facial recognition. The advantage of [18] is the use of an already developed method to solve another problem (human facial recognition). The disadvantage of the method reported in [18] is the dependence of the result of work on the choice of positive and negative samples.
In [19], a vehicle's license plate recognition method is proposed, based on characteristic extreme areas (ER) and Boltzmann hybrid discriminatory limited machines (HDRBM). Rough identification of the license plate is performed by detecting the vertical edge, morphological operations, and various checks. Then the ER for specific characters is acquired as the character area of the candidate license plates. After the appropriate selection of ER, character segmentation and identification of license plates are achieved simultaneously. Finally, a standalone trained HDRBM template classifier is used to recognize characters. The advantage of the method reported in [19] is resistance to changes in light and weather conditions during the day. The disadvantage of [19] is the complexity of implementation and the presence of a limited trained template classifier.
In [20], a two-step approach to the localization of the license plate in difficult lighting conditions is proposed. In the first phase, the Faster R-CNN-based convolutional neural network algorithm is used to detect all the vehicles in the image. This is necessary to collect scalable information to determine the location of license plates. At the second stage, morphological operations are used to reduce areas that do not belong to license plates. The advantage of the method reported in [20] is the ability to recognize several license plates at the same time. The disadvantage of [20] is that the method only works with images in the HSI color space.
In [21], a method of identifying elements of urban infrastructure objects based on Radon integral vector transform and Hough transformation is proposed. The advantage of the method reported in [21] is the ability to use it to determine geometric primitives on complex images. The disadvantage of [21] is its computational complexity.
Given the possibility of representing the original image (video stream frame) with a set of optical signals, consider similar methods of signal processing in radio equipment (for example, [22]). In [22], a method of decomposition of superpositions of unknown pulse signals by adaptive spectral analysis of the second order is proposed. The method reported in [22] is effective in processing pulse signals in radio engineering systems. The method reported in [22] does not take into consideration the peculiarities of the formation of images of vehicles' license plates.
Papers [23,24] propose metaheuristic optimization methods for thematic segmentation of images of on-board optoelectronic surveillance systems. In [23], a method of artificial bee colony is analyzed in detail. Study [24] proposes the use of the method of artificial bee colony developed in [23] to isolate objects of interest in the optoelectronic image. The methods reported in [23,24] have proven effective for processing complex structured images. The disadvantage of methods reported in [23,24] is their computational complexity.
Paper [25] provides a general algorithmic basis for varieties of image processing methods based on ant algorithms. The list of applications for effective solution of computationally complex problems with the help of the proposed approach is given. The authors of [26] devised a method based on the ant algorithm [25] to determine the contours of objects on tonal aerospace images. The disadvantage of methods reported in [25,26] is the difficulty of implementation in processing images in real time.
In [27], based on the improved algorithms of an ant colony, a method for determining the flight routes of groups of unmanned aerial vehicles is proposed. The method reported in [27] is effective for solving nondeterministic polynominal (NP) -complex optimization problems. Such NP-complex optimization problems include image processing. However, the method reported in [27] cannot be used for the task of identifying and recognizing vehicle license plates due to the lack of consideration of the peculiarities of the formation of such images.
Our review of known methods for image processing has revealed that the processing of images of license plates when filmed by a smartphone camera has not been considered yet. Therefore, it is a relevant task to devise a method for processing the image of a vehicle's license plate when shooting with a smartphone camera.

The aim and objectives of the study
The purpose of this study is to improve a method for processing the image of a vehicle's license plate when shooting with a smartphone camera. This would make it possible to reduce errors of the first and second kind when determining the license plate of a vehicle.
To accomplish the aim, the following tasks have been set: -to define the main stages of the method for processing the image of a vehicle's license plate when shooting with a smartphone camera; -to process images from streaming video to determine the license plate of a vehicle when shooting with a smartphone camera; -to conduct a comparative assessment of the quality of image processing of a vehicle's license plate by the known and devised methods when shooting with a smartphone camera.

The study materials and methods
During the study, the following research methods were used.
When defining the main stages of the method for processing the image of a vehicle's license plate when shooting with a smartphone camera: -methods of image processing theory; -methods of probability theory and mathematical statistics; -methods of morphological analysis; -methods of system analysis; -methods of differential calculus; -methods of digital signal processing; -mathematical apparatus of matrix theory. When processing images from streaming video to determine the license plate of a vehicle when shooting with a smartphone camera: -methods of image processing theory; -methods of morphological analysis; -methods of mathematical modeling; -methods of differential calculus; -methods of digital signal processing. When assessing errors of the first and second kind while processing the image of a vehicle's license plate when shooting with a smartphone camera: -methods of probability theory and mathematical statistics; -mathematical apparatus of matrix theory; -methods of mathematical modeling. When validating the proposed solutions, analytical and empirical methods of comparative research were used.
During the study, the following restrictions and assumptions were adopted: -the image of a license plate of a military vehicle (Ukraine) is considered; -OPPO Reno 5 Lite smartphone is used as a smartphone; -video shooting resolution -1080P/30k/s; -video streaming was divided into frames; -the streaming video frame is represented by the original image in the Red-Green-Blue color space (RGB); the impact of noise on the image is not taken into consideration; there is no turning the frame of a vehicle's license plate; -the vehicle approaches the person carrying out the filming; lighting conditions -natural background, natural light; -shooting angle -minor projective distortions, from afar, without turning.

1. The main stages of the method for processing the image of a vehicle's license plate
The main stages of the method for processing the image of a vehicle's license plate are shown in Fig. 1.
The method of processing the image of a vehicle's license plate includes the following stages: 1. Enter the source data -streaming video, characteristics of the smartphone camera.
2. Video streaming framing (acquire the original image (a streaming video frame) f (X)). X(x, y) -pixel coordinates in the streaming video frame (source image).
3. Pre-process the image of a vehicle's license plate (video streaming frame).
4. Find the area of a vehicle's license plate.
5. Recognize the symbols in a vehicle's license plate. 6. Refine character recognition using the signature of the vehicle registration plate.
7. Refine character recognition by combining the results from streaming video frames.
8. Obtain the result of processing. Let us consider in detail the main stages of the method for processing the image of a vehicle's license plate.

1. 1. Preprocessing the image of a vehicle's license plate
Classical preprocessing of the image of a vehicle's license plate involves the transition from the original color image to the image in grayscale [28,29]. This could lead to the loss of useful information depending on the color space of the image representation. Note that the text (symbols) on the license plate of a vehicle is contrasting with the background on the license plate. Therefore, unlike known methods, at the stage of preliminary processing of the image of a vehicle's license plate, we shall conduct a pixel-to-pixel multiplication of the brightness of the channels of the color space of the representation of the original image. For example, to image a vehicle's license plate acquired in the RGB color space, this operation can be represented by expression (1): where с(Х) is the brightness of the pixel with X coordinates in one of the channels of the output image f(X) in the RGB color space; g(Х) is the original image f(X) in grayscale.
In addition, in contrast to known methods, we shall apply the morphological operation imtophat T w (g (X)) to the image g(Х). The imtophat operation calculates the morphological discovery of the image (using the imopen morphological operation), and then deducts the result from the original image (expression (2) [30]): where b(X) is the structural element of the image g(X); • -morphological operation of opening (imopen) [30]. Thus, in contrast to known methods, the stage of pre-processing of a vehicle's license plate additionally involves the following: -pixel-to-pixel multiplication of the brightness of the channels of the color space of the representation of the original image; -application of morphological operation imtophat T w (g(X)) to the image g(Х).

1. 2. Finding the region of a vehicle's license plate
Finding the region of a vehicle's license plate in the original image g(Х) is performed during the following sequence of operations: -determine the membership function of the set of regions of a vehicle's license plate; -search for regions that satisfy the specified membership function.
For a set of regions of the license plate, the membership function can be represented by expression (3): where i=1, 2, …, n; ξ is the vector of values of informative features of the reference region; ξ is the vector of values of informative signs of the analyzed region; V is the vector of weights of elements of informative features; th is the threshold value; n is the number of informative signs; ∆ξ is the ratio of deviation of one value of the informative sign from another.
The function ( ) (3)) takes a value of 1 if the region under consideration belongs to the region of a vehicle's license plate. Value 0 otherwise.
Determining the ratio ∆ξ is a task that is difficult to formalize. For various informative signs, ∆ξ may vary. The selection of the threshold value th and the values of vector V can be solved by conducting experimental studies.
The informative signs of the region of a vehicle's license plate is to be understood as a set of numerical (formalized) characteristics. Informative signs regarding the analyzed region include the area, perimeter, shapes, dimensions, compactness of the region, the presence of "high" contrast and for vehicles in motion -changing the coordinates of the region in time.
If it is necessary to impose some restrictions on the elements of the set of informative signs, it is necessary to set the following values: -maximum and minimum permissible values of the area, perimeter, dimensions, compactness of the region; -permissible distortions of the shape of the region; -threshold value of contrast; -maximum and minimum permissible values of the rate of change of coordinates of the region in time.
Consequently, the values of informative signs should be included in the interval of permissible values or not exceed the threshold value.
Restrictions on the elements of a set of informative signs determine the region of permissible values of informative signs. The interval thresholds for the area, perimeter, dimensions, and compactness of the region depend on the remoteness of the camera from the vehicle and on the size of the image. The restriction on distortion of the shape of the region depends on the angles between the camera and the plane of the license plate. The contrast threshold depends on the illumination of the license plate area. The speed limit for changing the coordinates of the area over time depends on the possible speed of movement of the vehicle.
When finding the region of a vehicle's license plate, we shall assume that the smartphone camera shoots in such a way that the perspective distortions of the image are minimal. If there are promising distortions in the image for their compensation, it is advisable to find the homographic matrix in advance [28]. Hereafter, we shall assume that these prospective distortions are small or compensated.
In this case, to find the region of a vehicle's license plate, we shall use the search for a sample to the maximum of the cross-correlation function (expression (4) [31]): where f et (x, y) is the reference image; f t (x, y) -current image; (i, j) -coordinates of the pixel of the image; i=1, …, N is the number of columns of the image; j=1, …, M is the number of image lines; (N×M) is the image size.
The search sample is based on the template of a vehicle's license plate. It is enough to set rectangular areas for numbers and letters.
The peculiarity of finding the region of a vehicle's license plate is the fact of movement of the vehicle. Therefore, the size of the license plate would also change. Therefore, in contrast to known methods, for the effective operation of the correlator (expression (4)), it is necessary to choose the appropriate scale of the search sample. To do this, we shall find a maximum of the cross-correlation function (expression (4)) for different values of the scale s, respectively, scale the sample t=t(s) and choose the optimal scale s opt , which gives the maximum value of the cross-correlation function (expression (5)): Thus, the stage of finding the region of a vehicle's license plate according to the sample has the following advantage. It makes it possible to identify the position of each symbol of the license plate, even if the license plate in the image is only partially present. In the future, this will be used in the development of methods for optical recognition of the characters of the license plate.

1. 3. Recognition of vehicle license plate symbols
Recognition of the symbols of a vehicle's license plate will be carried out by searching the sample employing a maximum of the cross-correlation function -expression (4) [31]. In this case, the images of symbols used as samples must be determined.
Character images are scaled according to the optimal scale value that was determined at the previous stage to search for the license plate's region -expression (5).
After that, we shall carry out binarization according to Otsu [28]. Then we shall calculate the cross-correlation function between each symbol of the image of the found region of the license plate and each symbol of the sample binarized according to Otsu. The values of the maxima will be stored in the form of vectors; the maximum value, in this case, corresponds to the recognized symbol.
The above stages of the method for processing the image of a vehicle's license plate may not work correctly in the case of distorting factors. Such factors are, for example, the ( ) presence of dirt, lighting effects, significant prospective distortions, etc.
To improve the quality of character recognition of the license plate, we shall take into consideration the signature of the license plate. The signature of a license plate is to be understood as the information about which symbol (letter or number) is on each sign. This makes it possible to narrow the recognition alphabet and improve the quality of recognition.
In practical applications, when recognizing a vehicle's license plate, work is carried out not with a single image but with their sequence (or video). In such images, a vehicle approaches the camera (or moves away from it). In this case, one can improve the quality of recognition by taking into consideration the results of processing other images of this vehicle.
Thus, a method has been devised for processing the image of a vehicle's license plate. In contrast to known methods, our method additionally involves: -pixel-to-pixel multiplication of the brightness of the channels of the color space of the representation of the original image; -application of the morphological operation imtophat T w (g(X)) to the image g(Х); -search for a sample based on a maximum of the cross-correlation function; -taking into consideration the movement of the vehicle (selection of the appropriate scale of the search sample); -taking into consideration the signature of a vehicle's license plate; -taking into consideration the results of processing other images of this vehicle.

2. Processing video streaming images to define a vehicle's license plate
During experimental studies, the following restrictions and assumptions have been adopted: -the image of a license plate of a military vehicle (Ukraine) is considered; -OPPO Reno 5 Lite smartphone is used as a smartphone; -video shooting resolution -1080P/30k/s; -video streaming was divided into frames; -the streaming video frame is represented by the original image in the Red-Green-Blue color space (RGB); the impact of noise on the image is not taken into consideration; there is no turning the frame of a vehicle's license plate; -the vehicle approaches the person carrying out the filming; lighting conditions -natural background, natural light; -shooting angle -minor projective distortions, from afar, without turning.
In the experimental studies, we shall consider the license plate of a military vehicle (Ukraine) (Fig. 2). Dimensions of the license plate in Fig. 2 are given in centimeters.
The original image to be considered is the color image (Fig. 3).

Fig. 3. Original color image (streaming video frame)
This is an image of a car with a military license plate (Ukraine). This is a color image (frame from streaming video) in the RGB color space. The image size is (600×800) pixels.

2. 1. Preprocessing the image of a vehicle's license plate
Preliminary processing of a vehicle's license plate additionally involves the following: -pixel-to-pixel multiplication of the brightness of the channels of the color space of the representation of the original image (expression (1)); -application of the morphological operation imtophat T w (g(X)) to the image g(Х) (expression (2)).
The results of preprocessing the original image (Fig. 3) are shown in Fig. 4. a b Fig. 4. Results of preliminary processing of the original image (Fig. 3): a -the result of the pixel-to-pixel multiplication of channels in the RGB color space; b -the result of using the morphological operation imtophat Fig. 4 shows that when applying the morphological operation imtophat T w (g(X)) to the image g(Х) (expression (2)), the (9×9) size of the square window was chosen. Note that the application of the imtophat operation is effective if the text is lighter relative to the background (as in Fig. 3). If the text is darker than the background (for license plates of other types), one must use a similar operation imbothat [30]. The imbothat operation calculates the morphological closure of the image (using imclose), and then deducts the result from the original image [30].

2. 2. Finding the region of a vehicle's license plate
A sample of the search based on a maximum of the cross-correlation function (expression (4)) to be used is the following sample (Fig. 5).
a In Fig. 5, red color marks rectangular regions for numbers and letters. Fig. 6 provides an example of the surface of the cross-correlation function for the image of a vehicle's license plate (Fig. 4, b) and the sample (Fig. 5, b).
The surface in Fig. 6 is calculated from expression (4), red color indicates the maximum point of the cross-correlation function.
As noted above, as a result of the movement of the vehicle, the dimensions of the license plate would also change. Fig. 7 shows the dependence of a maximum of the cross-correlation function m on scale s. Red color in Fig. 7 marks the maximum point calculated from expression (5) that corresponds to the optimal scale of the sample (s opt =0.48).  The result of finding the region of a vehicle's license plate in Fig. 8 is overlaid with an optimally scaled search pattern.

2. 3. Recognition of symbols in a vehicle's license plate
The images of symbols used as samples are shown in Fig. 9.
The images of license plate characters are shown in Fig. 10.
In Fig. 10, the binarization of the image of a vehicle's license plate was carried out according to the Otsu method [28]. Plots of cross-correlation functions for the characters of a license plate are shown in Fig. 11-16.  (Fig. 4, b) and the sample (Fig. 5, b) 0  Fig. 16. Plot of the cross-correlation function for the symbols in a license plate; a maximum corresponds to symbol "1" In Fig. 11-16, red circles mark the points of maxima corresponding to the found symbols. In Fig. 11, a maximum corresponds to symbol "3". In Fig. 12, a maximum corresponds to symbol "7". In Fig. 13, a maximum corresponds to symbol "2". In Fig. 14, a maximum corresponds to symbol "7". In Fig. 15, a maximum corresponds to symbol "B". In Fig. 16, a maximum corresponds to symbol "1".

2. 4. Refining symbol recognition using the signature of a license plate
The above stages of the method for processing the image of a vehicle's license plate may not work correctly in the case of distorting factors. Such factors are, for example, the presence of dirt, lighting effects, significant prospective distortions, etc.
To improve the quality of symbol recognition in a license plate (for example, in the presence of dirt, lighting effects, significant perspective distortions), we shall take into consideration the signature of a license plate. Thus, for example, for a license plate of the type shown in Fig. 2, the signature is demonstrated in Fig. 17. Fig. 17. Signature of a license plate of the type shown in Fig. 2 In Fig. 17, symbol "8" denotes an arbitrary number from the set "0123456789", symbol "A" denotes an arbitrary letter from the set "ABCEHIKMOPTX".
Thus, the search for the region of a license plate is carried out according to the general template T, and, to determine whether this symbol is a number or letter, special templates T 8 and T A are used, respectively (Fig. 18).
At the same time, each symbol (binary image b) refers to numbers or letters, depending which template it fits. That is, the condition is checked (expression (6)): and and .

A x y
x y x y x y b T b T > ∑∑ ∑∑ (6) Note that the condition (expression (7)) is always met: Fig. 19, 20 provide an example of incorrect recognition and its clarification using the signature of a license plate. . An example of incorrect symbol recognition and its clarification using the signature of a license plate: a -a plot of the maxima of the cross-correlation functions of symbol "B" recognition for the set "0123456789ABCEHIKMOPTX"; b -a plot of the maxima of the cross-correlation functions of symbol "B" recognition for the set "ABCEHIKMOPTX" Fig. 19, a shows the original image, for which the above-described preliminary processing is carried out (Fig. 19, b, c), and a binarized region of the license plate is found (Fig. 19, d). Due to the easy blurring of the original image, the maxima of the cross-correlation functions of recognizing symbol "B" for the set "0123456789ABCEHIK-MOPTX" (Fig. 20, a) are close to symbols "1" and "B". In this case, the value for symbol "1" is greater, so symbol "B" is recognized as "1". However, if one narrows the recognition alphabet to the symbol set "ABCEHIKMOPTX", then this symbol is recognized correctly as symbol "B" (Fig. 20, b).

2. Refining symbol recognition by combining video streaming frame results
To improve the quality of recognition of a vehicle's license plate, it is proposed to average the obtained vectors of the probability of recognition p in all previous images (expression (8)): where t is the image number in the frame sequence; p t is the vector of probability of symbol recognition (normalized vector of the maxima of cross-correlation functions). The length of the vector p t is 10 for numbers, and 12 -for letters for the set "0123456789ABCEHIKMOPTX"; t p is the averaged vector of probability of symbol recognition.

3. Assessment of the quality of processing the image of a license plate by the known and devised methods
To compare the quality of symbol recognition in a license plate, we shall consider the following methods: -the standard method for optical symbol recognition systems Tesseract [32], which is an integral part of most vehicle license plate recognition systems [33]; -an improved method of symbol recognition using the signature of a license plate; -an improved method using combining the results of streaming video frames.
The results of recognition of the symbols in a license plate by various methods are given in Table 1.
Our analysis of Table 1 reveals that the improved method using combining the results of streaming video frames works qualitatively at the end of the sequence. There, despite the partial visibility of a license plate, the correct result is obtained. An additional and defining advantage of the improved method using combining the results of streaming video frames is that the number is most accurately determined immediately after the sequence is completed. Note that to obtain results using Tesseract and the improved method of symbol recognition using the signature of a license plate, it is necessary to define an additional criterion for selecting the correct value from the sequence. It should also be noted that Tesseract and the improved method of symbol recognition using the signature of a license plate require post-processing of symbols sequences. The improved method using combining the results of streaming video frames works with numerical probability vectors. This is simpler and makes it possible to directly use methods such as averaging, Bayesian approach, etc. While only rank statistics (e. g., median) are available for symbols sequences, and the use of other methods is not obvious and requires their redefinition. Table 1 Comparing the license plate symbol recognition results by different methods (correct value is "3727B1") Method Tesseract Improved method of symbol recognition using the signature of a license plate Improved method using the combination of video streaming frame results 1 ***7B1 ***7B1 ***7B1 2 ***7B1 ***7B1 ***7B1 To assess the quality of finding the region of a license plate, it is proposed to use such well performing indicators as errors of the I and II kind [11]. Errors in determining the region of a license plate of the І (α 1 ) and ІІ (β 2 ) kinds are determined by the criterion of maximum likelihood [11]. The criterion of maximum likelihood derives from the generalized criterion of minimum average risk [11]. Errors in determining the region of a license plate of the І (α 1 ) and ІІ (β 2 ) kinds are calculated from expressions (9), (10), respectively [11]: where S 1 (g(X)) is the plane of the image region, which is mistakenly assigned to the region of a license plate; S 2 (g(X)) is the plane of the image region without the license plate of the original image g(Х); S 3 (g(X)) is the plane of the correctly defined region of a license plate; S 4 (f(X)) is the plane of the license plate region in the original image g(Х).
To determine the errors in determining the region of a license plate of the І (α 1 ) and ІІ (β 2 ) kinds, 70 consecutive frames of the video were processed, in which the car completely drove past the camera. Processing was carried out by two methods: -the standard method Tesseract for optical symbol recognition systems (method 1); -the improved method using combining the results of streaming video frames (method 2).
The results of the calculation of errors in determining the region of a license plate of the І (α 1 ) and ІІ (β 2 ) kinds are given in Table 2. Table 2 The results of the calculation of errors in determining the region of a license plate of the І (α 1 ) and ІІ (β 2 ) kinds When calculating errors in determining the region of a license plate of the І (α 1 ) and ІІ (β 2 ) kinds (Table 2), it was believed that the license plate is absent if the maximum value of the cross-correlation function for any scale is less than the specified threshold. The value of the threshold is taken equal to 0.25. As a null hypothesis, it is accepted that the license plate is present in the image (at least partially), the alternative hypothesis is that the license plate is absent.
Thus, the total accuracy of finding the region of a license plate by method 1 is 61 %, by method 2 -76 %.
It should also be noted that minimizing the number of errors of the first kind is more important than reducing the number of errors of the second kind. If the license plate is incorrectly recognized, these results would definitely be discarded at the symbol recognition stage.

Discussion of the study results on devising a method for processing the image of a vehicle's license plate
The method of processing the image of a vehicle's license plate when shooting with a smartphone camera has been improved. The main stages of the method for processing the image of a vehicle's license plate (Fig. 1) are as follows: -enter source data -streaming video, characteristics of the smartphone camera; -split video streaming in frames; -preliminary process the image of a vehicle's license plate; -find the region of a vehicle's license plate; -refine symbol recognition using the signature of a vehicle's license plate; -refine symbol recognition using the combined results of streaming video frames; -obtain the result of processing. In contrast to known methods, the stage of pre-processing of a vehicle's license plate additionally involves: -pixel-to-pixel multiplication of the brightness of the channels of the color space of the representation of the original image; -application of the imtophat morphological operation to the original image.
It is taken into consideration that the peculiarity of finding the region of a vehicle's license plate is the fact of movement of the vehicle. Therefore, in contrast to known methods, for the effective operation of the correlator (expression (4)), it is proposed to choose the appropriate scale of the search sample. To do this, we find a maximum of the cross-correlation function (expression (4)) for different values of the scale s, respectively, the sample t=t(s) is scaled and the optimal scale s opt is selected, which produces the maximum value of the cross-correlation function (expression (5)).
The method of recognizing the symbols of a vehicle's license plate has been improved. Recognition of the symbols in a vehicle's license plate is carried out by searching the sample based on a maximum of the cross-correlation function (expression (4)). Symbol images are scaled according to the optimal scale value that was determined at the previous stage to search for the license plate region (expression (5)). After that, binarization is carried out according to Otsu. Then the cross-correlation function is calculated between each symbol of the binarized image of the found region of a vehicle's license plate and each symbol of the sample. The value of the maxima is stored in the form of vectors, the maximum value corresponds to the recognized symbol.
To recognize a license plate under the conditions of distorting factors, the method of recognizing the symbols of a license plate has been improved, taking into consideration the signature of the license plate. This makes it possible to narrow the recognition alphabet and improve the quality of recognition. The method of license plate recognition by taking into consideration the results of processing other images of this vehicle has also been improved.
Thus, a method for processing the image of a vehicle's license plate has been devised. In contrast to known methods, the method additionally involves: -pixel-to-pixel multiplication of the brightness of the channels of the color space of the representation of the original image; -application of the morphological operation imtophat T w (g(X)) to the image g(Х); -search using a sample based on a maximum of the cross-correlation function; -taking into consideration the movement of the vehicle (selection of the appropriate scale of the search sample); -taking into consideration the signature of a vehicle's license plate; -taking into consideration the results of processing other images of this vehicle.
Experimental studies were conducted on the processing of images of a license plate.
During experimental studies, the license plate of a military vehicle (Ukraine) was considered (Fig. 2). As a source image, a color image was considered (Fig. 3). The results of preprocessing of the original image (Fig. 3) are shown in Fig. 4. The result of finding the region of the license plate is shown in Fig. 8. Fig. 19, 20 provide an example of incorrect recognition and its clarification using the signature of the license plate.
A comparison of the quality of symbol recognition of the license plate was carried out. To compare the quality of symbol recognition of the license plate, the following methods were considered: -the standard method Tesseract for optical symbol recognition systems; -the improved method of symbol recognition using the signature of a license plate; -the improved method using the combined results of streaming video frames.
The results of recognition of the symbols in a license plate by various methods are given in Table 1. Our analysis of Table 1 reveals that the improved method using the combined results of streaming video frames works qualitatively at the end of the sequence. There, despite the partial visibility of the license plate, the correct result is obtained. An additional and defining advantage of the improved method using the combined results of streaming video frames is that the license plate is most accurately determined immediately after the sequence is completed. Note that for results obtained using Tesseract and the improved method of symbol recognition using the signature of a license plate, it is necessary to determine an additional criterion for selecting the correct value from the sequence. It should also be noted that Tesseract and the improved method of symbol recognition using the signature of a license plate require post-processing of symbol sequences. The improved method using the combined results of streaming video frames works with numerical probability vectors. This is simpler and makes it possible to directly use methods such as averaging, Bayesian approach, etc. While only rank statistics (e.g., median) are available for symbol sequences, and the use of other methods is not obvious and requires their re-definition.
The assessment of errors of the first and second kind in processing the image of a license plate was carried out. Errors in determining the region of a license plate of the І (α 1 ) and ІІ (β 2 ) kinds are calculated from expressions (9), (10), respectively. The results of the calculation of errors in determining the region of a license plate of the І (α 1 ) and ІІ (β 2 ) kinds are given in Table 2. The total accuracy of finding the region of a license plate by method 1 is 61 %, by method 2 -76 %. It should also be noted that minimizing the number of errors of the first kind is more important than reducing the number of errors of the second kind. If the license plate is incorrectly recognized, these results would definitely be discarded at the symbol recognition stage.
The following limitations and assumptions are inherent in the current study: -the image of a license plate of a military vehicle (Ukraine) is considered; -OPPO Reno 5 Lite smartphone is used as a smartphone; -video shooting resolution is 1080P/30k/s; -video streaming was divided into frames; -the streaming video frame is represented by the original image in the Red-Green-Blue color space (RGB); the impact of noise on the image is not taken into consideration; there is no turning the frame of a vehicle's license plate; -the vehicle approaches the person carrying out the filming; lighting conditions -natural background, natural light; shooting angle -minor projective distortions, from afar, without turning.
Further research should be directed to the development of methods that could process images that show several vehicles.

Conclusions
1. The method of processing the image of a vehicle's license plate when shooting with a smartphone camera has been improved. The main stages of the method for processing the image of a vehicle's license plate are as follows: -enter the source data; -divide video streaming into frames; -preliminary process the image of a vehicle's license plate; -find the region of a vehicle's license plate; -refine symbol recognition using the signature of a vehicle's license plate; -refine symbol recognition using the combined results of streaming video frames; -obtain the result of processing. 2. Experimental studies were conducted on the processing of images of a license plate. During experimental studies, the license plate of a military vehicle (Ukraine) was considered. As a source image, a color image was considered. The comparison of the quality of symbol recognition of a license plate was carried out. It has been established that the improved method using the combined results of streaming video frames works qualitatively at the end of the sequence. The improved method using the combined results of streaming video frames works with numerical probability vectors.
3. The assessment of errors of the first and second kind in processing the image of a license plate was carried out. The total accuracy of finding the region of a license plate by method 1 is 61 %, by method 2 -76 %. It should also be noted that minimizing the number of errors of the first kind is more important than reducing the number of errors of the second kind. If the license plate is incorrectly recognized, these results would definitely be discarded at the symbol recognition stage.