ACCURACY ASSESSMENT OF MARKER RECOGNITION USING ULTRA WIDE ANGLE CAMERA

Modern devices that support augmented reality technology are widely used in various fields of human activity, including medicine. Head mounted displays may provide an attractive alternative to traditional surgery navigation systems because allow users to stand at the first point of view and interact with objects in their surroundings naturally. Thus, the object of research in this study is recognition accuracy of fiducial markers in zones where ultra-wide angle camera distort the most. This is motivated by the need to increase user workspace for interaction with markers compare to the workspace provided with such popular augmented reality device as Microsoft HoloLens 2. In this study, the recognition accuracy is evaluated using ArUco square markers with taking into account different marker sizes and their positions in the camera view space. The marker positions include the center of the camera view space as well as such zones where lenses distort the most as top left, top right, bottom left, and bottom right corners. Obtained results show that recognition accuracy is good enough to be applicable for surgical navigation and failures referred to the distortion occurs are available in less than 0.2 % of all cases. This gives a possibility to increase workspace for interaction with markers compare to the Microsoft HoloLens 2. At the same time, the workspace for interaction could not reach the actual view space of the camera since recognition fails in cases where marker’s body is partially visible in the captured image (i. e., marker position is at the image boundaries).


Introduction
The modern state of augmented reality (AR) technology allows it usage in various fields of human activity, including medicine.AR makes it possible to interact with virtual objects in the real world.In this way, AR technology is more attractive compare to virtual reality when used in applications for medical visualization [1].
The most widely used cases of AR technology in medicine are the visualization of the patient's state indicators during examination and the visualization of surgical instrument positions during surgery.
For mentioned cases, the output can be provided into medical card helmet, interactive MRI scan, and any other device with AR technology on board.Conventional smartphone, tablet, and even computers with plugged camera can be used as AR devices to superimpose virtual objects onto physical objects in real space.However, the most convenient in case of visualization for medical purposes is usage of special helmets or head mounted displays.They do not take up hands and the imposition of information on the user's field of view (FoV) allows users to immerse fully in the process of interaction with augmented objects [2,3].
The most commercially successful AR device is the Microsoft HoloLens 2 (Microsoft Corporation, USA) due to the no need to use additional equipment, its comfortable ergonomics, wide range of provided features, and variety of tools for software development for this device [3].Despite all the advantages, Microsoft HoloLens 2 has limitations.Both design flaws of device and drawbacks of AR technology itself make the Microsoft HoloLens 2 ineffective to be applicable in certain scenarios.
Tracking method used in Microsoft HoloLens 2 is insideout tracking.Only those cameras and sensors that are built into the device itself provide information on the orientation of the device in the real world.In order to provide update of spatial relationships of virtual and real objects, such method needs information from the tracking of fiducial markers attached in the real scene.Fiducial markers provide unambiguous identification of object position and pose in three-dimensional space [4,5].
According to the manufacturer information, the monoscopic camera of Microsoft HoloLens 2 covers 52 degrees [6], while the FoV of user, excluding the rotation of the eyes and peripheral vision, is 114 degrees [7].In practice, usage of mentioned monoscopic camera results in a small area beyond which interaction with virtual objects are not possible.Presence of two pairs of stereoscopic cameras on the glasses sides could solve the issue with a small FoV.However, stereoscopic cameras of Microsoft HoloLens 2 TECHNOLOGY AUDIT AND PRODUCTION RESERVES -№ 3/2(65), 2022 ISSN 2664-9969 have a lower resolution of the light-sensitive matrix compared to monoscopic camera.This leads to obtaining of a lower level of recognition accuracy of fiducial markers [8,9].
Of the factors mentioned above, usability of such augmented reality-based system as Microsoft HoloLens 2 will benefit from an increased FoV and achieved a good enough level of recognition accuracy of fiducial markers.
Ultra-wide angle camera with high resolution can be a solution to increase FoV.Such a camera has a lens whose focal length is shorter than 24 mm in full-frame equivalent FoV.Depending on the shape and location of the lens group, the angle of view can range from 52 to 180 degrees (Fig. 1).
At the same time, usage of ultra-wide angle cameras provides images that usually suffer from distortions.Distortion of the geometry and proportions of the objects at the image can affect the accuracy of recognition [10,11].However, these defects can be corrected.The most common way to get rid of distortions is to calibrate the internal parameters of the camera with the help of calibration boards [12].The disadvantages are that calibration should be performed individually for each camera, requires additional time and availability of appropriate software.
Therefore, the object of research in this study is recognition accuracy of fiducial markers in AR-based system applicable for surgical navigation.
The aim of this research is to analyze a recognition accuracy of fiducial markers in zones where ultra-wideangle camera distort the most with taking into account different marker sizes.

Research methodology
In this study, binary square fiducial markers were used as visual markers to be recognized.The main advantage of such markers is that camera pose can be obtained with only one marker usage [13].Moreover, binary square fiducial markers are widely used in medical research and practice, as they are simple, highly reliable, and multifunctional solution [14][15][16].
ArUco library is an open source popular library for detection of square fiducial markers.ArUco library is an open source popular library for detection of square fiducial markers [17].ArUco library was chosen as it has high accuracy of recognition on all three axes of the Cartesian plane and faster than similar tools for markers detection.This choice also provides compliance with the hardware capabilities of AR devices (i.e., most of them are not equipped with any sensors other than cameras).
The ArUco library includes several dictionaries containing sets of tokens with different numbers of bits.Having a smaller bit size helps to identify markers better for cases when marker size in image is too small, but a larger bit size can help to get position in three-dimensional space more accurately [13].
Different physical sizes and number of bits of markers were used in this study to assess their possible impact on the recognition accuracy (Fig. 2).Experiments were conducted with markers that have 4×4 bits and 3×3 cm physical sizes (Experiment A), 4×4 bits and 5×5 cm physical sizes (Experiment B), 7×7 bits and 5×5 cm physical sizes (Experiment C).ISSN 2664-9969 In these experiments, the ultra-wide angle camera was represented by the 8MP ultra-wide angle camera of Xiaomi Redmi Note 8 Pro (Xiaomi Inc., China).The camera properties are: f/2.2 aperture, 13 mm focal length (FoV is 120 degrees), 1/4.0″diagonal light-sensitive matrix, 1.12 μm pixel size.
Distance from the camera position to the markers position was restricted to 70 cm.This value was taken from [18].It is based on the assumption that interaction with the object to which the markers are applied is constrained by anthropometric and biomechanical limits (i.e., the maximal distance an average individual is able to reach forward from shoulder to fingertip).
In each experiment, video data were collected from scenes with markers placed in the FoV of the camera in five different positions (i.e., in the center, top left, top right, bottom left, and bottom right corners of the camera view space).The position of the camera was fixed for all experiments.
Since the current study is interested in analyzing a recognition accuracy of fiducial markers in AR-based system applicable for surgical navigation, interaction with individual components in such system is an important aspect for its operator.Therefore, operator interacted with markers and rotated them in any direction during the video recording process.Video record lasted about 2 minutes for each marker position.In total, 15 videos were recorded.
The marker recognition was implemented using Python programming language (version 3.9.1),OpenCV open-source computer vision library (version 4.5.5), and cv2.aruco module [13].The program input was images previously extracted from the video files.In case of success recognition, the program saved the coordinates of marker's corners and center to be used to confirm marker recognition.The processed images also were saved at the program output.Images with failed marker recognition were marked and saved separately.

Research results and discussion
The recognition accuracy was calculated as the fraction of the number of images with recognized marker to the total number of images with certain marker position in given experiment.
Table 1 lists the recognition accuracy of the markers by experiments.In all three experiments, the used algorithm successfully recognized most of markers in images captured by ultra-wide angle camera.Regardless of the physical sizes and number of bits of markers, the recognition accuracy in the center of image is on average higher than in the corners.The recognition accuracy in the corners relative to the center of image degrades by 13.5 %, 8.35 %, and 11.5 % on average in Experiment A, B, and C, respectively.
For experiments A and B where the same bit size of markers were used, better recognition accuracy (by 5.4 % on average) was observed for markers with larger physical sizes.For experiments B and C where the same physical size of markers were used, better recognition accuracy (by 6.2 % on average) was observed for markers with smaller bit sizes.It can be explained as captured images of markers with different physical sizes have different resolution in case the markers have the same position and at a fixed distance from the camera.This, in its turn, could lead to loss of information on some marker's details or elements of inner binary matrix may not be clear in the captured image.
The analysis of the images with failed recognition showed that most of the failures occurred because of lighting issues, i. e., due to insufficient illumination of the scene or presence of glare from lighting on the marker surface.The other significant part of failures was referred to the cases where only part of the fiducial marker was visible in the image.As any specific preprocessing for distortion correction was not applied to the images extracted from the video files, few recognition failures were referred to the distortion occurs (i.e., less than 0.2 % of all cases).To find out failed recognition cases due to the distortion presence, correction algorithm was applied to the input images copied beforehand.The OpenCV module was utilized to provide radial and tangential correction.Coefficients of radial and tangential were determined from the calibration data obtained before the experiments.Difference between the initial recognition rate and the rated obtained on the corrected data was considered as a measure of failed cases referred the distortion occurs.
In order for the marker to be successfully recognized, it should be positioned on white-light background or have white border around the marker to enable detection.In addition, the white elements of inner binary matrix of the marker should have good contrast respect to the black frame and the marker's body should be fully visible in the image.
Among the mentioned issues related to markers recognition failures, the cases caused due to insufficient illumination can be solved through applying algorithms of exposure compensation and the cases caused due to distortion occurs can be solved through calibration procedure with further applying of distortion correction algorithms.As for the cases caused due to the image contained only part of the marker, possibility to increase the recognition rate is complicated by the lack of built-in methods to process markers partially visible in the image.For this reason, the workspace for interaction with markers is smaller than the actual view space provided by the ultra-wide angle camera (i.e., by 15.85 % smaller for markers that have 3×3 cm physical sizes and by 25.96 % smaller for markers that have 5×5 cm physical sizes in our experiments).
Since usage of ultra-wide angle camera aims to solve difficulties related to limited workspace in AR-based system, distribution of failed recognition cases where the fiducial marker left the workspace (i.e., partially visible in the image) was analyzed separately (Fig. 3).
The results of this analysis revealed that despite the limited workspace, interaction with marker placed in the corners of the camera view space leads to cases where only part of the fiducial marker was visible in the image.On average, the mentioned factor reached 20.47 % of the total number of all unrecognized cases.TECHNOLOGY AUDIT AND PRODUCTION RESERVES -№ 3/2(65), 2022 ISSN 2664-9969 There was a noticeable increase in the percentage of failed recognition cases where the fiducial marker crossed the right side of workspace and was partially visible in the image.Right-side cases occurred more than 2.15 times often than lift-side cases.It can be explained by biomechanical impact during the interaction with marker, as operator was right-handed person.Since a movement to the left side by the right hand would be farther than reaching to the right side, it was more easer to cross the right side of workspace for the right-handed operator.
Despite its contributions, this study has some limitations to be considered.First, let's use data collected under the same lighting conditions.Poor lighting conditions affect the result of the marker recognition.Especially, it happens when final texture or geometry of marker in the image gets corrupted.Second, this study has taken into account only 2D ArUco markers.Usage of 3D cube with an ArUco marker attached to each side of the cube can affect the recognition accuracy as angle between the camera plane and the marker plane does not exceed 45 degrees.Finally, our study did not compare results of marker recognition from different commercial or open-source software, which can be applied for that.Only OpenCV library was used.
In the future, it is necessary to further examine recognition accuracy at the image boundaries and find a way to expand the workspace for interaction with fiducial markers close to the maximum possible.

Conclusions
In this study, we evaluated recognition accuracy of fiducial markers in zones where ultra-wide angle camera distort the most with taking into account different marker sizes.
It is possible to conclude that the proposed usage of ultra-wide-angle camera is feasible for fiducial markers recognition in AR-based system applicable for surgical navigation.The accuracy that could be achieved in distortion zones of ultra-wide angle camera is sufficient in case where marker's body is fully visible in the captured image.The recognition failures referred to the distortion occurs were less than 0.2 % of all cases.However, issues of failed recognition of makers partially visible in the image reduce the workspace for interaction with fiducial markers compare to the actual view space provided by the ultra-wide angle camera.

Fig. 3 .
Fig. 3. Distribution of failed recognition cases where only part of the fiducial marker was visible in the image: a -Experiment A; b -Experiment B; c -Experiment C a b c

Table 1
Markers recognition accuracy by experiments, %