A HYBRID LIAR/RADAR-BASED DEEP LEARNING AND VEHICLE RECOGNITION ENGINE FOR AUTONOMOUS VEHICLE PRECRASH CONTROL

B a s s a n t M o h a m e d E l b a g o u r y PhD Department of Artificial Intelligence and Robotics Humboldt University in Berlin Unter den Linden, 6, Berlin, Germany, 10099 E-mail: bassantai@yahoo.com R y t i s M a s k e l i u n a s Professor Department of Multimedia Engineering Faculty of Informatics Kaunas University of Technology K. Donelai io str., 73, Kaunas, Lithuania, 44249 A b d e l B a d e e h M o h a m e d M . S a l e m PhD, Professor University of Economics – Varna Research Institute of the University of Economics – Varna Knyaz Boris I blvd., 77, Varna, Bulgaria, 9002 E-mail: abmsalem@yahoo.com Завдання передаварійного інтелектуального керування робота автономних транспортних засобів є дуже складною проблемою, особливо передаварійні умови транспортних засобів і в точках перетину в умовах реального часу. Метою даного дослідження є розробка нового штучного інтелектуального адаптивного регулятора для системи передаварійної безпеки автономних транспортних засобів, а також модуля розпізнавання транспортних засобів та тестування в MATLAB, включаючи деякі деталізовані модулі. Були поставлені наступні завдання: пошук об’єктів за даними датчиків (Лiдар, Радар), контроль швидкості та рульового управління, розпізнавання транспортних засобів з використанням згорткової нейронної мережі та Alexnet. У даній дослідницькій роботі було реалізовано обробку зображень та лiдарних даних в режимі реального часу. Спочатку була представлена система реального часу, яка складається з комплексних модулів, а саме: модулі виявлення тривимірних об’єктів, групування та пошуку об’єктів, видалення землі, глибинного навчання з використанням згорткових нейронних мереж. Починаючи з модуля найближчого транспортного засобу, завданням було – знайти найближчий попереду автомобіль і вважати його основною перешкодою. У статті представлена адаптивна передаварiйна система керування швидкістю та розпізнавання транспортних засобів. Модуль адаптивної передаварійної системи керування швидкістю залежить від даних глибинного навчання та лідарного датчика, які призначені для управління безрозсудною поведінкою водія на дорозі шляхом регулювання швидкості транспортного засобу для підтримки безпечної відстані від об’єктів попереду (таких як автомобілі, люди, велосипед або будь-який інший об’єкт), коли водій намагається підвищити швидкість. Наразi, модуль розпізнавання транспортних засобів виявляє і розпізнає транспортні засоби навколо автомобіля Ключові слова: глибинне навчання, набір даних лідарного датчика, алгоритми K-мірного дерева, хмара точок, модуль розпізнавання транспортних засобів UDC 621 DOI: 10.15587/1729-4061.2018.141298


Introduction
How many accidents we see every day due to car accidents?There are many causes to these accidents such as: Driver's inattention or poor insight about certain spots around the vehicles (i.e.blind spots) or what attracts us the most, the drivers with visual disorders such as, night blindness, color blindness and so on.Egypt loses about 12,000 lives due to road traffic crashes every year.It has a road traffic fatality rate of 42 deaths per 100,000 populations.This huge percentage was an inspiration behind the proposed system, where we hope that the following objectives to be met, to save as much lives on the road as we can.In this paper, we propose an intelligent system aiming at reducing the driver's workload by assisting the driver through interpreting the environment autonomously and hence supporting the driver.The proposed system features two main modules that have our concerns which are: Adaptive cruise pre-crash system and vehicle recognition each of which will be discussed in the upcoming sections.The goal of this research is to develop a complete intelligent adaptive controller for autonomous vehicle Pre-Crash system.
The coming section describes the main modules and experimental results of the presented research.

Literature review and problem statement
Currently, a collision avoidance system is a system of sensors that is placed within a car to warn its driver of any dangers that may lie ahead on the road.However, Recent Research is still missing Pre-Crash Planning, which needs Intelligent (Adaptive) Controller to be embedded for complete Car Safety and path-planning.The main research problem is focused on two main goals, which are path planning and some cases of intersection safety.Intersection safety covers applications related to approaching or passing intersections, with emphasis on the cooperation between vehicles, while Path planning problem usually exists in an environment which has many obstacles and constraints.The problem has been proven to be NP-Hard problem [1][2][3][4][5][6].The path planning technology is an important aspect of AI and robotics.A star algorithm is a kind of path planning method which is applicable to the situation that the global environmental information is already known.
The frequency of vehicle crashes caused due to human errors (such as driver inattention or fatigue or the poor insight for the blind spots) can be decreased with the aid of an automatic system providing some mechanisms to avoid such accidents (Fig. 1).

Fig. 1. Truck Accident
An autonomous car (driverless car, self-driving car, robotic car) is a vehicle that is capable of sensing its environment, and navigating without human input.Autonomous vehicles detect surroundings using radar, GPS and computer vision.It is fundamentally defined as a passenger vehicle that drives by itself.It is also referred to as an autopilot, driverless car, auto-drive car, most prototypes that have been built so far performed automatic steering that were based on sensing the painted lines in the road.Today's researchers are using sensors and advanced software together with other custom-made hardware in order to assemble autonomous cars [7][8][9][10].
Although the prototypes seem to be very successful, a fully autonomous car that is reliable enough to be on the streets has not been constructed yet.This is mostly because of the difficulties involved in controlling a vehicle in the unpredictable traffic conditions of urban areas.While better hardware is being developed there are important limitations on the artificial intelligence side of the research.It would be fair to say that the future of the autonomous cars mostly depends on the development of better artificial intelligence software.On the other hand, most hardware that is being used on the cars seems to be doing well in terms of reliability, response time and accuracy.
Intelligent or Driverless cars/vehicles will be an inevitable thing in the future, thus we should focus our efforts in revitalizing the technologies and methodologies used to bring their deployment on a sooner timeline than anticipated.
Not only Fortune 500 Companies like Google and Tesla are undergoing research in this field, but they continuously find new ways every day to make cars fully autonomous and self-reliant.
Two main debacles set in this field are related to A) Path & Trajectory Planning and Calculations & B) Sensory and Environmental Awareness of surroundings.These two are the two main things we are going to focus on in our research.
Few of the most notable developers of autonomous vehicles and their systems are: -Ford [11][12][13][14][15]; -Tesla; -KITTI (German Research -University Based) [14].Both Ford & KITTI [14] are still in the early stages of development as tests are inducte each and every other day to ensue optimizations are applied to their algorithms properly and that they would be able to push and scale their methodologies for the greater good of autonomous driving.
Tesla is ahead in the sense that they have been investing more resources and time into this field for a long time now; we see them as one of the first to bring Level 1 Autonomy to the table of driverless cars.Because not only they develop and patent their hardware that will eventually lead them to full autonomy, they also think how would they make it safe for every human riding a car that they develop.A car that they would eventually sell to the average end-consumers which means they will be the first to tackle this field in a way appropriate to real life application.
So, without further ado, we will tackle both the issues we aforementioned at the start.And that is the path planning algorithm and the environment sensory subsidiaries.We will be implementing a LiDAR based Path Planning Algorithm and alongside this we will implemented Deep Learning for Vehicle Recognition.Other algorithms are required for Environment mapping and localization like SLAM [9] and EKF [16].

The aim and objectives of the study
This research paper aims at presenting a new artificial intelligent adaptive controller for autonomous vehicle Pre-Crash system along with vehicle recognition module and tested in MATLAB including some detailed modules.
To achieve the objective, the following tasks were set: 1. Finding Objects in sensor Data (LiDAR.RADAR).
2. Speed and Steering control.
3. Vehicle Recognition using convolution neural network and Alexnet.

Proposed System Architecture and Modules Engines
Regarding to our presented research paper, we are mostly interested in the implementation of the autonomous vehicles modules (Vehicle Detection and Recognition, Speed and Steering control).
How we do it scientifically?We use LIDAR and cameras with wide-angle lens that captures frames of high speed videos, and then we analyze each frame by applying different algorithms.Main system architecture is shown in Fig. 2, each will be described in details in coming sections.

1. LIDAR Odometry Algorithm: Processing data from LIDAR dataset
Lidar Odometry Algorithms is shown in Fig. 3, it is based on LOAM algorithm presented by Zhang [ref.].This algorithm took the point cloud from the last Scan as inputs, Lm, Is the growing point cloud of the present Scan, Lm+1, and constitute a shift from the last recursion, Pk l+1.If a new sweep is started, Pk l+1 sets to zero (line 4-6).Then, the algorithm works on feature extraction from Lm+1 to construct Nm+1 and Im+1 in line 7.For each feature point, its coincidence is found in Lm (line 9-19).The motion estimation is being adapted to a strong fitting [27].In line 15, the algorithm assigns a bisquare weight for each feature point.The feature points that have larger distances to their coincidences which are designated with more small weights and points of landmarks with greater distances than the threshold are considered extreme values and are assigned zero weights.Then, in line 16, the pose transform is updated for one repetition.The nonlinear optimization process ends if convergence is reached or the maximum repetition no. is met.If the algorithm reaches the end of the scanning process, Lm+1 is reestablishment to new time stamp Lm+2 using the predestined motion during the scan.Further, only the transform Pk l+1 is returned for the new next round of The recursion.LiDAR processing results are presented in coming sections.

2. LiDAR Sensor Dataset and Preprocessing
Regarding to our presented research paper, we are mostly interested in the implementation of the autonomous vehicles modules (Vehicle Detection and Recognition, Speed and Steering control).
How we do it scientifically?We use Light Detection and Ranging (LIDAR) and cameras with wide-angle lens that captures frames of high speed videos, and then we analyze each frame by applying different algorithms that will be described in details later on (Fig. 4).
Main components of the project: 1. Setup.
The coordinate systems are defined the following way, where directions are informally given from the driver's view, when looking forward onto the road: -Camera: x: right, y: down, z: forward; -Velodyne: x: forward, y: left, z: up; -GPS/IMU: x: forward, y: left, z: up.All coordinate systems are right-handed.3. Lidar.Lidar (also called LIDAR, LiDAR, and LADAR) is a surveying method that measures distance to a target by illuminating that target with a pulsed laser light, and measuring the reflected pulses with a sensor.Differences in laser return times and wavelengths can then be used to make digital representations of the target (Fig. 5).
-LiDAR stands for Light Detection and Ranging.
-LiDAR is the most reliable sensor for object detection.
-Measures Time of Flight (TOF), the round-trip travel time for a laser pulse reflected off obstacles.

Stereo camera:
A stereo camera is a type of camera with two or more lenses with a separate image sensor or film frame for each lens.This allows the camera to simulate human binocular vision, and therefore gives it the ability to capture three-dimensional images, a process known as stereo photography.
Stereo cameras may be used for making stereo views and 3D pictures for movies, or for range imaging.
The distance between the lenses in a typical stereo camera (the intra-axial distance) is about the distance between one's eyes (known as the intra-ocular distance) and is about 6.35 cm, though a longer base line (greater inter-camera distance) produces more extreme 3-dimensionality.

1. Ransac algorithm
Random sample consensus (RANSAC) is an iterative method to estimate parameters of a mathematical model from a set of observed data that contains outliers, when outliers are to be accorded no influence on the values of the estimates.Therefore, it also can be interpreted as an outlier detection method.It is a non-deterministic algorithm in the sense that it produces a reasonable result only with a certain probability, with this probability increasing as more iterations are allowed.The algorithm was first published by Fischer and Boles at SRI International in 1981.They used RANSAC to solve the Location Determination Problem (LDP), where the goal is to determine the points in the space that project onto an image into a set of landmarks with known locations.
A simple example is fitting of a line in two dimensions to a set of observations.Assuming that this set contains both inliers, i. e., points which approximately can be fitted to a line, and outliers, points which cannot be fitted to this line, a simple least squares method for line fitting will generally produce a line with a bad fit to the inliers.The reason is that it is optimally fitted to all points, including the outliers.RANSAC, on the other hand, can produce a model which is only computed from the inliers, provided that the probability of choosing only inliers in the selection of data is sufficiently high.There is no guarantee for this situation, [citation needed] [clarification needed] however, and there are a number of algorithm parameters which must be carefully chosen to keep the level of probability reasonably high (Fig. 22).

2. k-d tree algorithm:
In computer science, a k-d tree (short for k-dimensional tree) is a space-partitioning data structure for organizing points in a k-dimensional space.k-d trees are a useful data structure for several applications, such as searches involving a multidimensional search key (e.g. range searches and nearest neighbor searches).k-d trees are a special case of binary space partitioning trees The k-d tree is a binary tree in which every node is a k-dimensional point.Every non-leaf node can be thought of as implicitly generating a splitting hyperplane that divides the space into two parts, known as half-spaces.Points to the left of this hyperplane are represented by the left subtree of that node and points right of the hyperplane are represented by the right subtree.The hyperplane direction is chosen in the following way: every node in the tree is associated with one of the k-dimensions, with the hyperplane perpendicular to that dimension's axis.So, for example, if for a particular split the «x» axis is chosen, all points in the subtree with a smaller «x» value than the node will appear in the left subtree and all points with larger «x» value will be in the right subtree.In such a case, the hyperplane would be set by the x-value of the point, and its normal would be the unit x-axis A 3-dimensional k-d tree.The first split (the red vertical plane) cuts the root cell (white) into two sub cells, each of which is then split (by the green horizontal planes) into two sub cells.Finally, those four cells are split (by the four blue vertical planes) into two sub cells.Since there is no more splitting, the final eight are called leaf cells.

Vehicle Recognition Module
This module consists of 2 parts: -Extracting features from input image.
-Classifying those features between 2 categories vehicles and nonvehicle.

Speed Planning
In trajectory generation for unstructured environment, or even structured with lake of speed information we need another step predict speeds on different parts of the path.some planning algorithm optimize for speed and position in the same step as in DARPA Urban challenge winner BOSS [5] also in some situation like intersections and city streets tight curves speed profiling will be useful for smoother and safer autonomous drive.
This step is executed after smoothing first step and simulation step.
The main idea is simple: we analyze the curvature along the generated trajectory when curvature is small so it is a tight turn and speed should be small and vice versa.
Parameter of this algorithm: s max : Maximum speed limit.N curve : Number of points used for curvature calculation.By default we use 3, but according to how many points is gene rated in the smoothing step, this could produce misleading results, that why we check if all N curve does not cover more than the cat wheel base length.
w: smoothness weight, to control how driving style how smooth driving through curves will be.If we can collect a lot of data with different driving style we can define high level parameter to control this weight (race, sports, comfort, super comfort).By applying polynomial regression this could be easily predicted.

Deep Learning: Convolutional Neural Networks ALEXNET
Alexnet is a convolutional neural network, it was trained over 1.3 million images and can classify between 1000 categories.
The neural network, which has 60 million parameters and 500,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and two globally connected layers with a final 1000-way SoftMax (Fig. 25).
We use alexnet only to extract features from input images.We take the output from layer number 23 which has output of 4096 values then we use these values as input to a support vector machine to classify those features.
What is a Support vector machine?
In machine learning, support vector machines (SVMs, also support vector networks) are supervised learning models with associated learning algorithms that analyze data used for classification and regression analysis.New examples are then mapped into that same space and predicted to belong to a category based on which side of the gap they fall.
In addition to performing linear classification, SVMs can efficiently perform a non-linear classification using what is called the kernel trick, implicitly mapping their inputs into high-dimensional feature spaces.
When data are not labeled, supervised learning is not possible, and an unsupervised learning approach is required, which attempts to find natural clustering of the data to groups, and then map new data to these formed groups.The clustering algorithm which provides an improvement to the support vector machines is called support vector clustering and is often [citation needed] used in industrial applications either when data are not labeled or when only some data are labeled as a preprocessing for a classification pass.
More formally, a support vector machine constructs a hyperplane or set of hyperplanes in a high-or infinite-dimensional space, which can be used for classification, regression, or other tasks.Intuitively, a good separation is achieved by the hyperplane that has the largest distance to the nearest training-data point of any class (so-called functional margin), since in general the larger the margin the lower the generalization error of the classifier (Fig. 26).We have used MATLAB to implement our functions which consist of modules: -Reading dataset from (Lidar/camera); -Remove ground from Lidar; -Cluster 3d points into objects based on Euclidean distance; -Project each object on camera and draw a bounding box; -Crop each object and extract its features using ALEXNET; -Classify features using SVM.«Remove ground» This function removes ground from a LIDAR frame (Fig. 27).
«Cluster 3d» This function divides 3d points from LIDAR into objects based on distance between points.
«Nearest object» This function gets the nearest heading car.

«Training images preparation»
This function crops labeled images and prepare them to be the same size as AlexNet input size (Fig. 28).
«Train neural network» This function trains the SVM classifier on the features obtained from ALexNet.
«Detect car» This function detects heading car to get its distance and velocity (Fig. 29).
«Speed limit detector» This function calculates self-speed to estimate the Braking distance (Fig. 30).
«Vehicle recognition» This function detects and recognizes cars in the scene (Fig. 31).
5. Set the video path of your choice; 6. Run «gp_play2» function; 7. It will go through each frame in the dataset you have downloaded; 8. Wait until the program finishes executing; 9.The result video is in the path you specified.

3. Future Work
In this paper, we implemented a real-time image/Lidar processing.
At the beginning, we presented a real-time system which is composed of comprehensive modules, these modules are 3d object detection, object clustering and search, ground removal, deep learning using convolutional neural networks.Starting with nearest vehicle module our target is to find the nearest ahead car and consider it as our primary obstacle (Fig. 33).
Detecting all objects in the scene after removing ground presented in Fig. 34.

Conclusions
In this research paper, we implemented a real-time image/Lidar processing.At the beginning, we presented a real-time system which is composed of comprehensive mo dules, these modules are 3d object detection, object cluste ring and search, ground removal, deep learning using convolutional neural networks.Starting with nearest vehicle module our target is to find the nearest ahead car and consider it as our primary obstacle.
Presented an Adaptive cruise pre-crash system and vehicle recognition.The Adaptive cruise pre-crash system module depends on Deep Learning and LiDAR sensor data, which meant to control the driver reckless behavior on the road by adjusting the vehicle speed to maintain a safe distance from objects ahead (such as cars, humans, bicycle or whatever the object) when the driver tries to raise speed.At the very moment the vehicle recognition module, detects and recognizes the vehicles surrounding to the car.
The effectiveness of the results obtained is confirmed for the following test cases: -detecting all objects in the scene after removing groun; -recognizing only vehicles (filtering out non-vehicles); -calculating distances from each vehicle.

Acknowledgments
Thanks to student Yasmine Bakr and her colleague, our graduate students for some dataset programming which are supervised by Dr. Bassant Elbagoury at Faculty of Computer and information sciences, ain shams university, cairo, Egypt.

Introduction
Today, it is impossible to imagine a modern society without such innovations as the Internet, digital TV, mo-

FIRMWARE IMPLEMENTATION AND EXPERIMENTAL RESEARCH OF THE PHASE-LOCKED LOOP WITH IMPROVED NOISE IMMUNITY A . B o n d a r i e v
Doctor of Technical Sciences, Professor* E-mail: bondap@ukr.net

S . A l t u n i n
Postgraduate student* E-mail: serg.alt.i@gmail.com

I . H o r b a t y i
Doctor of Technical Sciences, Associate Professor* E-mail: giv@polynet.lviv.ua

I . M a k s y m i v
PhD, Assistant* E-mail: Ivan.P.Maksymiv@lpnu.ua*Department of theoretical radio engineering and radio measurement Lviv Polytechnic National University S. Bandery str., 12, Lviv, Ukraine, 79013

Fig. 19 .
Fig. 19.Display vehicles sample (1) Given a set of training examples, each marked as belonging to one or the other of two categories, an SVM training algorithm builds a model that assigns new examples to one category or the other, making it a non-probabilistic binary linear classifier (although methods such as Platt scaling exist to use SVM in a probabilistic classification setting).An SVM model is a representation of the examples as points in space, mapped so that the examples of the separate categories are divided by a clear gap that is as wide as possible.