Fuzzy Logic and Deep Steering Control based Recommendation

2018 18th International Conference on Control, Automation and Systems (ICCAS 2018) Oct. 17–20, 2018; YongPyong Resort, PyeongChang, GangWon, Korea

Fuzzy Logic and Deep Steering Control based Recommendation System for Self-Driving Car Nam Dinh Van and Gon-Woo Kim School of Electronics Engineering, Chungbuk National University, Chungbuk 28644, South Korea Abstract: Automatic cruise control system such as steering and velocity control are complex tasks that work as fundamental to all automatic systems in self-driving cars. The challenging task is to deal with steering signals at high speeds and to manage the vehicle’s velocity with significant changes in steering signals. In order to reach a destination with optimized time and stable trajectory, an autonomous vehicle system should utilize steering control and velocity signal in constraints. In this article, we propose a recommendation system based on Fuzzy logic and Deep Steering Neural Networks. Convolution neural networks(CNNs) works as a front-end stage for predicting steering control, and fuzzy logic in a back-end stage works as natural inferences for recommending velocity and adapting new steering control. The front-end stage uses CNNs with the input from raw sensory data, and the output extracts steering control prediction that feeds forward to the back-end stage. Key functions of back-end stage combine a number of dynamic vehicle information including, steering prediction to extract more effective steering control, and velocity for the autonomous car. The convolution neural network was trained on the Udacity driving datasheet with a number of hours of training and testing data, and the all system is built on MATLAB with GPU coder. Keywords: Autonomous vehicle, Convolution neural network, Fuzzy logic, Deep steering.

1. INTRODUCTION

test using a Quadcopter robot platform, poorly performed when tested with Udacity testing datacheet[3]. Several works [4][7] could be considered temporal and environmental characteristics such as velocity and tongue by using a sequence of images. It can be argued that their factors in a self-driving car would be supported by the car system, and estimating the elements would not be required. Another reason is that the velocity and tongue are hard to train correctly because of inattention of drivers follow these factors when they drive on driving way to record the training datasheet. In order to complete the work, new datasheet containing exact the elements with an attention of the drivers would be required. Although the end-to-end learning has achieved good results for predicting steering angle control, but the steering angle alone is not enough for vehicle control. A lack of speed commands greatly limits the potential applications of the endto-end methods. Udacity challenge[3] held in 2016, the best performer used 3D CNNs combined with RNN, LSTM archives with mean error of 2.6 degrees. Although this architecture gets the highest accuracy, the computation needs are expensive. The second high-ranking team used a simpler method with the error of 3.3 degrees. CNNs architecture similar to the Pilotnet architecture[3] was proposed, however instead of using direct images, the input passed was subtraction between the image at the current time and historical or future time (using V2V communication [8]) of the datasheet. Despite the fact that our goal is not to propose any enhanced method for predicting steering angle using Machine learning methods, methodology in this paper utilizes modifier of a simple and fast architecture CNNs to

Over the past decade, significant progress is made in machine learning domain pertaining to autonomous vehicles. Many competitions were organized to develop algorithms in Unmanned Vehicle System, one of the historic one being DARPA grand challenge[6] held in 2004. A completely autonomous car contains various state-of-theart fields: the perception, localization, mapping, motion planning, and control system. In the traditional methodology, the first step is to build a mathematical model of the car, and then to design a control system for velocity and steering. Numerous researchers have developed control algorithms based on PID controllers [10] as fundamental control system in cars. Sliding mode controller[11], fuzzy logic control[12], are also used for steering and tracking trajectory respectively. Other recently proposed methods utilize camera based deep neural network as an end-to-end self-driving system. This methodology uses data sheets collected by human driving in the outdoor environment and tries to learn the behavior. Nvidia team [1][9] is a pioneer in solving steering control prediction task as an end-to-end black box. The deep neural network(CNN) learns how to control a steering system of a Drive-less car from previous experiments of drivers with cameras mounted on the car. In order to predict the steering angle and understand the dynamic characteristics, several works [3][4][7] proposed 3D-LSTM architectures that consider temporal history images of the car. Antonio et al.[2] proposed a Dronet architecture based on Resnet 8 with two outputs that are steering angle and probability of collision. In spite of the fact that this research works effectively for short outdoor

978-89-93215-15-1/18/$31.00 ⓒICROS

1107

predict steering control. The steering factor works as an input of the front-end stage including fuzzy logic type -1 architecture to adapt new steering control and proposing velocity. Clearly recognized that, when a car was driven on the roads, the steering angle will affect to the way that a driver applies throttle. In order to robust the car at a high speed, driver needs less steer the wheel compared with normal speed. In contrast, when steering at a high angle, the drivers need to decrease the speed of the car for the same reason. Similarly, the steering angle control and velocity need to be automatically modified by a system that could infer like human. This work chooses fuzzy logic to solve the problem in a simple way such as a human thinking. This paper is not implemented on in a real car or dynamic simulation, instead Matlab is used for training, testing, simulating and comparing the results.

Steering angle

Fig. 2 Convolution neural network for predicting steering control angle. 2.2 Back-end stage with Fuzzy logic Fuzzy logic system

D

Steering angle

2. SYSTEM ARCHITECTURE

Pre steering angle

Image

C

Vehicle Information System

CNNs

It is hard to precisely describe a relationship between steering angle and velocity in mathematical equations to optimize trajectory of the car. Fuzzy logic helps to indicate the constraints by using natural human rules experiment. Fuzzy logic[13] using 2 processes are fuzzification and defuzzification, outputs are inferred by experiment rules. This stage shows in fig. 3 demonstrating a straightforward architecture using fuzzy logic[13]. This architecture works effectively with steering control angles and velocity by considering information of the predicting steering angle, previous velocity, and angular velocity of steering angle. Assume that the current time step is k + 1 , the Fuzzy logic system with the inputs:the different angle ∆θk+1 between current steering angle θk+1 at time step k + 1 from first stage and previous steering angle θk∗ modified by fuzzy logic system. The second input is car’s velocity vk , and the third input is angular velocity of steering angle γk . The outputs of the system are calculated by adding steering angle ∆θ∗k+1 and velocity ∆γ ∗k+1 with previous themselves respectively. Gausian functions membership Function (gaussmf ) are used for the inputs and the outputs with range values indicated in table 1 and table 2. Brief characters: RFP,FP, TN, FN, RFN are illustrated fast positive, fast positive, neutral, fast negative, really fast negative respectively. Similarly, RF, F,N,L,RL presents really fast, fast, normal, low and really low respectively. Rule set of the system follows the idea that if small changing steering angle, the outputs approximately maintain compared with

D  Pre Steering angle

Steering angle

Previous Velocity

Fuzzy Logic System

d dt

Fig. 3 Overview of Fuzzy Logic sytem.

Back-end stage

Front-end stage

Velocity

Steering velocity angle

FDRS System

Camera

Fuzzy Logic

Previous velocity

A system architecture that includes front-end and back-end stage is shown in Fig. 1. First part is using Convolution Neural Networks( CNNs) to predict steering angle, while the second part combines steering angle of the first stage and velocity of the Vehicle System. The angle and velocity are fed to the Fuzzy logic structure that is applied to extract steering and velocity of the car as a recommendation system.

Velocity

Fig. 1 System overview includes front-end and backend stage, first part using Convolution Neural network and the second part using fuzzy logic. 2.1 Front-end stage with Convolution neural network In this stage, a modifier of PilotNet[1] is applied to train with raw data input to extract steering wheel angles that work as an input of the Fuzzy logic system. The CNNs architecture is illustrated in the figure 2. Images are cropped and used only a half bellow of them that include vital information for predicting steering control. The CNNs also Using only 4 convolution layers, 3 fully layers different with 5, 4 convolution layer in [1] respectively. Batch Normalization is applied after all the 4 Conv Layers, and Dropout 20 percentage after first three fully layers compare with Pilotnet[1] without using the techniques.

1108

Table 1 fuzzify table with three inputs ∆θk+1 , vk , γk

(1) (2) (3) (4) (5)

∆θk+1 [-0.05;0.05 ] RFP FP N FN RFN

Table 2 ∆γ ∗k+1 (1) (2) (3) (4) (5)

vk [0;30] RF F N L RL

γk [-0.05;0.05] RFP FP N FN RFN

fuzzify table with two outputs ∆θ∗k+1 and ∆θ∗k+1 [ -0.05;0.05 ] RFP FP N FN RFN

∆γ ∗k+1 [-0.6;0.6] RFP FP N FN RFN

Fig. 5 A surface result of fuzzy logic: x-axis is the previous diffSteer, y-axis is preVelocity are inputs, and diffSteer is z-axis.

the inputs. In contrast, if steering angle prediction is applied at high velocity, the system decreases the steering angle and velocity recommendation. 125 rule set can be used to present fully the fuzzy system in detail, but this work simplified the rule set and using only 16 rules.

3. RESULTS Figures 4 to 6 shows the rule results of fuzzy logic system. Fig 4 presents acting of the outputs when changing the inputs. When we test by changing the input, the outputs also change similarly the human thinking. The Fig 7 presents training process in Matlab with 0.05 value of validation error approximately, and it also shows the same training and validation error compared with previous research[3]. Fig 8 illustrates the measurement of our approach tracking well with the ground-truth line. Fig 9 proves that the results of the system using fuzzy logic is better than without fuzzy logic. In the Udacity challenge [3] the best team using 3D CNNs, LSTM, RNN got 3.7310 in root-mean-square error (RMSE), RMSE of our system without the fuzzy system is 8.5298. When the Fuzzy logic system is applied, the RMSE shows 7.3775. when smooth filter technique is applied, the result presents approximately 7.

Fig. 6 The surface result of fuzzy logic: x-axis is the previous diffSteer, y-axis is preVelocity are inputs, and diffVelocity is z-axis.

Fig. 7 The training process result of Deep steering in Matlab.

Fig. 4 The rules interacting of inputs and outputs of the Fuzzy logic.

1109

Promotion (IITP) grant funded by the Korea government (MSIP) (No R7117-16-0164, Development of wide area driving environment awareness and cooperative driving technology which are based on V2X wireless communication).

1 ground-truth model

steering angle

0.5

0

REFERENCES

-0.5

[1]

-1 0

1000

2000

3000

4000

5000

6000

time

Fig. 8 Ground-truth and measured steering angles are displayed and compared.

[2]

1

[3] Steering angle

0.5

[4]

0 -0.5

[5]

-1 Steering without Fuzzy system Steering using Fuzzy system

-1.5 -2 0

1000

2000

3000

4000

5000

6000

time

[6]

Fig. 9 Comparing steering angles using the Fuzzy logic system and model without using it.

4. CONCLUTIONS

[7]

This work proposes a recommendation System for a driver-less vehicle based on fuzzy logic and deep neural network, which aims to drive autonomously a vehicle with cameras at the front of. Different from other proposed methods that predict steering control, we propose fast CNN model of modifier Pilotnet and a Fuzzy logic system to extract steering angle and velocity that roles as a recommendation for the autonomous car system. Besides estimating the steering angles, we have considered several vital dynamic elements by using the Fuzzy logic system in order to get stability trajectory of the vehicle. Although we tested our system with only changing previous steering control angle, the simulation results show better without using fuzzy logic. This result is just beginning, and we need to continuously test and complete the system. In the future, dynamic simulator and real car model will be built to implement and test this algorithm. To solve a problem with dense traffic, pass over the cars, etc. deep reinforcement[5] learning will be applied.

[8]

[9]

[10]

[11]

[12]

5. ACKNOWLEDGMENT This research is financially supported in part by the Ministry of SMEs and Startups (MSS), Korea, under the “Regional Specialized Industry Development Program (R&D, 2017020338)” supervised by the Korea Institute for Advancement of Technology (KIAT) and in part by Institute for Information&communications Technology

[13]

1110

M. Bojarski, D. Del Testa, D. Dworakowski, et al. “End to end learning for self-driving cars”, https://arxiv.org/abs/1604.07316, 2016. Antonio Loquercio, Ana I. Maqueda, et al. “DroNet: Learning to Fly by Driving”, IEEE Robotics and Automation Letters , Volume: 3, Issue: 2, April 2018. Udacity. “An Open Source Self-Driving Car”, https://www.udacity.com/self-driving-car, 2016. H. Xu, Y. Gao, F. Yu, and T. Darrell. “End-to-end learning of driving models from large-scale video datasets,”, CVPR Proc., July 2017. A. El Sallab, M. Abdou, E. Perot, and S. Yogamani. “Deep reinforcement learning framework for autonomous driving”, NIPS 2016 Workshop, Dec. 2016. Sebastian Thrun, Mike Montemerlo, et al. “Stanley: The Robot that Won the DARPA Grand Challenge”, Journal of Field Robotics , 23(9), pp.661–692, 2006. Shuyang Du, Haoli Guo, Andrew Simpson. “Self-Driving Car Steering Angle Prediction Based on Image Recognition”, CS231n, 2017, http://cs231n.stanford.edu/reports/2017/pdfs/626.pdf. Gaurav Bansal, Dhruv Chloudhary.“Convolution architectures for Self Driving cars”, CS231n, 2017, http://cs231n.stanford.edu/reports/2017/pdfs/629.pdf Mariusz Bojarski, Philip Yeres, Anna Choromanska, Krzysztof Choromanski, Bernhard Firner, et al. “Explaining how a deep neural network trained with end-to-end learning steers a car”, arXiv preprint arXiv:1704.07911, 2017. ToruYamamoto, KenjiTakao. “Design and Experimental Evaluation of a Data-Driven PID Controller”, IFAC Proc, Vol. 2010, 40, 391–396. Razvan Solea, Adrian Filipescu, Silviu Filipescu. “Sliding-mode controller for four-wheel-steering vehicle: Trajectory-tracking problem”, 8th World Congress on Intelligent Control and Automation, Jinan, China, pp. 1185–1190, July 2010. J.T. Economou ,R.E. Colyer, A. Tsourdos, B.A. White. “Fuzzy logic approaches for wheeled skidsteer vehicles”, Proceedings IEEE 56th Vehicular Technology Conference, Vancouver, BC, Canada, Canada, ISBN: 0-7803-7467-3, 24-28 Sept. 2002. Erdal Kayacan, Mojtaba Ahmadieh Khanesar. “Fuzzy Neural Networks for Real Time Control Applications, Concepts, Modeling and Algorithms for Fast Learning”, ISBN: 978-0-12-802687-8, 2016.