A Hybrid Method in Driver and Multisensor Data Fusion, Using a Fuzzy Logic Supervisor for Vehicle Intelligence
Driving is a very complex task which, at its core, involves the interaction between the driver and his/her environment. It is therefore extremely important to develop driver assistance systems that are centered on the driver. In this regards, various state of the art methods such as multi data sensor fusion techniques are applied on various types of physical sensors such as 3D cameras, Ultrasounds, Sonar and Infrared. However, sensor data alone are always uncertain to some extent due to noise and possible sensor failures. Also the driver, alone, is uncertain because of several parameters such as tiredness and drowsiness. In this paper we show the importance of providing a novel hybrid data fusion algorithm using both physical sensors and the driver's senses, as a supplementary data fusion system while driving. In this article, we use a fuzzy logic controller to manage and fuse the gathered data so take better action than individual driver or sensor fusion.
Driving is a complex task involving a great amount of interaction between the driver and vehicle. Drivers regularly divide their attention between operating the vehicle, monitoring traffic and nearby obstacles, and performing secondary tasks such as conversing, adjusting comfort settings (e.g. temperature, radio.) The complexity of the task and uncertainty of the driving environment make driving a very dangerous task. Several types of safety systems have therefore been proposed to help lessen the danger and assist the driver in his task.
Passive safety systems such as seatbelts and airbags provide protection in the case of collision; more recently however, active safety systems have been introduced to help the driver avoid collisions in the first place. Nowadays, systems such as lane departure warning systems and rear-end collision avoidance system have been introduced. These active safety systems are required to interact much more with the driver than passive safety systems, creating a closed loop between the driver, the vehicle, and the environment. Examples of such systems from the Laboratory for Intelligent and Safe Automobiles (LISA) can be found in .
In recent years, significant attention has focused on multisensor data fusion for both military and nonmilitary applications. Data fusion techniques combine data from multiple sensors and related information to achieve more specific inferences than could be achieved by using a single, independent sensor.
The concept of multisensor data fusion is hardly new. As humans and animals have evolved, they have developed the ability to use multiple senses to help them survive. For example, assessing the quality of an edible substance may not be possible using only the sense of vision; the combination of sight, touch, smell, and taste is far more effective. Similarly, when vision is limited by structures and vegetation, the sense of hearing can provide advanced warning of impending dangers. Thus, multisensory data fusion is naturally performed by animals and humans to assess more accurately the surrounding environment and to identify threats, thereby improving their chances of survival , .
Nonmilitary applications of multisensor data fusion include monitoring of manufacturing processes, condition based maintenance of complex machinery, medical applications, robotics and vehicle navigation. Techniques to combine or fuse data are drawn from a diverse set of more traditional disciplines, including digital signal processing, statistical estimation, control theory, artificial intelligence, and classic numerical methods. Historically, data fusion methods were developed primarily for military applications. However, in recent years, these methods have been applied to civilian applications and a bidirectional transfer of technology has begun . In Vehicle Navigations, The goal is to support the human operator in critical decision making situations or even eliminate the human operator altogether. Such systems are only desirable if they are able to perform at least as good as the human operator. For example fusing various sensors such as 3D cameras, sonar sensors and millimeter wave radar has the advantage of maintaining "higher reliability even in inclement weather or dusty conditions" , . Technology in vehicles has been rapidly increasing in recent years, notably in the areas of chassis control, telematics, infotainment, cruise control, lane detection, and obstacle avoidance . The area of driver assistance systems is also receiving much attention with the goal of improving road safety and driving efficiency. In this paper we focus on a complementary method for assisting the vehicle driver by a fuzzy controller system and a novel hybrid sensor fusion technique.
In this research we provide a fuzzy logic algorithm to fuse and manage the gathered data both from driver scenes (human's sensors) and from vehicle-mounted sensors (physical sensors). The Fuzzy Logic Supervisor (FLS) is a steering algorithm for managing the overall direction, speed and acceleration of a vehicle during a traveling in a road. Despite of public image that may think the driver is the supervisor of the vehicle, We'd like to show by managing and fusing both the driver command (as primary data) and physical sensors (as auxiliary data) a well-trained FLS can be the main supervisor. Keep in mind that the FLS action is different from autopilot and autonomous driving. This is a new approach and a hybrid data fusion algorithm which make better and safer performance in a vehicle driving. Figure1 shows a graphical image for our approach.
Step1: The proposed architecture performs both the tasks of sensor validation and sensor fusion. Also as depicted in figure 1 left Inputs to this architecture are the raw data from sensor readings and the output is a corrected value after sensor validation and sensor fusion. This value can be used for the FLS along with driver decisions as "brain fusion".
Step2: After sensor validation we have a real value from car sensor fusion and a real value from brain human senses fusion. But which of them are more important for next departure decision of the vehicle? In this section we use a simple but more efficient weighted average fusion method with following formula:
Where is the new determined value of hybrid sensor fusion to command the vehicle, is the previous value, is value from driver side and is the value from car sensor side. ,,are the importance factor for each term respectively, while . According to several practical experiments, we considered a constant value of, and a flexible value for,; so the final equation is as :
This means the FLS as a predictor make determines the next value and decision based on the degree of driver or sensor awareness. On the other word any perturbation in each of them, will lowers a or b respectively. So we define 20 rules with 4 essential fuzzy rules as below:
- IF driver is More-aware then is high
- IF driver is Less-aware then is low
- IF driver is Medium-aware then a and b are medium
- IF both driver and car sensors are High-aware then
We use a standard Gaussian function to form the input membership and triangular functions for output membership function of fuzzy logic supervisor block
Step3: Now, we are going for more detail. All sensor values are assigned a confidence value. According to our need (e.g. determining next speed, direction, or brake pressure) this confidence value depends on the specific sensor characteristics, the predicted value, and the physical limitations of the sensor value. The assignment takes place in a validation gate which is bound by the physically possible changes of the system . As the time pasts, according to last overall value determined by the FLS, the confidence value of each sensor changes from 0 to 1. that means that if the a specific sensor value is more similar to FLS value scope, then the confidence value increase to the limit of 1 and if value gathered by a specific sensor is less similar to FLS value scope, then it decrease to minimum confidence value of 0 and in this case this sensor may be eliminated in next evolution.
Step4: In the case of multisensor feature-level fusion, features are extracted from multiple sensor observations and combined into a single concatenated feature vector that is input to FLS. Finally fusion is performed through a weighted average of confidence values and distance measured as
is Fused Value , is Measurements and is Confidence Values.
Considering all mentioned relations, we have:
Our approach encapsulated two main benefits:
- Giving 40% importance degree to previous state in order to preventing sudden changes for the next state of the vehicle
- Possibility to change importance degree of gathered data from the driver and car sensors.
That means if the driver command is more similar to the predicted value by the FLS, so limits to maximum value (0.6) and if driver commands are illegal or not mindfully, so the control switches to sensors and so or limits to its maximum value so all of these, direct the vehicle to a smooth and safe driving.
In next section, we provide a simulation for our method.
3. Fuzzy Logic Controller Supervisor
In this section, we performed our approach for determining speed of the vehicle according the data reading from various sensor such as ultrasound distance detector, obstacle detection camera, road dense detection camera, road curve detection sensor, previous speed, driver command and so on. In this simulation the driver forces the pedal to change the speed of vehicle according to his/her brain decision. But the FLS checks the drivers command with auxiliary mounted sensors data in order to have a safe action. So it may changes the final command something different to the initial driver command. Figures 3 and 4 show membership functions and a 3D graphs for Speed of the vehicle in different situation.
Note: In case of noticeable difference in driver command (Brain Fusion) versus the Sensor fusion data, we use McCall facial processing method  and in case of positive answer for tiredness or drowsiness of the driver, the importance factor of the driver will decrease significantly, especially in higher speeds. This is visible in figure 4.
5. Deceleration in dangerous area
In previous steps we determined the appropriate speed according to driver request through human brain fusion and then environmental sensor fusion. Now as the next experiment we would like to determine the deceleration rate (Km/h) and determining a safe distance according to current speed of the vehicle and angular distance from an unexpected and instantaneous obstacle or vehicle. In fact we are going to fuse data one level more than previous. The sensor suite for this application consists of six readily available state-of-the-art sensors that are either in production or are near to production. Sensors are radar and are used for detecting vehicles and one is a video lane detection system . Of the six radar sensors two operate at 77GHz (about 30m coverage) and four at 24 GHz. The 77GHz sensors are radars based on the TRW production ACC unit  with one being installed forward facing and the other rear facing. The 24GHz radars are supplied by Smart Microwave Sensors (SMS) a leading radar manufacturer. Two of the four 24GHz radars are forward facing (and whose data is combined) and other two 24GHz radars are side facing. Fig. 5 shows the radar sensor configuration and approximate areas of coverage.
We fuse resultant of all the entire overtaking vehicles and obstacles around the vehicle by two parameters, or its angle (Positive or negative) and, its distance to vehicle. Whatever is greater and is smaller, the dangerous of collision is more probable and so the vehicle should be decelerated more quickly and it is essential to keep a safe distance without diverting from its path. Figure 6 shows an overtaking vehicle beside our desire vehicle.
According to result of , the coverage area of this real time embedded system is very acceptable and can be optimized by KF and EKF. Figure 7 shows the deceleration action by the FLS in such situations.
In Figure 7, the blue lines show small deviation in current speed and the more red lines shows the more probability of collision detection according to obstacle angle and distance. (In our system, Deceleration% is percentage of speed decrement in each 3 seconds). Looking exactly to the symmetric graph provided by FLS proves a logical decision making like a perfect driver.
Simultaneously to that, it is also necessary to make correction in distance to the unexpected obstacle or overtaking vehicle. In this section our goal is to determine the steer angle change in order to define an immune margin whiles other vehicles antecedence from our desire vehicle. In this fuzzy based method, there are 3 input variables and 1 output. We have defined 27 optimal and corrected rule according to an expert driver experience.
The inputs are: distance of closest overtaking vehicle, its angle and distance of vehicle to the verge of the road.
The output causes some correction to the steering angle and directs the vehicle to a safe distance to the overtaking vehicle on an unexpected obstacle. This change should be done in a manner that do not deviate the vehicle from lane of the road.
As depicted in figure 6, D1 is distance of overtaking vehicle from our vehicle, D2 is the distance of vehicle to the verge of the road, and angle shows the angle of side vehicle from our vehicle. The importance degree of angle increases from 0 to +180 asymmetrically as the defined Membership function (MBF) in figure 8.
This paper, proposed a logical, effective and practical hybrid sensor fusion using fuzzy method applicable in various depth of fusion for high speed vehicles in roads and highways. We used some important intuitive and linguistic experiences of an expert driver as our fusion rules and on the other hand gave the driver to control the vehicle in conjunction with sensor fusion. The control section (FLS) played an acceptable rule (much better than an individual driver) to control the vehicle safely and observant in encountering unexpected obstacles. As the future works it is recommended to test this algorithm with more sophisticated mathematical approaches and filters for better performance.
 M. M. Trivedi, T. Gandhi, and J. McCall. "Looking-in and looking-out of a vehicle", Selected investigations in computer vision based enhanced vehicle safety. IEEE Transactions on Intelligent Transportation Systems, 2007.
 Davil. L. Hall, James Llias, "Handbook of Multisensor data fusion", CRC Press LLC, 2001
 L. A. Klein, "Sensor and Data Fusion Concepts and Applications", SPIE Optical Engineering Press, Volume TT 14 (1993)
 S.Tokoro, K. Moriizumi, T. Kawasaki, T. Nagao, K. Abe and K. Fujita, "Sensor Fusion System for Pre-crash Safety System", IEEE Intelligent Vehicles Symposium, University of Parma, Italy,2004
 S. Tokoro, K. Kuroda, T. Nagao, T. Kawasalti, T.Yamamoto "Pre-Crash Sensor for Pre-Crash Safety", The 18th ESV Conference, Paper No.545-W, 2003
 eSafety Working Group, "Final Report on the eSafety Working Group on Road Safety", 2002.
 K. Goebel, A. Agogino, "An Architecture for Fuzzy Sensor Validation and Fusion for Vehicle Following in Automated Highways", 29th International Symposium on Automotive Technology and Automation (ISATA), Florence, Italy, 1996.
 J. McCall, M.M. Trivedi, "Driver Monitoring for a Human-Centered Driver Assistance System", HCM 06, Santa Barbara, California, USA, 2006
 Thompson, M.J., Trace, A.L., and Buchanan, A.J., "An Embedded Image Processing System for Driver Assistance System Algorithm Development", 4th International Conference on Vehicle Electronic Systems, Coventry, UK, 2001.
 Shooter, C. and Buchanan, A.J., "A Flexible Architecture for the Development of Multi-Sensor Driver Assistance Systems" , 4th International Conference on Vehicle Electronic Systems, Coventry, UK, 2001.
 Langheim, J., Henrio, J.-F., Liabeuf, B., "ACC Radar System "Autocruise" with 77 GHz MMIC Radar". ATA. Florence, 1999.
 Mark Tucker, A. Heenan, A. Buchanan," Real Time Embedded Sensor Fusion for Driver Assistance", Proceedings of the 8th International IEEE Conference on Intelligent Transportation Systems Vienna, Austria, 2005