KonSensData

Mobile Sensor Technology for the Acquisition, Analysis and Transmission of Traffic Data

In order to make assisted and automated driving more efficient, safe and sustainable, intelligent transportation systems require a comprehensive array of sensors for environment perception. In the near future, not only the vehicles will communicate with each other, but the the infrastructure will also be involved in this process, thus ensuring better assessment of traffic situations. In this context, traffic and risk analyses will become increasingly important as fixed components of connected mobility as a concept. Fraunhofer scholars developed a mobile, AI-based senor unit that acquires and analyzes data, which is the necessary basis for intelligent infrastructure. This sensor unit allows making predictions about the traffic flows of passenger cars, trucks, bicyclists as well as pedestrians, and it enables communication with autonomous vehicles or advanced driver assistance systems in real-time. Real-time communication is also an important instrument in traffic management, for example, for coordinating traffic light settings or for determining accessible routes for emergency responders. The system can be operated highly flexibly: it is equally possible to connect the sensor box to existing infrastructure and to use it as a mobile sensor pole that works autonomously over longer periods of time. The technology is expected to have been installed in several German cities by the end of 2022.

The work was carried out in close cooperation with Fraunhofer IAIS and Fraunhofer FHR, headed by Fraunhofer IVI. The project receives funding within the scope of the Fraunhofer Cluster of Excellence Cognitive Internet Technologies CCIT gefördert.

Robustness Achieved by Combining Radar and Infrared Data

In contrast to conventional kinds of AI models for detecting vehicles and persons, the Fraunhofer approach is based on the combination of two different modes of data acquisition: Combined radar and infrared data create a robust analysis system for which even the temporary occlusion of traffic participants is not a problem – the system records data extremely reliably even under difficult lighting conditions, which makes it independent from time of day and operable in almost any kind of weather. The object detection process is complemented by a tracking procedure that provides additional robustness and allowes the analysis of traffic participants both in image sequences and in 3D.

Special attention to data protection is paid to the acquisition and analysis of data in public spaces.

The Algorithm and the AI Model – What is Innovative About the Approach?

Different sensors provide muti-modal data that can complement each other. By fusing the 4D radar scatter plots (3D dots with accompanying speeds) with infrared images, the sensors' respective advantages can be combined. Because the quality of the object detection depends on the fusion architecture, fusion is carried out on two levels and then evaluated. On the one hand, the process is continued on an object level as developed in the previous project (KonSens). Here, objects are detected and attributed separately by the infrared camera and the radar sensor. In the process, an unsupervised-learning algorithm used for clustering the radar scatter plots and a deep-learning approach with accpmpanying convolutional neural network (CNN for infrared image detection are improved. Thanks to the fusion, the final detection result profits from attributes provided by both sensors (e. g., the distance and speed of objects from the radar sensor and of object classes from the infrared camera). On the other hand, fusion on a feature level is investigated. Here, the sensors provide object features that are established with the help of, e. g., CNN backbones. These features are fused and then used as input for a neural network that carries out the final object detection process. This way, the quality of the object detection can be improved.

Independence from Light and Weather Conditions – How does the Technology Function Despite Difficult Weather and Darkness?

Neither infrared nor radar sensors are directly dependent on sunlight or other lighting. Put simply, most objects emit »radiation« in the infrared spectrum that is related to their temperature. This ability to radiate makes objects as well as people and other (moving) motorized vehicles visible even without lighting. In addition, unfavorable weather conditions such as fog are more transparent in the long-wave infrared spectrum. The technology is complemented by radar, which is an active sensor system. Radar emits its own signal and detects the environment's reply to it – independently from current weather and light conditions. Also, due to their operating principle, radar systems are robust against interferences such as the temporary occlusion of traffic participants.

Data Protection – How Can Data Protection-Compliant Acquisition and Evaluation be Achieved?

Neither of the two sensors is suitable for the identification of persons, which reduces the danger of data protection issues during the data acquisition phase. Furthermore, images and videos are not saved after processing, so as to protect the privacy of traffic participants.

Advantages of the Mobile Sensor Pole

• Combination of radar and infrared data for better robustness
• Artificial intelligence for traffic data analysis: Detection and tracking of traffic participants
• Detection independent from time of day and weather
• Individually adaptable to your use case
• Mobile and flexibly operable sensor pole/ sensor box
• Data protection-compliant

Application Areas

• Automated and autonomous driving
• Traffic management
• Smart cities

Industries

• Sensor manufacturers
• Infrastructure suppliers in the traffic sector
• Cities and communities