4D Radar Imaging and Camera Fusion for Road Crossing Detection and Classification Using Deep Learning

Liyaana Shahirah Wan Abd Aziz; Farah Nadia Mohd Isa; Faridah Abd Rahman; Arvind Hari Narayanan; Ahmad Reza Alghooneh; George Shaker

doi:10.31436/iiumej.v26i1.3268

Authors

Liyaana Shahirah Wan Abd Aziz International Islamic University Malaysia https://orcid.org/0009-0001-8437-5699
Farah Nadia Mohd Isa International Islamic University Malaysia https://orcid.org/0000-0002-4305-4151
Faridah Abd Rahman International Islamic University Malaysia https://orcid.org/0000-0002-7683-7419
Arvind Hari Narayanan https://orcid.org/0009-0005-4198-6413
Ahmad Reza Alghooneh University of Waterloo https://orcid.org/0009-0002-9374-3630
George Shaker University of Waterloo

DOI:

https://doi.org/10.31436/iiumej.v26i1.3268

Keywords:

4D radar imaging, sensor fusion, deep learning, YOLOv7, Keras, TensorFlow

Abstract

This paper presents the development of an object detection and classification system for road crossing areas, integrating 4D radar imaging and a mono-camera dataset with a deep-learning neural network. The system utilizes deep neural networks implemented via Keras and TensorFlow to detect and classify multiple targets, including pedestrians, cars, buses, and trucks. At the core of this work is Retina-4F, a multi-chip radar imaging system developed by Smart Radar System, which offers high-resolution object detection and localization capabilities. Retina-4F provides real-time 4D information on detected objects, operating in a cascading architecture with three transmitters and four receivers per chip. Two road-crossing scenes were simulated to collect data, generating a point cloud dataset labeled with target classes for neural network training and testing. Data from two main sensors—Retina-4F and a mono-camera—were pre-processed using DBSCAN and YOLOv7 for enhanced accuracy. Operating at 77 GHz, Retina-4F was tested in two road environments, generating a dataset with approximately 10,000 frames. The deep learning model demonstrated an accuracy of 84% in classifying multiple targets, including cars, pedestrians, buses, and trucks. The fusion of radar point cloud data with visual sensor data proved effective, showing strong results in distinguishing target types.

ABSTRAK: Kertas ini membentangkan pembangunan sistem pengesanan dan pengelasan objek untuk kawasan lintasan jalan raya, menggabungkan pengimejan radar 4D dan set data mono-kamera dengan rangkaian neural pembelajaran mendalam. Sistem ini menggunakan rangkaian neural mendalam yang dilaksanakan melalui Keras dan TensorFlow untuk mengesan dan mengelaskan pelbagai sasaran, termasuk pejalan kaki, kereta, bas, dan trak. Inti daripada kajian ini adalah Retina-4F, sistem pengimejan radar berbilang cip yang dibangunkan oleh Smart Radar System, yang menawarkan keupayaan pengesanan objek dan penentuan lokasi resolusi tinggi. Retina-4F menyediakan maklumat 4D masa nyata mengenai objek yang dikesan, beroperasi dengan tiga pemancar dan empat penerima bagi setiap cip dalam seni bina kaskad. Dua adegan lintasan jalan disimulasikan untuk mengumpul data, menghasilkan set data awan titik yang dilabel dengan kelas sasaran untuk latihan dan ujian rangkaian neural. Data daripada dua sensor utama—Retina-4F dan mono-kamera—dipra-proses menggunakan DBSCAN dan YOLOv7 untuk meningkatkan ketepatan. Beroperasi pada 77 GHz, Retina-4F diuji dalam dua persekitaran jalan yang berbeza, menghasilkan set data dengan kira-kira 10,000 bingkai. Model pembelajaran mendalam menunjukkan ketepatan sebanyak 84% dalam mengelaskan pelbagai sasaran, termasuk kereta, pejalan kaki, bas, dan trak. Penggabungan data awan titik radar dengan data sensor visual terbukti berkesan, menunjukkan hasil yang kuat dalam membezakan antara jenis sasaran.

Downloads

Download data is not yet available.

Metrics

Metrics Loading ...

References

“Road traffic injuries,” World Health Organization, 20-Jun-2022. [Online]. Available: https://www.who.int/news-room/fact-sheets/detail/road-traffic-injuries. [Accessed: 25-Aug-2022].

C. Waldschmidt, J. Hasch and W. Menzel, "Automotive Radar — From First Efforts to Future Systems," in IEEE Journal of Microwaves, vol. 1, no. 1, pp. 135-148, Jan. 2021. DOI: https://doi.org/10.1109/JMW.2020.3033616

X. Gao, G. Xing, S. Roy and H. Liu, "Experiments with mmWave Automotive Radar Test-bed," 2019 53rd Asilomar Conference on Signals, Systems, and Computers, 2019, pp. 1-6.

I. Bilik, O. Longman, S. Villeval and J. Tabrikian, "The Rise of Radar for Autonomous Vehicles: Signal Processing Solutions and Future Research Directions," in IEEE Signal Processing Magazine, vol. 36, no. 5, pp. 20-31, Sept. 2019. DOI: https://doi.org/10.1109/MSP.2019.2926573

E. Ragonese, G. Papotto, C. Nocera, A. Cavarra and G. Palmisano, "CMOS Automotive Radar Sensors: mm-Wave Circuit Design Challenges," in IEEE Transactions on Circuits and Systems II: Express Briefs, vol. 69, no. 6, pp. 2610-2616, June 2022. DOI: https://doi.org/10.1109/TCSII.2022.3170317

I. Bilik et al., "Automotive multi-mode cascaded radar data processing embedded system," 2018 IEEE Radar Conference (RadarConf18), 2018, pp. 0372-0376. DOI: https://doi.org/10.1109/RADAR.2018.8378587

Y. Cheng, J. Su, H. Chen and Y. Liu, "A New Automotive Radar 4D Point Clouds Detector by Using Deep Learning," ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 8398-8402. DOI: https://doi.org/10.1109/ICASSP39728.2021.9413682

S. Sun and Y. D. Zhang, "4D Automotive Radar Sensing for Autonomous Vehicles: A Sparsity-Oriented Approach," in IEEE Journal of Selected Topics in Signal Processing, vol. 15, no. 4, pp. 879-891, June 2021. DOI: https://doi.org/10.1109/JSTSP.2021.3079626

M. Stolz, M. Wolf, F. Meinl, M. Kunert and W. Menzel, "A New Antenna Array and Signal Processing Concept for an Automotive 4D Radar," 2018 15th European Radar Conference (EuRAD), 2018. DOI: https://doi.org/10.23919/EuRAD.2018.8546603

J. Martínez García, D. Zoeke and M. Vossiek, "MIMO-FMCW Radar-Based Parking Monitoring Application With a Modified Convolutional Neural Network With Spatial Priors," in IEEE Access, vol. 6, pp. 41391-41398, 2018. DOI: https://doi.org/10.1109/ACCESS.2018.2857007

G. Li et al., "Novel 4D 79 GHz Radar Concept for Object Detection and Active Safety Applications," 2019 12th German Microwave Conference (GeMiC), 2019, pp. 87-90. DOI: https://doi.org/10.23919/GEMIC.2019.8698172

X. Gao, S. Roy, G. Xing and S. Jin, "Perception Through 2D-MIMO FMCW Automotive Radar Under Adverse Weather," 2021 IEEE International Conference on Autonomous Systems (ICAS), 2021, pp. 1-5. DOI: https://doi.org/10.1109/ICAS49788.2021.9551127

J. Wu, Z. Zhu and H. Wang, "Human Detection and Action Classification Based on Millimeter Wave Radar Point Cloud Imaging Technology," 2021 Signal Processing Symposium (SPSympo), LODZ, Poland, 2021, pp. 294-299. DOI: https://doi.org/10.1109/SPSympo51155.2020.9593690

Y. Kim, I. Alnujaim and D. Oh, "Human Activity Classification Based on Point Clouds Measured by Millimeter Wave MIMO Radar With Deep Recurrent Neural Networks," in IEEE Sensors Journal, vol. 21, no. 12, pp. 13522-13529, 15 June15, 2021. DOI: https://doi.org/10.1109/JSEN.2021.3068388

I. Alujaim, I. Park and Y. Kim, "Human Motion Detection Using Planar Array FMCW Radar Through 3D Point Clouds," 2020 14th European Conference on Antennas and Propagation (EuCAP), Copenhagen, Denmark, 2020, pp. 1-3. DOI: https://doi.org/10.23919/EuCAP48036.2020.9135381

Z. Yu et al., "A Radar-Based Human Activity Recognition Using a Novel 3-D Point Cloud Classifier," in IEEE Sensors Journal, vol. 22, no. 19, pp. 18218-18227, 1 Oct.1, 2022. DOI: https://doi.org/10.1109/JSEN.2022.3198395

R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in Proc IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., 2014, pp. 580–587. DOI: https://doi.org/10.1109/CVPR.2014.81

X. Cai, M. Giallorenzo and K. Sarabandi, "Machine Learning-Based Target Classification for MMW Radar in Autonomous Driving," in IEEE Transactions on Intelligent Vehicles, vol. 6, no. 4, pp. 678-689, Dec. 2021. DOI: https://doi.org/10.1109/TIV.2020.3048944

“Smart Radar System,” srs.ai. https://www.smartradarsystem.com/kr/index.html (accessed Mar. 02, 2021).

S. Lim, J. Jung, B. -h. Lee, J. Choi and S. -C. Kim, "Radar Sensor Based Estimation of Vehicle Orientation for Autonomous Driving," in IEEE Sensors Journal, 2022. DOI: https://doi.org/10.1109/JSEN.2022.3210579

X. Gao, G. Xing, S. Roy and H. Liu, "Experiments with mmWave Automotive Radar Test-bed," 2019 53rd Asilomar Conference on Signals, Systems, and Computers, Pacific Grove, CA, USA, 2019, pp. 1-6. DOI: https://doi.org/10.1109/IEEECONF44664.2019.9048939

Nabati, R., & Qi, H. (2021). Centerfusion: Center-based radar and camera fusion for 3d object detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 1527-1536). DOI: https://doi.org/10.1109/WACV48630.2021.00157

M. Abdalwohab, W. Zhang, A. M. S. Abdelgader and I. Abdelazeem, "Deep learning based camera and radar fusion for object detection and classification," 2021 IEEE 4th International Conference on Automation, Electronics and Electrical Engineering (AUTEEE), Shenyang, China, 2021, pp. 322-326. DOI: https://doi.org/10.1109/AUTEEE52864.2021.9668695

A. Sengupta, L. Cheng and S. Cao, "Robust Multiobject Tracking Using Mmwave Radar-Camera Sensor Fusion," in IEEE Sensors Letters, vol. 6, no. 10, pp. 1-4, Oct. 2022. DOI: https://doi.org/10.1109/LSENS.2022.3213529

J S. Guo, P. Wang, J. Ding and H. Liu, "Deep Model Based Road User Classification Using mm-Wave Radar," 2021 CIE International Conference on Radar (Radar), Haikou, Hainan, China, 2021, pp. 2843-2846. DOI: https://doi.org/10.1109/Radar53847.2021.10028068

F. Jin, A. Sengupta, S. Cao and Y. -J. Wu, "MmWave Radar Point Cloud Segmentation using GMM in Multimodal Traffic Monitoring," 2020 IEEE International Radar Conference (RADAR), Washington, DC, USA, 2020, pp. 732-737.

A. P. Rangari, A. R. Chouthmol, C. Kadadas, P. Pal and S. Kumar Singh, "Deep Learning based smart traffic light system using Image Processing with YOLO v7," 2022 4th International Conference on Circuits, Control, Communication and Computing (I4C), Bangalore, India, 2022, pp. 129-132. DOI: https://doi.org/10.1109/I4C57141.2022.10057696

C. Wang, A. Bochkovskiy and H. Liao, "YOLOv7: Trainable Bag-of-Freebies Sets New State-of-the-Art for Real-Time Object Detectors," in 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, 2023 pp. 7464-7475. DOI: https://doi.org/10.1109/CVPR52729.2023.00721

Y. A. Khan, S. Imaduddin, A. Ahmad and Y. Rafat, "Image-based Foreign Object Detection using YOLO v7 Algorithm for Electric Vehicle Wireless Charging Applications," 2023 5th International Conference on Power, Control & Embedded Systems (ICPCES), Allahabad, India, 2023, pp. 1-6. DOI: https://doi.org/10.1109/ICPCES57104.2023.10075892

J. Oh, K. -S. Kim, M. Park and S. Kim, "A Comparative Study on Camera-Radar Calibration Methods," 2018 15th International Conference on Control, Automation, Robotics and Vision (ICARCV), Singapore, 2018, pp. 1057-1062. DOI: https://doi.org/10.1109/ICARCV.2018.8581329

S. D. Boncolmo, E. V. Calaquian and M. V. C. Caya, "Gender Identification Using Keras Model Through Detection of Face," 2021 IEEE 13th International Conference on Humanoid, Nanotechnology, Information Technology, Communication and Control, Environment, and Management (HNICEM), Manila, Philippines, 2021, pp. 1-6. DOI: https://doi.org/10.1109/HNICEM54116.2021.9731814

M. Si, T. J. Tarnoczi, B. M. Wiens and K. Du, "Development of Predictive Emissions Monitoring System Using Open Source Machine Learning Library – Keras: A Case Study on a Cogeneration Unit," in IEEE Access, vol. 7, pp. 113463-113475. DOI: https://doi.org/10.1109/ACCESS.2019.2930555

S. -H. Chen, C. -S. Hung, J. -Y. Wang, C. -H. Chen and K. -C. Hsu, "The Implementation of Hybrid Electric Vehicle Battery Fault and Abnormal Early Warning System Using Keras Neural Network Technology," 2021 9th International Conference on Orange Technology (ICOT), Tainan, Taiwan, 2021, pp. 1-6. DOI: https://doi.org/10.1109/ICOT54518.2021.9680661

Wang, Y., Li, Y., Song, Y., & Rong, X. (2020). The influence of the activation function in a convolution neural network model of facial expression recognition. Applied Sciences, 10(5), 1897.as DOI: https://doi.org/10.3390/app10051897

Jin, F., Sengupta, A., Cao, S., & Wu, Y. J. (2020, April). Mmwave radar point cloud segmentation using gmm in multimodal traffic monitoring. In 2020 IEEE International Radar Conference (RADAR) (pp. 732-737). IEEE. DOI: https://doi.org/10.1109/RADAR42522.2020.9114662