Kalman Filter-Based Fusion of LiDAR and Camera Data in Bird’s Eye View for Multi-Object Tracking in Autonomous Vehicles

Abstract

Accurate multi-object tracking (MOT) is essential for autonomous vehicles, enabling them to perceive and interact with dynamic environments effectively. Single-modality 3D MOT algorithms often face limitations due to sensor constraints, resulting in unreliable tracking. Recent multi-modal approaches have improved performance but rely heavily on complex, deep-learning-based fusion techniques. In this work, we present CLF-BEVSORT, a camera-LiDAR fusion model operating in the bird’s eye view (BEV) space using the SORT tracking framework. The proposed method introduces a novel association strategy that incorporates structural similarity into the cost function, enabling effective data fusion between 2D camera detections and 3D LiDAR detections for robust track recovery during short occlusions by leveraging LiDAR depth. Evaluated on the KITTI dataset, CLF-BEVSORT achieves state-of-the-art performance with a HOTA score of 77.26% for the Car class, surpassing StrongFusionMOT and DeepFusionMOT by 2.13%, with high precision (85.13%) and recall (80.45%). For the Pedestrian class, it achieves a HOTA score of 46.03%, outperforming Be-Track and StrongFusionMOT by (6.16%). Additionally, CLF-BEVSORT reduces identity switches (IDSW) by over 45% for cars compared to baselines AB3DMOT and BEVSORT, demonstrating robust, consistent tracking and setting a new benchmark for 3DMOT in autonomous driving.