How can I do centralized tracking with multi sensors?

Hi, I would like to know how can I implement centralized tracking with multiple sensors. I have camera and radar and I want to fuse them and create one track for a target using trackerGNN, trackerJPDA and trackerMHT. My tracks's state will be of world coordinates. I would like to know if my detection input of the tracker should always be in the same frame as track state (here world coordinates)? I am trying to convert my measurements from camera and radar to world coordinates in my measurement function of trackingEKF using converted measurement model. So my measurements are still in their respective frames. Will it be possible to achieve centralized tracking using this?

Thank you in advance,

AS

Answer 1

[recent works]

Centralized tracking with multiple sensors involves fusing the measurements from different sensors into a common state representation and updating a single track for the target. Here's a general outline of how you can achieve centralized tracking using trackerGNN, trackerJPDA, and trackerMHT with multi-sensor data:

1. Sensor Measurements and Fusion:

• Obtain measurements from your camera and radar sensors. These measurements will be in their respective sensor frames.
• Implement a sensor fusion algorithm to convert the measurements from different sensors into a common world coordinate frame. This process is called "sensor fusion" or "data association."
• The goal is to establish correspondences between measurements from different sensors and associate them with the same target. Various techniques like Kalman filters, data association algorithms (e.g., nearest neighbor, Hungarian algorithm), or probabilistic methods (e.g., Joint Probabilistic Data Association - JPDA) can be used for this purpose.

1. Centralized Tracking:

• Implement your selected centralized tracking algorithm (e.g., trackerGNN, trackerJPDA, trackerMHT) using the fused measurements in the common world coordinate frame.
• These algorithms use the fused measurements to update the target track's state and perform data association, as necessary, to handle sensor uncertainties and associations.

1. Measurement Function and Conversion:

• In your tracking Extended Kalman Filter (EKF) implementation, you'll have a measurement function that converts the predicted state to a measurement in the sensor's frame.
• Since your tracks' state is in the world coordinate frame, you need to convert the measurements from the camera and radar sensors to the world coordinate frame using the converted measurement model. This involves transforming the sensor measurements from their respective frames to the world coordinate frame.
• The converted measurements in the world coordinate frame will then be used for data association and track updates in the centralized tracking algorithm.

1. State and Measurement Covariance:

• When fusing measurements from multiple sensors, it's crucial to consider the covariance of both the state and measurements.
• The covariance reflects the uncertainty associated with the sensor measurements and the track's state, and it plays a crucial role in the data association process.

In summary, to achieve centralized tracking with multi-sensor data, you need to properly fuse the measurements from different sensors into a common world coordinate frame and then use these fused measurements in your centralized tracking algorithms like trackerGNN, trackerJPDA, or trackerMHT. Your converted measurements in the world coordinate frame will be used for the data association and track updates within the centralized tracker.

[Aatif Sulaimani]

Thank you for your answer. To begin,

1. I have bounding box from camera and range azimuth elevation from radar as measurements.
2. I create detections using objectDetection object for camera and radar with different sensor indexes.
3. I concatenate these detections in one.
4. These detections are input in my tracker.
5. In Extended Kalman filter, I separate these detections using sensor index and convert them to world coordinates through my states. The measurement function converts these states to measurement space of camera and radar respective to the sensor index and calculates the distance and residual.

Am I doing correctly? Because, for example, for the radar measurement space there will be no measurements from camera so how will the fusion happen and data association will take place?

[recent works]

Your approach is generally on the right track for centralized multi-sensor tracking.

[Aatif Sulaimani]

That's great to know. But I still have couple of questions,

1) So, If I have only camera measurements for this time instance, the tracker will predict the state using the kalman filter model for camera from the sensor index I specified. Now , it will look for detections to assign and if there is a radar detection which satisfies the assignment cost, it should assign the track to this radar detection. For, the next prediction step, will it switch to the radar kalman filter model specified by sensor index ?? as now it should update with radar measurements so to calculate residual from this equation :

y = Z - h(x)

y is residual,

Z is measurements,

h() is measurement fuction which maps my state to measurement space,

x is my track state in world coordinates.

2) If the normalized distance is calculated from the residual for the assignment, My residual calculation happens in measurement space but I would like to track in state space which is world coordinates. How can I achieve this?