bboxLidarToCamera

Estimate 2-D bounding box in camera frame using 3-D bounding box in lidar frame

Syntax

bboxesCamera = bboxLidarToCamera(bboxesLidar,intrinsics,tform)

bboxesCamera = bboxLidarToCamera(bboxesLidar,intrinsics,tform,L)

[bboxesCamera,boxesUsed] = bboxLidarToCamera(___)

[___] = bboxLidarToCamera(___,'ProjectedCuboid',true)

Description

bboxesCamera = bboxLidarToCamera(bboxesLidar,intrinsics,tform) estimates 2-D bounding boxes in the camera frame from 3-D bounding boxes in the lidar frame bboxesLidar. The function uses the camera intrinsic parameters intrinsics and a lidar to camera transformation matrix tform.

bboxesCamera = bboxLidarToCamera(bboxesLidar,intrinsics,tform,L) further refines the 2-D bounding boxes to the edges of the object inside it using L. L is the corresponding labeled 2-D image of the 2-D bounding boxes, where the objects are labeled distinctively.

[bboxesCamera,boxesUsed] = bboxLidarToCamera(___) indicates for which of the specified 3-D bounding boxes the function detects a corresponding 2-D bounding box in the camera frame.

[___] = bboxLidarToCamera(___,'ProjectedCuboid',true) returns 3-D projected cuboids instead of 2-D bounding boxes.

Examples

Estimate 2-D Bounding Box in Image from 3-D Bounding Box in Point Cloud

Load data from a MAT file into the workspace.

ld = load("bboxData.mat");

Extract the image, point cloud, and camera intrinsics.

I = ld.I;
ptCloud = ld.ptCloud;
intrinsics = ld.intrinsics;

Extract one of the 3-D bounding boxes to project from the point cloud to the image.

bboxLidar = ld.bboxPtCloud(1,:);

Display the 3-D bounding box overlaid on the point cloud.

figure
pcshow(ptCloud)
showShape("cuboid",bboxLidar,LineWidth=1,Color="white")

Estimate the projection of the 3-D bounding box in the image using the transformation from the lidar sensor to the camera.

lidarToCameraTform = ld.lidarToCameraTform;
bboxesImage = bboxLidarToCamera(bboxLidar,intrinsics,lidarToCameraTform);

Undistort the image using the camera intrinsics.

I = undistortImage(I,intrinsics);

Display the 2-D bounding box annotated in the image.

Iannotated = insertObjectAnnotation(I,"rectangle",bboxesImage,"Vehicle");
figure
imshow(Iannotated)

Estimate Projected Cuboid in Image from Bounding Box in Point Cloud

Load data from a MAT file into the workspace.

ld = load("bboxData.mat");

Extract the image, point cloud, and camera intrinsics.

I = ld.I;
ptCloud = ld.ptCloud;
intrinsics = ld.intrinsics;

Extract one of the 3-D bounding boxes to project from the point cloud to the image.

bboxPtCloud = ld.bboxPtCloud(2,:);

Display the 3-D bounding box overlaid on the point cloud.

figure
pcshow(ptCloud)
showShape("cuboid",bboxPtCloud,Color="white")

Use the fuseCameraToLidar function with the transformation from the camera to the lidar sensor to get a point cloud of the points within the field of view of the camera.

cameraToLidarTform = ld.cameraToLidarTform;
cameraFOVPtCloud = fuseCameraToLidar(I,ptCloud,intrinsics,cameraToLidarTform);

Use the findPointsInModel function of the cuboidModel object to verify that the object enclosed by the point cloud bounding box is in the image.

model = cuboidModel(bboxPtCloud);
indices = findPointsInModel(model,cameraFOVPtCloud);
isObjectInImage = ~isempty(indices)

isObjectInImage = logical
   1

Visualize the point cloud of the field of view of the camera with the color from the image and the 3-D bounding box.

figure
pcshow(cameraFOVPtCloud,MarkerSize=30)
showShape("cuboid",bboxPtCloud,Color="white")

Estimate the projection of the 3-D bounding box in the image using the transformation from the lidar sensor to the camera.

lidarToCameraTform = ld.lidarToCameraTform;
bboxesImage = bboxLidarToCamera(bboxPtCloud,intrinsics,lidarToCameraTform,ProjectedCuboid=true);

Undistort the image using the camera intrinsics.

I = undistortImage(I,intrinsics);

Annotate the image to display the projected cuboid overlaid on the image.

Iannotated = insertObjectAnnotation(I,"projected-cuboid",bboxesImage,"Vehicle");
figure
imshow(Iannotated)

Input Arguments

bboxesLidar — 3-D bounding boxes in lidar frame

cuboidModel object | N-by-9 matrix of real values

3-D bounding boxes in the lidar frame, specified as a cuboidModel object or an N-by-9 matrix of real values. N is the number of 3-D bounding boxes. Each row of the matrix has the form [x_ctr y_ctr z_ctr x_len y_len z_len x_rot y_rot z_rot].

• x_ctr, y_ctr, and z_ctr — These values specify the x-, y-, and z-axis coordinates, respectively, of the center of the cuboid bounding box.

• x_len, y_len, and z_len — These values specify the length of the cuboid along the x-, y-, and z-axis, respectively, before it is rotated.

• x_rot, y_rot, and z_rot — These values specify the rotation angles of the cuboid around the x-, y-, and z-axis, respectively. These angles are clockwise-positive when you look in the forward direction of their corresponding axes.

This figure shows how these values determine the position of a cuboid.

Note

The function assumes that the point cloud data that corresponds to the 3-D bounding boxes and the image data are time synchronized.

Data Types: single | double

intrinsics — Camera intrinsic parameters

cameraIntrinsics object

Camera intrinsic parameters, specified as a cameraIntrinsics object.

tform — Camera to lidar rigid transformation

rigidtform3d object

Camera to lidar rigid transformation, specified as a rigidtform3d object.

L — Labeled 2-D image

matrix of real values

Labeled 2-D image, specified as a matrix of real values. The matrix size is the same as the ImageSize property of intrinsics.

Note

Labeled images are assumed to be undistorted.

Data Types: single | double | int8 | int16 | uint8 | uint16

Output Arguments

bboxesCamera — 2-D bounding boxes in camera frame

M-by-4 matrix of real values | M-by-8 matrix of real values

2-D bounding boxes in the camera frame, returned as an M-by-4 matrix of real values. M is the number of detected bounding boxes. Each row of the matrix contains the location and size of a rectangular bounding box in the form [x y width height]. The x and y elements specify the x and y coordinates, respectively, for the upper-left corner of the rectangle. The width and height elements specify the size of the rectangle.

If 'ProjectedCuboid' is set to true, the 2-D bounding boxes are returned as an M-by-8 matrix of real values. The bounding boxes have a cuboid shape and enclose the object. Each row of the matrix contains the size and location of the cuboid bounding box in the form [frontFace backFace]. Both the faces are represented as 2-D bounding boxes.

Data Types: single | double

boxesUsed — Bounding box detection flag

N-element row vector of logicals

Bounding box detection flag, returned as an N-element row vector of logicals. 2 is the number of input 3-D bounding boxes. If the function detects a corresponding 2-D bounding box in the camera frame, then it returns a value of true for that input 3-D bounding box. If the function does not detect a corresponding 2-D bounding box, then it returns a value of false.

Data Types: logical

Version History

Introduced in R2021a

R2022b: Supports rigidtform3d objects

You can now specify tform as a rigidtform3d object, which uses the premultiply convention. Although you can still specify tform as a rigid3d object, this object is not recommended because it uses the postmultiply convention. For more information, see Migrate Geometric Transformations to Premultiply Convention.