extractEmbeddings

Extract feature embeddings of lidar point cloud from Segment Anything Model (SAM) encoder

Since R2024b

collapse all in page

    Syntax

    embeddings = extractEmbeddings(lidarSAM,ptCloud)
    embeddings = extractEmbeddings(lidarSAM,ptCloud,ExecutionEnvironment=executionEnvironment)

    Description

    embeddings = extractEmbeddings(lidarSAM,ptCloud) extracts the feature embeddings of the input point cloud ptCloud from the encoder of a Segment Anything Model (SAM), lidarSAM, by running a forward pass on the neural network.

    example

    embeddings = extractEmbeddings(lidarSAM,ptCloud,ExecutionEnvironment=executionEnvironment) additionally specifies the hardware resource with which to process the input data.

    Note

    This functionality requires Deep Learning Toolbox™ and the Image Processing Toolbox™ Model for Segment Anything Model support package. You can download and install the Image Processing Toolbox Model for Segment Anything Model from Add-On Explorer. For more information about installing add-ons, see Get and Manage Add-Ons.

    Examples

    collapse all

    This example uses:

    Open Live Script

    Specify a full file path for a LAS file that contains aerial lidar data. Then, read the point cloud data from the file using the readPointCloud function of the lasFileReader object.

    filename = fullfile(matlabroot,"toolbox","lidar", ...
         "lidardata","las","aerialLidarData2.las");
lasReader = lasFileReader(filename);
ptCloud = readPointCloud(lasReader);

    Remove the ground plane from the point cloud to get better segmentation results. Display the resulting point cloud.

    [~,nonGroundPtCloud] = segmentGroundSMRF(ptCloud);
figure
pcshow(nonGroundPtCloud)

    Create a Segment Anything Model object for aerial lidar point cloud segmentation.

    samLidar = segmentAnythingAerialLidar;

    Extract the feature embeddings from the point cloud.

    embeddings = extractEmbeddings(samLidar,nonGroundPtCloud);

    Specify a bounding box that contains an object to segment, and display it in green.

    boxPrompt = [429321 3680081 79.89 14 7 3 0 0 0];
showShape("cuboid",boxPrompt,Color="green")

    Segment the object using the segmentObjectsFromEmbeddings function, which runs the SAM decoder on the feature embeddings.

    mask = segmentObjectsFromEmbeddings(samLidar, ...
    embeddings,nonGroundPtCloud,BoundingBox=boxPrompt);

    Visualize the segmentation mask overlaid on the point cloud.

    figure
pcshow(nonGroundPtCloud.Location,single(mask))

    Input Arguments

    collapse all

    Segment Anything Model for aerial lidar data, specified as a segmentAnythingAerialLidar object.

    Unorganized point cloud, specified as a pointCloud object.

    Hardware resource on which to process the input point cloud data, specified as "auto", "gpu", or "cpu".

    • "auto" — Use a GPU, if available. Otherwise, use the CPU.

    • "gpu" — Use the GPU. To use a GPU, you must have Parallel Computing Toolbox™ and a CUDA® enabled NVIDIA® GPU. If a suitable GPU is not available, the function returns an error. For information about the supported compute capabilities, see GPU Computing Requirements (Parallel Computing Toolbox).

    • "cpu" — Use the CPU.

    Data Types: char | string

    Output Arguments

    collapse all

    Feature embeddings extracted from the Segment Anything Model encoder, returned as a 64-by-64-by-256 array.

    Version History

    Introduced in R2024b

    See Also

    | |