Using pointCloudInputLayer with convolution3D

I have a [10000, 3] point cloud in R^3 and I want to pass it into a network that uses convolution3d, but I keep getting an error about the number of spatial dimensions being passed to the conv3d layer. This is the code I have so far:

X = randn(10000,3);
dsTrain = arrayDatastore(X,IterationDimension=1);

numOutputs = 1;
miniBatchSize = 128;

mbq = minibatchqueue(dsTrain,numOutputs, ...
    MiniBatchSize = miniBatchSize, ...
    MiniBatchFcn=@preprocessMiniBatch, ...
    MiniBatchFormat="SSSCB", ...
    PartialMiniBatch="discard");

numLatentChannels = 5;
featureSize = [10000 3];

%encoding layers
layersE = [
    pointCloudInputLayer(featureSize)  %image3dInputLayer(gridSize, 'Name', 'input')
    convolution3dLayer(3,32,Padding="same",Stride=2)
    reluLayer
    convolution3dLayer(3,64,Padding="same",Stride=2)
    reluLayer
    fullyConnectedLayer(numLatentChannels)];

projectionSize = [7 7 64];
numInputChannels = 1;

%decoding layers
layersD = [
    featureInputLayer(numLatentChannels)
    projectAndReshapeLayer(projectionSize)
    transposedConv3dLayer(3,64,Cropping="same",Stride=2)
    reluLayer
    transposedConv3dLayer(3,32,Cropping="same",Stride=2)
    reluLayer
    transposedConv3dLayer(3,numInputChannels,Cropping="same")
    sigmoidLayer];

netE = dlnetwork(layersE);
netD = dlnetwork(layersD);

%initialize parameters for ADAM solver
trailingAvgE = []; trailingAvgSqE = [];
trailingAvgD = []; trailingAvgSqD = [];

numObservationsTrain = 10000;
numIterationsPerEpoch = ceil(numObservationsTrain / miniBatchSize);
numIterations = numEpochs * numIterationsPerEpoch;

epoch = 0;
iteration = 0;

% Loop over epochs
while epoch < 10
    epoch = epoch + 1;

    % Shuffle data.
    shuffle(mbq);

    % Loop over mini-batches.
    while hasdata(mbq) && ~monitor.Stop
        iteration = iteration + 1;

        % Read mini-batch of data.
        X = next(mbq);

        % Evaluate loss and gradients.
        [loss,gradientsE,gradientsD] =...
            dlfeval(@ModelLoss,netE,netD,X);

        % Update learnable parameters.
        [netE,trailingAvgE,trailingAvgSqE] = adamupdate(netE, ...
            gradientsE,trailingAvgE,trailingAvgSqE,iteration,learnRate);
        
        [netD, trailingAvgD, trailingAvgSqD] = adamupdate(netD, ...
            gradientsD,trailingAvgD,trailingAvgSqD,iteration,learnRate);
    end
end

Where ModelLoss is defined as

function [Loss,gradientsE,gradientsD] = ModelLoss(netE,netD)
Z = forward(netE,X);
Y = forward(netD,Z);

Loss = mse(Y,X)
[gradientsE,gradientsD] = dlgradient(loss,netE.Learnables,netD.Learnables);
end

and projectAndReshapeLayer is defined as

classdef projectAndReshapeLayer < nnet.layer.Layer ...
        & nnet.layer.Formattable ...
        & nnet.layer.Acceleratable

    properties
        % Layer properties.
        OutputSize
    end

    properties (Learnable)
        % Layer learnable parameters.

        Weights
        Bias
    end

    methods
        function layer = projectAndReshapeLayer(outputSize,NameValueArgs)
            % layer = projectAndReshapeLayer(outputSize)
            % creates a projectAndReshapeLayer object that projects and
            % reshapes the input to the specified output size.
            %
            % layer = projectAndReshapeLayer(outputSize,Name=name)
            % also specifies the layer name.

            % Parse input arguments.
            arguments
                outputSize
                NameValueArgs.Name = "";
            end

            % Set layer name.
            name = NameValueArgs.Name;
            layer.Name = name;

            % Set layer description.
            layer.Description = "Project and reshape to size " + ...
                join(string(outputSize));

            % Set layer type.
            layer.Type = "Project and Reshape";

            % Set output size.
            layer.OutputSize = outputSize;
        end

        function layer = initialize(layer,layout)
            % layer = initialize(layer,layout) initializes the layer
            % learnable parameters.
            %
            % Inputs:
            %         layer  - Layer to initialize
            %         layout - Data layout, specified as a 
            %                  networkDataLayout object
            %
            % Outputs:
            %         layer - Initialized layer

            % Layer output size.
            outputSize = layer.OutputSize;

            % Initialize fully connect weights.
            if isempty(layer.Weights)

                % Find number of channels.
                idx = finddim(layout,"C");
                numChannels = layout.Size(idx);

                % Initialize using Glorot.
                sz = [prod(outputSize) numChannels];
                numOut = prod(outputSize);
                numIn = numChannels;
                layer.Weights = initializeGlorot(sz,numOut,numIn);
            end

            % Initialize fully connect bias.
            if isempty(layer.Bias)

                % Initialize with zeros.
                layer.Bias = initializeZeros([prod(outputSize) 1]);
            end
        end

        function Z = predict(layer, X)
            % Forward input data through the layer at prediction time and
            % output the result.
            %
            % Inputs:
            %         layer - Layer to forward propagate through
            %         X     - Input data, specified as a formatted dlarray
            %                 with a "C" and optionally a "B" dimension.
            % Outputs:
            %         Z     - Output of layer forward function returned as
            %                 a formatted dlarray with format "SSCB".

            % Fully connect.
            weights = layer.Weights;
            bias = layer.Bias;
            X = fullyconnect(X,weights,bias);

            % Reshape.
            outputSize = layer.OutputSize;
            Z = reshape(X,outputSize(1),outputSize(2),outputSize(3),[]);
            Z = dlarray(Z,"SSCB");
        end
    end
end

And preprocessMiniBatch is defined as

function X = preprocessMiniBatch(dataX)

% Concatenate.
X = cat(1,dataX{:});

end

I keep getting the error:

Layer 'conv3d_1': Input data must have three spatial dimensions only, two spatial dimensions and one temporal dimension, or three spatial dimensions and one temporal dimension. Instead, it has 1 spatial dimensions and 0 temporal dimensions.

It was my understanding that pointCloudInputLayer could take raw point cloud data. I also tried to voxelize the point cloud (following the example from Encode Point Cloud Data For Deep Learning) to use with image3dInputLayer instead, but wasn't sure how to follow since my data does not have any labels.

Any help is appreciated.

[Matt J]

It was my understanding that pointCloudInputLayer could take raw point cloud data.

Yes, but the error message is talking about the input to the convolution3dLayer, not the input to the network as a whole. It is not clear what you would like the 3D convolutional layer to do with point cloud data. Point cloud coordinates are not the same kind of data as the voxel values of a 3D image.

Answer 1

[Gayathri]

Hello @J Schreiber,

I understand that you are facing error while using “convolution3dLayer” along with “pointCloudInputLayer”. You've also attempted to use “image3dInputLayer” by voxelizing the input data.

The error occurs because the input to the “convolution3dLayer” does not have the expected number of dimensions. To use 3D convolutions, your input data should be in a 4D format that represents a 3D volume. One alternative is to replace the “convolution3dLayer” with a “fullyConnectedLayer”.

To continue using the “convolution3dLayer”, we will have to convert the point cloud to voxel grid representation. Also, we will have to use “image3dInputLayer” instead of the “pointCloudInputLayer”, as mentioned in the question.

% Example point cloud data
X = randn(10000, 3); % 10000 points with (x, y, z) coordinates
% Parameters for voxelization
voxelGridSize = [32, 32, 32]; % Define the size of the voxel grid
% Convert point cloud to voxel grid
voxelGrid = pointCloudToVoxelGrid(X, voxelGridSize);
% Create an arrayDatastore with voxelized data
dsTrain = arrayDatastore(voxelGrid, IterationDimension=4);
numLatentChannels = 1;
% Encoding layers using 3D convolutions
layersE = [
    image3dInputLayer(voxelGridSize, 'Name', 'input')
    convolution3dLayer(3, 32, Padding="same", Stride=2)
    reluLayer
    convolution3dLayer(3, 64, Padding="same", Stride=2)
    reluLayer
    fullyConnectedLayer(numLatentChannels)];
netE = dlnetwork(layersE);
% Function to convert point cloud to voxel grid
function voxelGrid = pointCloudToVoxelGrid(points, gridSize)
    % Initialize a zero grid
    voxelGrid = zeros([gridSize, 1]);
    
    % Normalize and scale points to fit within the grid
    minVals = min(points);
    maxVals = max(points);
    scaledPoints = (points - minVals) ./ (maxVals - minVals);
    voxelIndices = floor(scaledPoints .* (gridSize - 1)) + 1;
    
    % Populate the voxel grid
    for i = 1:size(points, 1)
        voxelGrid(voxelIndices(i, 1), voxelIndices(i, 2), voxelIndices(i, 3)) = 1;
    end
end

This code will help resolve the error pertaining to the input dimensions of the “convolution3dLayer”.

For more information on “image3dInputLayer” function, please refer to the following documentation link.

https://www.mathworks.com/help/releases/R2024a/deeplearning/ref/nnet.cnn.layer.image3dinputlayer.html

Hope you find this information helpful.