How can I extend 3D dataset into 4D dataset without memory issue?

I have 1500 images with 400x400 resolution. each pixel has integer data between 1 to 800.

[1500 400 400]

I want to convert this dataset into [1500 400 400 800] with logical data using one-hot encoding since I have to train the network with softmax layer.

new = zeros([1500 400 400 800],'logical')

But it says memory needs over 100GB and i only have 15GB on my device.

Is it necessary to load entire dataset in memory like Pandas?

Is there other option to convert dataset in my case?

Answer 1

[Sahaj]

Hi Brian.

Instead of loading the entire dataset at once, you can process it in smaller batches and then convert it into the desired format.

In MATLAB's Deep Learning Toolbox, you can use the imageDatastore function to efficiently read and process large image datasets. It allows you to read images in batches, reducing memory requirements. You can then use MATLAB's built-in functions for preprocessing and data augmentation to convert your dataset into the desired format.

Additionally, MATLAB supports one-hot encoding through functions like ind2vec or categorical arrays. These functions can be used to convert your integer labels into one-hot encoded logical arrays.

This way, you can avoid loading the entire dataset into memory at once and convert it into the desired format without requiring excessive memory.