Multistage neural network training as an alternative to large sample data set

Hello,

I need to train a neural net of 30 x few million (inputs x samples) using a function fitting algorithm. My problem is obviously on the few million samples in which I have an out of memory issue.

As a result I am trying to come up with a way to divide the sample data and insert them in several parts to the NN. Could you point me out any documentation in mathworks on that direction or how could I possibly do that? Because I assume that many applications require way larger samples than mine to train a network so there must be a solution.

Thanks in advance,

Kyriakos

Answer 1

[Raunak Gupta]

Hi,

You may try to save the data into a bunch of ‘.mat’ files and can load them into fileDatastore. From this you may read each file data one-by-one and then pass it to the neural network in each iteration. The number of samples in each ‘.mat’ file can be decide based on how much samples can fit into the memory.