Matlab Develop Custom Mini-Batch Datastore Example Bug?

Hello,

I was following the example of developing a custom mini-batch datastore through this link: https://www.mathworks.com/help/deeplearning/ug/develop-custom-mini-batch-datastore.html

In the function read of the example, after a file in the datastore is read in, the CurrentFileIndex is incremented by one. However, I was wondering if a client keeps reading the datastore that when it goes BEYOND the NumObservations, it will generate an error. To fix this problem, at the beginning of the read function, I would add the following:

if hasdata(ds)==false

reset(ds);

end

I'd appreciate very much if you could share your thoughts about this issue.

Answer 1

[Dinesh]

Hi Yong,

You're on the right path to ensure that your custom mini-batch datastore handles the end-of-data scenario gracefully. In MATLAB, when a datastore's read method is called and it has reached the end of the data (when "CurrentFileIndex" exceeds "NumObservations"), it is indeed good practice to check if there is more data to read using "hasdata". If there isn't, resetting the datastore with "reset(ds)" is a standard way to cycle back to the start of the data. This allows for continuous data reading, which is particularly useful during machine learning model training that requires multiple passes over the dataset (epochs).

Implementing this check and reset mechanism inside your read function makes your datastore robust and self-sufficient, as it can manage its state without relying on external control to avoid reading beyond the available data. Remember that if your datastore is going to be used in a parallel processing context, make sure that the reset operation is appropriately managed to prevent conflicts between different workers accessing the datastore. The documentation link that you mentioned will help you with everything.