How to use ImageDataStore together with tall array?
Show older comments
So I'm trying to work with rather large amounts of data, that'll by no means fit into memory (~10TB of data), even though the computer I'm working on has 256GB memory.
I run into an issue using the following code (simply as an example, trying to understand the functionality of ImageDataStore and Tall arrays):
% Please don't mind the variable names!
ds = datastore('/Users/dummyPath/blabla/','Type','image');
K = tall(ds);
D = 0;
for ii = 1:10
D = D + mean(mean(mean(K(ii))));
end
N = gather(D);
OUTPUT:
Evaluating tall expression using the Parallel Pool 'local':
- Pass 1 of 2: Completed in 17 sec
- Pass 2 of 2: Completed in 0 sec
Evaluation 100% complete
Error using tall/mean (line 22)
Argument 1 to MEAN must be one of the following data types: numeric logical duration datetime char.
Learn more about errors encountered during GATHER.
Error in untitled (line 5)
D = mean(mean(mean(K(ii))));
Error in tall/gather (line 50)
[varargout{:}] = iGather(varargin{:});
Error in untitled (line 7)
N = gather(D)
Hope someone can help, I was unable to find anyone having the issue with datastore and tall arrays anywhere :/ Using them together could potentially alleviate a tedious process for me (shifting data in and out of memory, etc.).
Accepted Answer
More Answers (0)
Categories
Find more on Tall Arrays in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!