how to read data and process it from text file in chunks without saving the data in Matlab

Hi everybody I have a .csv file of 126000 rows with single column, I want to read them in chunks of 128 each time I process it, The problem is that the size of .csv file may increase more than that hence without saving the data how can I process it just to reduce memory cosumption

[Jan]

Do you want to split the file, or import the file in chunks?

[madhuri G]

Import file in chunks process each block

Answer 1

[Edric Ellis]

In recent versions of MATLAB (R2014b or later), you can use a datastore, which might possibly be slightly simpler than @Jan's approach. Here's how you might do that:

ds = datastore('myfile.csv');
ds.ReadSize = 128;
while ds.hasdata
  data = ds.read(); % Note: returns data as a MATLAB 'table' object
end

In R2016b, you could use a tall array to process this data. A tall array behaves like an ordinary MATLAB array, but the data remains on disk, and is read in chunks only when necessary. For example, you might do:

ds = datastore('myfile.csv');
tallTable = tall(ds);
% Perform calculations on the tall table
minVal = min(ds.Var1);
maxVal = max(ds.Var1);

% force evaluation - actually reads the data from disk
[minVal, maxVal] = gather(minVal, maxVal);

[madhuri G]

Thanks a lot Edric Ellis,this is what I was searching for

[Jan]

@madhuri: Then please unaccept my answer and accept this one. Thanks.

[EDITED, I've accepted this question. 08.02.2017 01:16 UTC]

[Stephane]

I've been limited by the number of lines despite setting a high ds.ReadSize of 1e6.

Maximum number of line read is around 300,000 on my system with matlab 2020a.

Answer 2

[Jan]

fid = fopen(FileName);
if fid == -1
  error('Cannot open file: %s', FileName);
end

chunkLen = 128;
while ~feof(fid)
  data = fscanf(fid, '%g', [1, chunkLen]);

    ... Process the data here ...
    ... The last chunk might be shorter than chunkLen ...
  end
  fclose(fid);

Importing 126000 numbers occupies 984kB of RAM only. Are you sure that this will be a problem?

[madhuri G]

Thanks for the reply Jan Simon, Here the problem is not reading blocks, but if I have some 30 files to be read at each iteration to keep track of all file ids in upcoming iterations the processor may take long time, So i thought of using Datastore for saving file ,reading files & writing to another file