How to read a portion of file?

I have a text file which contains 10 colomns, tab seperated floating numbers. Colomn 3 is the data for Length. it is formatted in ascending order. I hope to select only the portion of file within a certain length range. The code attached reads the whole file, stores them in an array and selects the portion in the range. Is there a way to read from file only the portions in range, not the whole file. Thanks.

    WL_Low =1330;  % Length range
    WL_High = 1500; 

    fid = fopen('test.txt');%read in file

    data = fscanf(fid,'%f',[10 inf]);% store to temporary data file
    fclose(fid);
    DT = data(:,data(3,:)>WL_Low & data(3,:)<WL_High)'; % select data in  this range for processing.

Answer 1

[dpb]

Not nearly as efficiently as the above solution, no.

To do it on the fly while reading means processing each record one-at-a-time a much less efficient operation than operating on the file as a whole.

About the only exception to this could be if the file is so large as to cause memory problems or in that the file i/o is excessive. But, in the latter, it'll be even slower and the alternative there would be to go to stream instead of formatted file i/o.

[Chong Tao]

Thanks dpb,

is it possible to read in a portion of the file to an array and then process? It dosent need to process on the fly line by line. I did run into memory issue for large files.

[dpb]

Yeah, you can use the [M,N] size arguments in fscanf instead of [M,inf] as you have done and then repeat that process in a loop until feof. Matlab i/o is such that it will return the elements successfully read to that point.

[Chong Tao]

thanks again dpb. this will work. is it possible to do what you suggested for large files, say if the file is 20 M bytes or larger, and keep my original code for small files?

[dpb]

Sure, w/ a little logic. You can inquire for the file the size via dir() and then set the size of the number of elements to read dynamically if above some threshold. The [M,N] values can be variables; they need not be constants.