how can I gracefully exit a while loop, when reading from a data file of variable length, and the file may terminate abruptly and incompletely when written

here is some sample code with comments on what is happening... (actual code reads huge file with huge messages... shortened here to simplify)

% create sample data
A = [0 0 2 1 2];
B = [0 0 5 1 2 3 4 5];
C = [0 0 7 1 2 3 4 5 6 7];
D = [0 0 ];  % D is incomplete data.... not following protocol described below...

% create a file with sample data, assume you don't know what order...
fileID = fopen('test.bin','w');
fwrite(fileID,B);
fwrite(fileID,C);
fwrite(fileID,A);
fwrite(fileID,D);
fclose(fileID);

% read a file of sample data
fileID = fopen('test.bin');
ALL = fread(fileID).'
fclose(fileID)

% But, suppose the file has a huge amount (over a gigabyte) of data...
% don't want to read all that data in at the beginning,
% want to read in a block of data at a time and process it,
% then move to the next block...

% Imagine that a block of data is very very large...
% for this example A = 5 bytes, B = 8 bytes, and C = 10 bytes !!! much simplification
% suppose the first 2 bytes in the block are 0 0 
% and suppose the 3rd byte in the block contains the number of remaining bytes in the
% block you are trying to read...

fileID = fopen('test.bin');

while(1)
    'in while loop'
    DATA1_1 = fread(fileID,3);
    DATA1_2 = fread(fileID,DATA1_1(3));
    DATA1_ALL = cat(1,DATA1_1,DATA1_2).'
    
    'pausing now, press any key...'
    pause
end

% imagine that this is remaining code...  
% how can we exit the while loop gracefully 
% and still execute these lines of code !?   :-)

' this is remaining code.... want to see this line...'
' this is remaining code.... want to see this line...'
' this is remaining code.... want to see this line...'

Output will look something like this, before correcting the code...

ALL =
  Columns 1 through 14
     0     0     5     1     2     3     4     5     0     0     7     1     2     3
  Columns 15 through 25
     4     5     6     7     0     0     2     1     2     0     0
ans =
     0
ans =
    'in while loop'
DATA1_ALL =
     0     0     5     1     2     3     4     5
ans =
    'pausing now, press any key...'
ans =
    'in while loop'
DATA1_ALL =
     0     0     7     1     2     3     4     5     6     7
ans =
    'pausing now, press any key...'
ans =
    'in while loop'
DATA1_ALL =
     0     0     2     1     2
ans =
    'pausing now, press any key...'
ans =
    'in while loop'
Index exceeds matrix dimensions.
Error in break_out_of_while_loop (line 37)
    DATA1_2 = fread(fileID,DATA1_1(3)); 
    

question is... how to stop the 'while' loop more gracefully, and execute remaining code after while loop... !!!

[Geoff Hayes]

Robert - is the "Index exceeds matrix dimensions" error just an example, or is this happening with your code and so the app terminates? If you are concerned about encountering this condition, then check for it. If there will be an error - because the index exceeds the matrix dimensions - then just break out of the loop.

[Robert Brown]

you're close, this is 1/2 the answer !!!

either the index exceeds matrix dimensions, as you identified here, which is the more unlikely error,

or

if you don't have enough bytes of data for the fread, you run into an end-of-file error on the fread statement.

The "index exceeds matrix dimensions" in DATA1_2 = fread(fileID,DATA1_1(3)), is there because there isn't a 3rd byte of data in DATA1_1 to tell the fread how many elements to read from the file.

Since I only had 0 0 in my example case, DATA1_1(3) does not exist, and the fread bombs out there.

The real problem comes in when I have the 3rd byte, (which in my actual case is a INT16 word), and it may be for example 64000. Then DATA1_2 attempts to read 64000 bytes, but the data stream shuts off abruptly and there aren't 64000 bytes to be read, so I end up with an end-of-file error. I run past the end-of-file and error out, aborting program execution.

[Rik]

Personally I'm a fan of try,catch constructs, which seem the solution here.

This sounds like an attempt to write a DICOM-reader, so you could try the File Exchange for some other attempts at implementing a reader.

[Robert Brown]

Thanks for the comment !!!

@Image Analyst has answered the question using exactly this approach of the "try - catch" construct !!!

I didn't really know about the "try - catch" construct, so this was very educational to me.

I do appreciate you trying to help, and offering this advice !!!

Answer 1

[Image Analyst]

Not sure you created the demo/sample file correctly. Here, see if this more robust way of reading solves your problem:

% Demo by Image Analyst.
clc;    % Clear the command window.
fprintf('Beginning to run %s.m ...\n', mfilename);
close all;  % Close all figures (except those of imtool.)
clearvars;
workspace;  % Make sure the workspace panel is showing.
format short g;
format compact;
fontSize = 14;

% create sample data
A = uint8([0 0 2 1 2]);
B = uint8([0 0 5 1 2 3 4 5]);
C = uint8([0 0 7 1 2 3 4 5 6 7]);
D = uint8([0 0]);  % D is incomplete data.... not following protocol described below...

% Create a file with sample data, assume you don't know what order...
filename = 'test.bin'
% if ~isfile(filename)
	% Create file if it does not exist yet.
	fileID = fopen('test.bin', 'wb');
	fwrite(fileID, B, 'uint8');
	fwrite(fileID, C,'uint8');
	fwrite(fileID, A,'uint8');
	fwrite(fileID, D,'uint8');
	fclose(fileID);
% end

% Read a file of sample data
fileID = fopen(filename, 'rb');
fprintf('--------------------------------\nContents of %s:\n', filename);
ALL = fread(fileID)
fclose(fileID);
fprintf('--------------------------------\n');

% But, suppose the file has a huge amount (over a gigabyte) of data...
% don't want to read all that data in at the beginning,
% want to read in a block of data at a time and process it,
% then move to the next block...

% Imagine that a block of data is very very large...
% for this example A = 5 bytes, B = 8 bytes, and C = 10 bytes !!! much simplification
% suppose the first 2 bytes in the block are 0 0
% and suppose the 3rd byte in the block contains the number of remaining bytes in the
% block you are trying to read...

fileID = fopen('test.bin', 'rb');
loopCounter = 1;
maxIterations = 100; % Failsafe
while loopCounter < maxIterations
	fprintf('In while loop, on iteration #%d...\n', loopCounter);
	try
		DATA1_1 = fread(fileID, 3, 'uint8')
	catch ME
		break;
	end
	if length(DATA1_1) < 3
		% Not 3 bytes in what was read.
		break;
	end
	fprintf('   Going to read %d bytes...\n', DATA1_1(3));
	try
		DATA1_2 = fread(fileID, DATA1_1(3), 'uint8');
	catch ME
		% There were not that many bytes left in the file so it throws an error.  
		% Break out of loop in that case.
		break; 
	end
	fprintf('   Here are the %d bytes for iteration #%d...\n', DATA1_1(3), loopCounter);
	fprintf('   %d ', DATA1_2);
	fprintf('\n');
	DATA1_ALL = cat(1,DATA1_1,DATA1_2).'
	
	fprintf('pausing now, press any key...\n');
	promptMessage = sprintf('That is after iteration %d.\nDo you want to Continue processing with loop #%d,\nor Quit processing?', loopCounter, loopCounter+1);
	titleBarCaption = 'Continue?';
	buttonText = questdlg(promptMessage, titleBarCaption, 'Continue', 'Quit', 'Continue');
	if contains(buttonText, 'Quit', 'IgnoreCase', true)
		break; % or break or continue.
	end
	loopCounter = loopCounter + 1;
end
fprintf('Done with loop after %d iterations.\n', loopCounter);

fprintf('Done running %s.m\n', mfilename);

[Robert Brown]

Thank you for the help, @Image Analyst !!! I learned a lot from your answer !!! You have answered my question !!!

Also, It looks like you spent some time putting your answer together, and I appreciate that as well !!!

I didn't know much about the "try" construct... so it was good to see that in action !!!

I spent a small amount of time trying to figure out the "ME" label, only to find that it wasn't necessary and

could be deleted... :-)

I had also never seen the use of "mfilename", holding the name of the mfile currently being run...

Again, thanks very much !!! You excelled with your answer !!!

[Image Analyst]

The ME is not needed in the code I posted there, but it's useful if you want to delve into it since it gives you all kinds of useful information on the error and the traceback (line numbers and code) that threw the error. I'm attaching some utilities that I use that take advantage of that error object.

The main one I use is GetErrorMessage which gives the error plus a complete traceback into a string that you can display for the user. I've included some others that are, or may be, needed, but let me know if there is anything I forgot.

[Robert Brown]

@Image Analyst Wow you really are an overachiever !!! People like you are good to know !!! :-)

Thanks for all your help on this !... Maybe I can, in turn, impress someone else with all these tricks you've taught me !

I really really do appreciate all the hard work you put in on answering this question. "Accepted Answer" is far too little of a reward for the help you've given... so you also have my eternal admiration, for whatever that's worth, lol.

[Image Analyst]

Robert, well if you really also want to be an overachiever and award me extra "Reputation points" you can do so by clicking on the Vote icon on my two answers. Accepting (which you can do only once) gives two points. Votes (which you can do multiple) give two additional points per vote. Thanks in advance.

Also check out my File Exchange, which has a small fraction of my 300+ demos:

Image Analyst's File Exchange

Answer 2

[Image Analyst]

Try memmapfile(). It's meant for dealing with enormous files. Sorry - I have not used it myself so write back here or call tech support if you need help with that function.

[Robert Brown]

This looks like very good advice as well !!! I don't really know how to use the memmapfile(), but I think it has high payoff for the types of data and array sizes that I am subjected to processing. I'm going to study up on this and learn how it is done. Thanks again !!!