read large amount of formatted data

Hi!

I have the following problem: From an experiment I get large amounts of data, each equally formatted and put into one huge datafile.

Each set of data has a header, that consists of a number of lines like these:

Measurement 
Device Name: lu234
Measurement Name: 1a1
Date: 13.12.2011 12:33
Comment: 
Sweep: Channel B
Sweep: b=10.00. e=-60.00. s=-2.00
Delay Time = {0.100000.0.500000.2.000000}

and the data following ...

   TransferData
1.000110E+1  -1.999950E+0  7.803810E-10  -4.414860E-10  -2.000000E+0
8.000990E+0  -1.999950E+0  -1.840490E-10  -8.939780E-10  -2.000000E+0
6.000650E+0  -1.999950E+0  5.676950E-10  -2.541150E-11  -2.000000E+0
4.001500E+0  -1.999970E+0  -8.976500E-8  -5.087180E-7  -2.000000E+0
1.999960E+0  -1.999940E+0  -1.073830E-9  -2.164310E-10  -2.000000E+0
...
...
1.000120E+1  -3.999680E+1  1.862440E-8  -5.154440E-9  -4.000000E+1
TransferData End
Measurement End

I started with using textscan to import the data, extracting the Device Name and the Measurement Name and importing the data into an array.

But now I am kind of stuck.. how can I move forward through the datafile? I would need to jump to the next set of data and so on until I reach EOF...

any help is appreciated!

Thank you in advance Chris

Answer 1

[Dr. Seis]

Looks like a similar problem to the one answered here:

http://www.mathworks.com/matlabcentral/answers/23351-parsing-complicated-text-file-header

To borrow from his answer:

% Open file and save necessary header info
fid = fopen('data.txt');
foo = fgetl(fid); 
device_name = textscan(fid,'Device Name: %s');
measurement_name = textscan(fid,'Measurement Name: %s');
foo = '';
while isempty(strfind(foo,'TransferData'))
  foo = fgetl(fid);
end
% Save data from 5-column text file
foo = textscan(fid,'%f%f%f%f%f');
import_data = zeros(size(foo{1},1),5);
for i = 1 : 5
  import_data(:,i) = foo{i};
end
fclose(fid);

[EDIT BELOW]

Assuming your file looks like:

Measurement 
Device Name: lu234
Measurement Name: 1a1
Date: 13.12.2011 12:33
Comment: 
Sweep: Channel B
Sweep: b=10.00. e=-60.00. s=-2.00
Delay Time = {0.100000.0.500000.2.000000}   
 TransferData
1.000110E+1  -1.999950E+0  7.803810E-10  -4.414860E-10  -2.000000E+0
8.000990E+0  -1.999950E+0  -1.840490E-10  -8.939780E-10  -2.000000E+0
6.000650E+0  -1.999950E+0  5.676950E-10  -2.541150E-11  -2.000000E+0
4.001500E+0  -1.999970E+0  -8.976500E-8  -5.087180E-7  -2.000000E+0
1.999960E+0  -1.999940E+0  -1.073830E-9  -2.164310E-10  -2.000000E+0
1.000120E+1  -3.999680E+1  1.862440E-8  -5.154440E-9  -4.000000E+1
TransferData End   
 TransferData
1.000110E+1  -1.999950E+0  7.803810E-10  -4.414860E-10  -2.000000E+0
8.000990E+0  -1.999950E+0  -1.840490E-10  -8.939780E-10  -2.000000E+0
6.000650E+0  -1.999950E+0  5.676950E-10  -2.541150E-11  -2.000000E+0
4.001500E+0  -1.999970E+0  -8.976500E-8  -5.087180E-7  -2.000000E+0
1.999960E+0  -1.999940E+0  -1.073830E-9  -2.164310E-10  -2.000000E+0
1.000120E+1  -3.999680E+1  1.862440E-8  -5.154440E-9  -4.000000E+1
TransferData End   
Measurement End

Then this should be able to go through each chunk of data. Right now the "import_data" is being written over for each chuck... I will leave it to you to store the data in the way you feel most appropriate.

% Open file and save necessary header info
fid = fopen('matlabtest.txt');
foo = fgetl(fid); 
device_name = textscan(fid,'Device Name: %s');
measurement_name = textscan(fid,'Measurement Name: %s');
while ~feof(fid)
  foo = '';
  while isempty(strfind(foo,'TransferData')) && ...
        isempty(strfind(foo,'Measurement End'))
      foo = fgetl(fid);
  end
  if ~isempty(strfind(foo,'Measurement End'))
      break;
  end
  % Save data
  foo = textscan(fid,'%f%f%f%f%f');
  import_data = zeros(size(foo{1},1),5);
  for i = 1 : 5
      import_data(:,i) = foo{i};
  end
  foo = fgetl(fid);
end
fclose(fid);

[Matt Tearle]

Rinse and repeat, inside a while ~feof(fid). One of the beauties of textscan is that it will "fail" at the "TransferData End" line (because it encounters a string instead of 5 numbers as was specified), with the file open at that point, ready to keep going.

[Walter Roberson]

feof(fid) never predicts when an EOF is about to happen: feof() is only true if a file reading command already attempted to read and found nothing waiting for it. You thus need to test the result of each file reading command anyhow, so while ~feof() might as well be while true

[christoph wolf]

I am already using this one, thank you for adjusting the code to my problem!

The problem arises after the "Measurement end", there is a gap of two empty lines and then the header of another measurement. Would it be possible (wise?) to just jump two lines ahead and start all over again (using fseek)?

Thank you guys for your input! =)

[Walter Roberson]

It is not possible to jump forward lines using fseek() . But if you wish you could add 'HeaderLines', 2 to the textscan() options. It happens that it will not matter because textscan() will skip over whitespace (including empty lines) when trying to match a %f format.

Answer 2

[christoph wolf]

Thank you so much for your help guys, unfortunately it appears as if I have been away from Matlab for too long and I am still a little bit stuck with the code...

My data actually looks like this (sry for not posting this earlier):

Device Name: lu345
Measurement Name: 2r1t

TransferData
1.000110E+1  -1.999950E+0  7.803810E-10  -4.414860E-10  -2.000000E+0
8.000990E+0  -1.999950E+0  -1.840490E-10  -8.939780E-10  -2.000000E+0
6.000650E+0  -1.999950E+0  5.676950E-10  -2.541150E-11  -2.000000E+0
4.001500E+0  -1.999970E+0  -8.976500E-8  -5.087180E-7  -2.000000E+0
TransferData End

Device Name: lu123
Measurement Name: 2r1a

TransferData
-4.172330E-5  8.000000E+0  9.398220E-7  1.998930E-9  0.000000E+0
-2.360340E-6  5.999820E+0  -1.114880E-9  1.985950E-10  0.000000E+0
-5.865100E-6  3.999640E+0  -5.238760E-10  -1.002860E-10  0.000000E+0
-1.999990E+0  -1.999940E+0  -2.717410E-9  -2.599350E-11  -2.000000E+0
TransferData End

so what I am trying to do now (based on what Elige posted) is to sift through the lines with fgetl until it reaches the String "TransferData" and adds 1 to the counter of datasets. Then it should read in the "Device Name" and "Measurement Name" as Strings and store them accordingly to the value of counter.

After that textscan reads in the formatted %f%f%f%f%f data and stores it into the import_data array with counter being the "Page" it sits on ( import_data(:,i,count))

when I plot I could then use the counter as the only variable I need to plot the data plus read the device name and measurement name that belongs to it.

I do not completely understand, why my program does not do what I expect it, but as mentioned it has been a few years since I last worked with ML

If someone finds the time to look this over it would be very much appreciated!

Merry Christmas!

clear all;

% Open file and save necessary header info
fid = fopen('data.dat');
foo = fgetl(fid); 

count=0;

while ~feof(fid)
  foo = '';

    while isempty(strfind(foo,'TransferData')) &&  isempty(strfind(foo,'Measurement End'))
        foo = fgetl(fid);   
    end

     if ~isempty(strfind(foo,'TransferData'))
        count=count+1;
     end

     device_name(count) = textscan(fid,'Device Name: %s');
     measurement_name(count) = textscan(fid,'Measurement Name: %s');

      % Save data
    foo = textscan(fid,'%f%f%f%f%f');
    import_data = zeros(size(foo{1},1),5,20);

   for i = 1 : 5
            import_data(:,i,count) = foo{i};
   end

    foo = fgetl(fid);

   if ~isempty(strfind(foo,'Measurement End'))
        break;
    end

end
  fclose(fid);