How to split up a large dataset into separate sub-datasets?

Hello everyone,

I'm working on a analysis of a large dataset which exists of multiple measurements in time at multiple locations. All datasets are stored under each other in a big text file. It looks like this:

[2] [2011] [location1] [other info]

[X1] [Y1] [X2] [Y2] ...... [Xi] [Yj]

[2] [2012] [location1] [other info]

[X1] [Y1] [X2] [Y2] ...... [Xi] [Yj]

[2] [20xx] [locationx] [other info]

[X1] [Y1] [X2] [Y2] ...... [Xi] [Yj]

All subdataset titles start with the number 2. The amount of measurements differs per subdataset. What I would like to have is one large structure which contains all measurements separately, from which I can easily extract data of some specific years/locations to make comparisons. I was hoping to find some method with which Matlab could recognize the start of a new subdataset, and then write this data into a field of a structure, and use some kind of loop to do this for all subdatasets. However, I was unable to fix this and unfortunately could not find any similar questions on the internet. I hope someone could help me. Thanks!

Answer 1

[Nirav Sharda]

The low level file I/O functions in MATLAB can be useful to achieve this. You can use fgetl to read lines from the file. Then you can use some tokenize the file using some functions like strtok or strsplit after which you can check for the starting 2. If that is not the case you can read your data in a structure. I am also adding a small sample script for your reference.

fid = fopen('name of file');
% while the end of file is not reached
while ~feof(fid)
    % read a line
    line = fgetl(fid);
    % get the tokens from the line
    tokens = strsplit(line,' ');
    % check if the line doesnot start with 2
    if str2double(tokens{1}) ~= 2
        % add the data to some structure.
    end
end

I hope this helps !