How to organize data by reading specific name structure?

Hi,

I have multiple pairs of mat files named like "wk65_10_1044" or "wk85_10_1044" where:

• wk65/wk85 = instrument;
• 10 = day;
• 1044 = time.

Is there a way I can organize a code where after specifying instrument, day and time, it will check all the mat files and put the pairs under the same structure?

Thank you

[VINAYAK LUHA]

Hi Carola,

What's the basis of pairing ? same day and time ?

[Carola Forlini]

Hi,

Yes, same day and time.

[Jon]

Do you just want to put the names of the files into this structure, or the contents of the files? If the latter, can you please attach a few typical files, and say something about how you want the contents itself stored within the structure

[Carola Forlini]

I want to have the contents of the files too since I need to do further analysis pairing the data coming from the same day/time.

Each .mat file contain a structure with 3 arrays.

I am attaching an example coming from different instruments but equal day/time.

[Jon]

Please provide details out what you would like this structure (array of structures?)to look like

[Carola Forlini]

For each time values during the same day (each instrument wk65/wk85 has multiple times during the same day: i.e. wk65_14_0829/wk85_14_0829, wk65_14_0849/wk85_14_0849, wk65_14_0919/wk85_14_0910) I would like to obtain a structure containing per each time the substructure of wk65/wk85.

[Jon]

Sorry, still not clear to me. Suppose we call the data in each of your attached files a dataset. Then you will have collections of data sets for each day, within that day you will have collections of datasets for each time, within each time you will have a dataset for each instruments. So exactly what would you like the structure (array of structures) to look like (please provide and example of an element of your "structure" and what fields you would have). By the way I would suggest putting the data into a timetable rather than a structure, as it seems that however you design it, accessing particular elements of such a structure would be more difficult than if the data were in timetable. Here's how you could put that data into a timetable.

% Example of putting your data into timetable

% Get list of all the relevant data files
list = dir('wk*.mat');

% Get the part of the names with the day and hour
filenames = {list.name}'; % cell array of filenames
timestr = extractBetween(filenames,6,12);

% Get the part of the names with the instruments
instrument = extractBetween(filenames,1,4);

% Convert to datetime
time = datetime(timestr,"Format","dd_HHmm");
time.Format = 'dd-HH:mm'; % make display formatting a little more conventional

% Put filenames and instrument name into a timetable
tt = timetable(time(:),filenames(:),instrument(:),...
    'VariableNames',...
    {'filename','instrument'}); % use (:) to be sure they are columns

% Loop through filenames adding data to the timetable
for k = 1:numel(filenames)
    % Load the data from the file
    s = load(filenames{k});
    tt.fs{k} = s.fs;
    tt.sp{k} = s.sp;
    tt.ts{k} = s.ts;
end

[Jon]

Just edited that code a little, I forgot to put in the instrument name, and to name the variables

[Carola Forlini]

The structure I am looking for is something like this:

data.time1.wk65 = wk65;

data.time1.wk85 = wk85;

where time1 will correspond to one of the time. So, if I have 3 time in one day I will see something like:

data.time1.wk65 = wk65;

data.time1.wk85 = wk85;

data.time2.wk65 = wk65;

data.time2.wk85 = wk85;

data.time3.wk65 = wk65;

data.time3.wk85 = wk85;

I hope this explains better.

[Jon]

Generally preferable to use some kind of array to handle this type of data rather than embedding the array indexing in a field name, e.g. time1, time2. Much harder to too loop through data or access an individual one if you have to manipulate strings to build up field names. You could use something like

data(1).day = 10
data(1).time = 1044
data(1).instrument(1).name = 'wk65'
data(1).instrument(1).fs = fs % using corresponding data for this data set
data(1).instrument(1).sp = sp
data(1).instrunent(1).ts = ts
data(1).instrument(2).name = 'wk85'
data(1).instrument(2).fs = fs
% .
% .
% .
data(2).day = 12
data(2).time = 1333
% .
% .
% .

Answer 1

[VINAYAK LUHA]

Hi Carola,

I've made changes as per the said requirements, find the code attached below-

How to follow ?

1. Create a folder say A , and place your "rawData" folder in it, also create a new folder there and rename it as "organized".
2. With your current path set to A, run this code.
3. You can access the data as follows,
4. Yes this does cover the case when data is recorded multiple times during the same day.

            data.D14.T0829.Iwk85.fs/sp/ts

files=dir(fullfile(rawDataFolderPath,'*.mat'));
for i=1:numel(files)
    filename= files(i).name;
    splitfilename=split(filename,"_");
    compInstrument =cell2mat(splitfilename(1));
    compDay =cell2mat(splitfilename(2));
    compTime=cell2mat(splitfilename(3));
    compTime=compTime(1:end-4);
    organize("organized",[string(compDay),string(compTime),string(compInstrument)],1);
end

data=getsubs("organized",1);


%Recursive code to organize files in "rawData" folder into folders and subfolders in "organized" folder
function organize(spath,fname,i)
    if(i==4)
        copyfile("rawData"+"/"+fname(3)+"_"+fname(1)+"_"+fname(2)+".mat",spath+"/");
        return
    end
    if(exist(spath+"/"+fname(i), 'dir') ~= 7)
        mkdir(spath+"/"+fname(i))  
    end
    organize(spath+"/"+fname(i),fname,i+1)
end


%Recursive code to convert folder directory setup to MATLAB structure
function root= getsubs(spath,i)
    contents = dir(spath);
    contents = contents(~ismember({contents.name}, {'.', '..'}));
    if(numel(contents)==1&&contents.isdir==0)
        temp = load(spath+"/"+contents.name);
        root = temp;
        return 
    end
    root     =struct();
    for j =1:numel(contents)
          subs= getsubs(spath+"/"+string(contents(j).name),i+1);
          key = string(contents(j).name);
          if(i==1)
              root.("D"+key)=subs;
          elseif(i==2)
              root.("T"+key)=subs;
          elseif(i==3)
              root.("I"+key)=subs;
          end
    end
end

Hope this helps

[Carola Forlini]

Thank you very much for your help!

Answer 2

[VINAYAK LUHA]

Hi Carola,

Specify the qDay and qTime values in the below code, and access the data like data.wk65.fs/sp/ts , data.wk85.fs/sp/ts

qDay ='14';
qTime='0829';
qAns = [];
qInstrument=[];
files=dir(fullfile(EnterFolderPathHere,'*.mat'));
for i=1:numel(files)
    filename= files(i).name;
    splitfilename=split(filename,"_");
    compDay =cell2mat(splitfilename(2));
    compTime=cell2mat(splitfilename(3));
    compTime=compTime(1:end-4);
    if(strcmp(compDay, qDay) && strcmp(compTime, qTime))
        qAns =[qAns;string(filename)];
        qInstrument=[qInstrument,string(cell2mat(splitfilename(1)))];
    end
end
data = struct(qInstrument(1),load(qAns(1)),qInstrument(2),load(qAns(2)));

Hope this helps

[Carola Forlini]

Thank you!

I have a question, what if for the same day I have different times for both instruments? i.e.:

wk65_10_1044 wk85_10_1044

wk65_10_1144 wk85_10_1144

wk65_10_1213 wk65_10_1213

[VINAYAK LUHA]

Hi Carola,

Thanks, I've corrected the code, now it should work fine.

[Carola Forlini]

Hi Vinayak,

The code you provided works only for a specific time. What if I have multiple times during the same day?

Thank you