how to process multiple file in matlab

i an a beginner in mat lab,i want to open a spice simulator using mat lab provide a file to it using mat lab and the output generated will be in .DAT so i want mat lab to open this DAT file and do the various function to plot the graph.

Answer 1

[Image Analyst]

See the FAQ for code examples: http://matlab.wikia.com/wiki/FAQ?title=FAQ&cb=9092#How_can_I_process_a_sequence_of_files.3F like:

    myFolder = 'C:\Documents and Settings\yourUserName\My Documents';
    if ~isdir(myFolder)
      errorMessage = sprintf('Error: The following folder does not exist:\n%s', myFolder);
      uiwait(warndlg(errorMessage));
      return;
    end
    filePattern = fullfile(myFolder, '*.dat');
    dataFiles = dir(filePattern);
    for k = 1:length(dataFiles)
      baseFileName = dataFiles(k).name;
      fullFileName = fullfile(myFolder, baseFileName);
      fprintf(1, 'Now reading %s\n', fullFileName);
      t = readtable(fullFileName);
    end

[Siddharth Pande]

thank you for your precious help. can you help me out explaining this program that would be a big help thank you once again.

[Image Analyst]

dir() finds all the *.dat files. I assume you know what a for loop does. At each iteration it takes the k'th filename on the list and prepends the path. Then it prints a message saying what file it's going to process. readtable() reads that data file into a table variable in memory - it's only available starting with R2014a so if you have an older version, use something else like dlmread() or textscan() or fgetl() or something.

[Maciej Ejsmont]

Dear Image Analyst, I know i'm diggin up old topic, but isn't your code rewriting varialbe t at each step of the loop? So that at the end of the loop, you would have only data from the last *.dat file? I'm having similar issue, i want to load data form several files, i have read that creating sequentional variables (like data1, data2 with eval function) is not the best practice. In that case, how can you approach it?

[Image Analyst]

@Maciej Ejsmont Yes, that's true. You create t, then use it however you want to, and then you're done with it so it can be overwritten. Why do you think you need to keep all the individual tables after you've created and use it? Do you need to use them later in some other loop that must come after the original loop?

[Maciej Ejsmont]

Oh, indeed i haven't thought about it. My goal was to process several files, and then present the data on one plot. But i guess i can do processing,and create new matrix consisting of processed data with common x-values, and just add new y-values from each file as seperate column. My thinking was i wanted to load all the data before processing, but now i realized it will only increase my RAM usage, espiecially for large number of big files.

/edit

Ah also, my processing gets rid of like 96% od the initial data (only 4% are usefull) so it would be even more justified to work on each file seperately.

[Image Analyst]

Yes, you can do something like this:

myFolder = 'C:\Documents and Settings\yourUserName\My Documents';
if ~isdir(myFolder)
    errorMessage = sprintf('Error: The following folder does not exist:\n%s', myFolder);
    uiwait(warndlg(errorMessage));
    return;
end
filePattern = fullfile(myFolder, '*.dat');
dataFiles = dir(filePattern);
for k = 1:length(dataFiles)
    baseFileName = dataFiles(k).name;
    fullFileName = fullfile(myFolder, baseFileName);
    fprintf(1, 'Now reading %s\n', fullFileName);
    t = readtable(fullFileName);
    % Get results for this one table and save the result in a vector y.
    y(k) = ProcessTableToGetY(t);
end

[Thiago de Aquino Costa Sousa]

@Image Analyst I am grateful that you exhist here in this community, your comments and codes are helping me a lot. I have same issue as @Maciej Ejsmont, as I will have to use a secondary loop later after the original loop. I have the following structure on my folder:

So, for each subject I have data collected on 2 different systems, at this moment I just want to analyze the System 1, which saves the data in 5 different folders "Condition 1-5", and then in each condition folder, it is saved the txt files for the trials. The txt files names follows a pattern for each condition, and repeats for each subject, but doesn't bring the name of the subjects on them.

So, I ran the code you gave until the line t = readtable(fullFileName); but I would like to have a long table, with all the trials (txt files) instead of rewrite the variable t in each iteration, and add a column with the name of the subjects (that is te folder names Subj1 etc), following an alfabethical order. The code I have so far is:

mytopfolder = '/Users/Participants data';
filepattern = fullfile(mytopfolder, '**/Trial_*.txt'); 
thefiles = dir(filepattern);
nfiles = length(thefiles);
for k = 1:nfiles
    basefilename = thefiles(k).name;
    fullfilename = fullfile(thefiles(k).folder, basefilename);
    fprintf(1, 'Now reading %s\n', fullfilename);
    thisdata = readtable(fullfilename);
end

After I have this long table, I will run some calculations and plots. So, can you help me on that? Thank you.

[Image Analyst]

Try vertcat or brackets to append the tables together:

mytopfolder = '/Users/Participants data';
filepattern = fullfile(mytopfolder, '**/Trial_*.txt'); 
thefiles = dir(filepattern);
nfiles = length(thefiles);
for k = 1:nfiles
    basefilename = thefiles(k).name;
    fullfilename = fullfile(thefiles(k).folder, basefilename);
    fprintf(1, 'Now reading %s\n', fullfilename);
    thisTable = readtable(fullfilename);
    fprintf('    It has %d rows in it.\n', height(thisTable));
    % Load this table into our master table of ALL files.
    if k == 1
        masterTable = thisTable;
    else
        % Second and subsequent tables:
        masterTable = [masterTable; thisTable];
    end
    fprintf('    Now there are a total of %d rows in the master table.\n', height(masterTable));
end

[Thiago de Aquino Costa Sousa]

Hi @Image Analyst, thank you very much for your quick response. It started running but then I got the following error:

Error using tabular/vertcat

All tables being vertically concatenated must have the same variable names.

Error in Main_extracting_data_subfolders (line 16)

masterTable = [masterTable; thisTable];

Any cues on what would be wrong?

[Image Analyst]

It's pretty clear. The columns in your two tables don't have all the same names, so how can they be appended? You can't vertically append a table that has two columns called "IDNumber" and "Address" with another table that has two columns called "Date" and "PatientName". How would that work?

Examine the two tables and figure out what you want to do when the tables don't have the same column names.

[Thiago de Aquino Costa Sousa]

Is there anyway I could contact you directly @Image Analyst?

[Walter Roberson]

@Thiago de Aquino Costa Sousa

No, Image Analyst does not accept direct contacts.

Answer 2

[Thiago de Aquino Costa Sousa]

You were completely correct. For some reason, the system I use to collect the data didn't name my second-to-last column, and pulled the last column's name into place, leaving the last column unnamed. See the pictures:

Correct

Mislabelled

Do you have any idea on how could I fix this problem with a code, because I have many files to solve the same problem. Thanks a lot for your time.

[Walter Roberson]

When you readtable(), pass in the 'VariableNames' property . That will override whatever names it reads from the file.

[Thiago de Aquino Costa Sousa]

Like this: thisTable = readtable(fullfilename, "ReadVariableNames",true);??

I did it, but it didn't work.

[Thiago de Aquino Costa Sousa]

Ok. With false, it passed, it returned what I expected... But whitout the name of the variables. That would be possible, to only ignore the name of the two last variables, instead of the 85 variables I have?

[Image Analyst]

I believe Walter said that if you pass in some variable names it will use those instead of the column headers in the file. So why not do that?

thisTable = readtable(fullfilename, "VariableNames", {'Column1', 'Column2');

[Walter Roberson]

Are the variable names always the same for your files? If so you can hard-code them.

Is the number of data columns always the same for your files, even though there might be one fewer variable names than columns? If so then by detecting the variable names and asking how many were found, you might be able to detect that one is missing.

It is possible, though, that if you use detectImportOptions, it will create a variable named 'Var85' for the last variable. If so then that could be detected and the last couple of variable names fixed up. But if your variable names are not always the same, what name should be inserted for the missing variable name?

[Thiago de Aquino Costa Sousa]

Hi @Walter Roberson, yes the number os columns is always de same, but the name of the variables is the ame in 95% of the files. In some of the files, the last two variables are mislabeled (Var84 and Var85), in my opinon it is caused when in the beginning of the trial, the participant is not looking in one of the targets, then my system (Unreal) consider the column GazeActor (Var84) as empty, and then it pulls the name of the column GazeBone (Var85) for the column 84 and let the column 85 unnamed.

The line code @Image Analyst gave me didn't work.

thisTable = readtable(fullfilename, "VariableNames", {'Column1', 'Column2');

Then I opted for other approach, I ran the code with the ReadVariableNames property false, then I renamed the masterTable with the name of the all 85 colums. And it worked.

thisTable = readtable(fullfilename,"ReadVariableNames",false);
masterTable.Properties.VariableNames = ["Frame","Sys_time",etc...]

Thank you very much for all you help @Image Analyst and @Walter Roberson.

[Thiago de Aquino Costa Sousa]

So now, what I have is a masterTable, with all lines (total 271,928) and columns (total 85) of all my files. However, it doesn't show me anywhere to which ever participant this data come from, because I don't have a column with the subject id. This information can be obtained from the first structure of subfolders, remember this picture.

From now and so on, I will need to perform some maths and plots for each participant. Then, I would like to ask you your opinion on what could be a better approach, to create a new column Subject.Id and fill it with the subject identification for all the lines belonging to each participant (I guess I could create something like masterTable.Subject_Id = ?, but I don't know how to do it in the loop, for each participant). Or if could do it in another way, calling the subfolder for each participant, and then applying the next code. Please guys, I would really appreciate this another help.

[Thiago de Aquino Costa Sousa]

I added this line of code:

thisTable.Subject_Id = fullfilename(98:102);

That returns the Id of the subject, but when I run it I got the following error:

Error using . To assign to or create a variable in a table, the number of rows must match the height of the table.

Error in Extracting_data_and_making_mastertable (line 14)

thisTable.Subject_Id = fullfilename(98:102);

Any cues how can I correct it?