How to read text files from different sub folders in a folder ?

I have a collection of .txt files which are an output of a simulation software.The software puts the text files in different sub-folders within a folder.Is it possible to read all of those ?

For example:- In a single folder named top, there are 500 sub-folders(each subfolder is of format TC_SYS_01,TC_SYS_02,.....TC_SYS_500) with a text-file in each of them.I want to read those text files from each of those folders & do some operations.

I'm thinking of changing the simulation software code itself so that it outputs txt files within a folder(no sub-folders) & use this code:

F = dir('*.txt');
for i = 1:length(F)
   text = fileread(F(i).name) ;
   % Do further operations
end

but can i do it without changing the simulation software code and this : http://in.mathworks.com/matlabcentral/newsreader/view_thread/301929 doesn't help me

Answer 1

[Image Analyst]

Not sure which code you used in that link, but you can use genpath() to generate a list of all subfolders, then use a loop to recurse into each of them. In the innermost for loop is where you'll process your text file. Try it - just copy and paste and it should work. Here is the code:

% Start with a folder and get a list of all subfolders.
% Finds and prints names of all text files in 
% that folder and all of its subfolders.
% Similar to imageSet() function in the Computer Vision System Toolbox: http://www.mathworks.com/help/vision/ref/imageset-class.html
clc;    % Clear the command window.
workspace;  % Make sure the workspace panel is showing.
format long g;
format compact;

% Define a starting folder wherever you want
start_path = fullfile(matlabroot, '\toolbox');
% Ask user to confirm or change.
topLevelFolder = uigetdir(start_path);
if topLevelFolder == 0
  return;
end
% Get list of all subfolders.
allSubFolders = genpath(topLevelFolder);
% Parse into a cell array.
remain = allSubFolders;
listOfFolderNames = {};
while true
  [singleSubFolder, remain] = strtok(remain, ';');
  if isempty(singleSubFolder)
    break;
  end
  listOfFolderNames = [listOfFolderNames singleSubFolder];
end
numberOfFolders = length(listOfFolderNames)

% Process all text files in those folders.
for k = 1 : numberOfFolders
  % Get this folder and print it out.
  thisFolder = listOfFolderNames{k};
  fprintf('Processing folder %s\n', thisFolder);
  
  % Get filenames of all TXT files.
  filePattern = sprintf('%s/*.txt', thisFolder);
  baseFileNames = dir(filePattern);
  numberOfFiles = length(baseFileNames);
  % Now we have a list of all text files in this folder.
  
  if numberOfFiles >= 1
    % Go through all those text files.
    for f = 1 : numberOfFiles
      fullFileName = fullfile(thisFolder, baseFileNames(f).name);
      fprintf('     Processing text file %s\n', fullFileName);
    end
  else
    fprintf('     Folder %s has no text files in it.\n', thisFolder);
  end
end

[RC]

Hi, this is very helpful but is there a way to do this without using the uigetdir function which results in a pop-up window for user selection? Reason being I have to pass the script through a job on a server for which I'm not sure how this would work.

[Stephen Cobeldick]

@RC: so don't use uigetdir: use a hard-coded directory path, or pass the path as a function input argument, or get it from your UI, or whatever suits your operations.

[Sam Webb]

This is brilliant code thank you, as this code answers alot of my questions about looping through subfolders and extracting information. However this code does not work on my macbook pro but will work nicely on my Windows 10 work computer, is there a reason why it may not work on Macs?

[Image Analyst]

You don't need this code anymore since R2016b. You can use dir() with two asterisks, or use fileDatastore. See attached demo and see if it works with your Mac. If not, post your code in a new question.

[Michael Browne]

@Image Analyst This code is getting me the full list of file names. Is there a simple modification that I could make to this to feed every file on that list into function like:

[output] = extract_data(filename_in);

I've already made the extract_data fuction, and it essentially goes through a text file, and outputs the a time as a Matlab duration. The goal is to do this to every file, and then put all these durations into a histogram.

Or am I barking up the wrong tree right now?

[Image Analyst]

You could certainly put a call to your function inside the loop over f

[output] = extract_data(fullFileName); % Do some custom operations in this function.

You might also want to look into the imageDatastore() or dir() with two asterisks to have just one loop rather than a pair of nested loops.

Answer 2

[Meade]

You can try the code below. It will do the same thing as Image Analyst's post, but is a little cleaner if you don't need any of the nested folder information, i.e. you only want the file paths.

 mainFolder = uigetdir(); % Select your Main folder
 [~,message,~] = fileattrib([mainFolder,'\*']);
 fprintf('\n There are %i total files & folders.\n',numel(message));

 allExts = cellfun(@(s) s(end-2:end),{message.Name},'uni',0);% Get file ext
 TXTidx = ismember(allExts,'txt');% Search extensions for "CSV" at the end 

 TXT_filepaths = {message(TXTidx).Name};  % Use idx of TXTs to list paths.
 fprintf('There are %i files with *.txt file ext.\n',numel(TXT_filepaths));

 for ii = 1:numel(TXT_filepaths)
    % Do import here
 end

[StephMarine]

Do you know a way of using this if the file name has multiple . in it? the code runs well but reads the extension after the frist . so i have a few file names such as 000.111.222.csv so the ext is read as .111 any advice is appreciated!

[Stephen Cobeldick]

"the code runs well but reads the extension after the frist "

There is nothing in this code that detects the period character, so it doesn't even know where the first period character is.

Note that the code is buggy as it incorrectly assumes that all file extensions have three characters: this can easily be fixed using fileparts.

"any advice is appreciated!"

Write simpler, more robust code using dir, fullfile, and fileparts.

Answer 3

[KSSV]

Hi

You can get the names of the sub folders inside a folder using the following piece of code.

dirinfo = dir(pathfolder);

dirinfo(~[dirinfo.isdir]) = []; %remove non-directories

dirinfo(1:2) = []; % Remove the first two fields as they are . and ..

dirinfo is a structure with all the information. You can run a loop along this structure, go to the specific folder and then you can select the text files in it.

Cheers

Srinivas

[Luffy]

Thanks for the code.The above code gives the directory information in dirinfo variable. Now i know the names of all sub-folders. To access those text files i have to go inside each of those sub-folder,do text operations & go back to root directory.

For this i've tried:

length = numel(dirinfo);
for ii = 1:length
% folder name = dirinfo(ii).name
    cd dirinfo(ii).name
% file operations
%
end

But i throws up an error:- Error using cd; name is non-existent or not a directory*

[Stephen Cobeldick]

Note that the line commented with "Remove the first two fields as they are . and .." is buggy:

https://www.mathworks.com/matlabcentral/answers/345841-unexplained-error-on-dir#answer_271627

https://www.mathworks.com/matlabcentral/answers/21150-function-call#comment_46042

https://www.mathworks.com/matlabcentral/answers/394807-create-text-file-containing-list-of-file-names#comment_556641

https://www.mathworks.com/matlabcentral/answers/288815-gui-for-loop-not-working-correctly#answer_224956

https://www.mathworks.com/matlabcentral/answers/484430-using-struct-dir-selpath-what-do-and-mean-as-struct-name#answer_395754

https://www.mathworks.com/matlabcentral/answers/40949-omitting-pointers-when-listing-folder-contents#answer_50568

https://www.mathworks.com/matlabcentral/answers/13978-list-of-files-sorting#answer_19278

Answer 4

[Thorsten]

Use getAllFiles from http://stackoverflow.com/questions/2652630/how-to-get-all-files-under-a-specific-directory-in-matlab/26449095#26449095

filenames = getAllFiles('.', '*.txt', 1);

and then loop through the filenames and work on them; you do not need to cd to the directories.