Can you use DIR to list files in subfolders ?

Hello,

Would somebody be able to advise me on how to use DIR to find the dates that each file was last modified with a .mat extension in all sub folders in current directory?

This code does not look in sub directories. Also how to you specify the .mat extension.

files = dir(datenum)

Many thanks

Answer 1

[Walter Roberson]

You cannot do that in a single dir() call.

You need one dir() call on the current folder, and you look at the isdir() field of the results to see which names correspond to folders:

dirinfo = dir();
dirinfo(~[dirinfo.isdir]) = [];  %remove non-directories

Then do a step to remove the folder names "." and ".." so you do not infinite loop.

Then you loop over all those names and look inside each of the designated folders:

subdirinfo = cell(length(dirinfo));
for K = 1 : length(dirinfo)
  thisdir = dirinfo(K).name;
  subdirinfo{K} = dir(fullfile(thisdir, '*.mat'));
end

Now subdirinfo{K} is the structure of information about the .mat files in the directory dirinfo(K).name

[John]

Hello Walter,

Thank you for your reply.

Does the structure contain the date the file was last modified? The subfolders all contain a file called data.mat. It is these files that I plan to use the SORT function on to sort them by file name and save them as file1, file2...file1000.xls.

Many thanks

John

[Walter Roberson]

There is a field named datenum in the structure, which is a serial date number (numeric value) corresponding to the last modified date.

[Vids deo]

Hi Walter,

Can you shed some light on following statement ??

dirinfo(~[dirinfo.isdir]) = [];  %remove non-directories

Can you explain the syntax here please ??

--Vidya

[Walter Roberson]

http://www.mathworks.com/help/matlab/matlab_prog/comma-separated-lists.html

dirinfo is a structure array. Picking one field out of a structure array results in structure expansion. Putting [] around the structure expansion results in a vector of the values. So it is like

[dirinfo(1).isdir, dirinfo(2).isdir, dirinfo(3).isdir, ....]

The ~ is logical negation, so ~ of the expression is going to result in true for the places that are not directories. And then that is used as a logical index to select the entries in the structure to delete.

[Vids deo]

Hi Walter,

I'm kinda understanding what you're saying. But please clarify if the statement means the following in words - "Assign NULL to the elements of the 'dirinfo' structure array which have '0' or FALSE in the isdir field." --Vidya

[Walter Roberson]

Only if you understand that assigning NULL means to delete them.

Answer 2

[Jan]

In modern Matlab versions:

files = dir('D:\Data\**\*.mat')

[Xizhu Deng]

Very useful, Thanks!

[Sam Walder]

Hi Jan, I am having trouble with this. Would you expect it to work in R2016a? All the best

[Youngjea]

Great answer! Thank you Jan:)

[Steven Lord]

Sam Walder: I know this is a bit late, but no this will not work in release R2016a. The capability Jan used was introduced in release R2016b.

[Lisanne Vania]

Amazing, thank you

[Coco]

Hi Jan, Thank you. i want to put those files in the new folder, can you please help me with the coding? i am a newbby and have been tring for days with no luck.

[Koustubh Shirke]

Yes , Right it works in 2017a and onwards.

[Walter Roberson]

Koustubh Shirke it works in R2016b and onwards.

[Jorge Castillo]

For those using a previous version of R2016b, I propose to download the subdir function from here. Then create a script with the foll. lines and run it (F5):

a = mfilename('fullpath');
b = mfilename();
a = strrep(a,b,'');
addpath(a) % Adding the script current path 

spath= 'D:\titi'; % Directory where the files are placed
list_struc = subdir(fullfile(spath, '*.c')); %Result : a List (as structure)...
%  having the fullfile names of the desired file (eg. *.c)

[Malte Voß]

I have a problem doing exactly this. I'm using MATLAB R2021a and searching for specific data\**\*_blah.csv files. The file system and the matlab script is on a remote server with more thas 4000 subfolders. However only calling dir() takes so long that I aborted after half an hour. Pausing than also takes so long that I had to abort. Do you have any idea why this happens and how I can avoid it? The amount of files I am searching for is about 60 files...

[Walter Roberson]

@Malte Voß

I suggest you use the technique I show in https://www.mathworks.com/matlabcentral/answers/32038-can-you-use-dir-to-list-files-in-subfolders#answer_40544 of looping through the folders. That reduces the work to be done at any one time, so it should be less likely to freeze.

Does the remote system happen to be MacOS or Linux? If so then it might be worth using rsh or equivalent to run a remote unix find command.

[Yiting Zhu]

This solution really works!

[Jorge Leon]

Useful and simple. Thanks!

[Xuelin]

Such a simple and elegant solution!

Answer 3

[Walter Roberson]

filetofind = 'data.mat';

dirinfo = dir();
dirinfo(~[dirinfo.isdir]) = [];  %remove non-directories
tf = ismember( {dirinfo.name}, {'.', '..'});
dirinfo(tf) = [];  %remove current and parent directory.

numsubdir = length(dirinfo);
lastmod = inf * ones(numsubdir,1);

for K = 1 : numsubdir
  subdirinfo = dir(fullfile(dirinfo(K).name, filetofind));
  if ~isempty(subdirinfo)
    lastmod(K) = subdirinfo(1).datenum;
  end
end

[sortedmod, sortorder] = sort(lastmod);
sordorder(~isfinite(sortedmod)) = [];  %directories without data.mat

for K = 1 : length(sortorder)
  thisdirnum = sortorder(K);
  thisoutname = sprintf('file%d.xls', K);
  %copy from the subdirectory in to a sequentially named .xls file
  copy( fullfile( dirinfo(thisdirnum).name, filetofind ), thisoutname );
end

[John]

Hello Walter,

Thank you for your help with the code, it is of great assistance.

Should I have the Main folder containing the sub folder as the current directory?

I am getting an error

??? Undefined function or method 'copy' for

input arguments of type 'char'

Would you know why this is happening?

Kind regards

John

[Walter Roberson]

Sorry, should be copyfile()

[John]

Thanks for your time and help

[John]

Hello Walter,

Apologies for coming back to you again on this. The data in the excel is not readable. Would you know what is causing this?

Regards

John

[Walter Roberson]

Renaming a .mat file to be a .xls file extension does not make the .mat file readable to Excel.

You have omitted describing all the steps between locating an appropriate .mat file and saving the file as .xls . .mat files may have multiple variables in different formats, and may contain objects that *cannot* be represented in Excel, so there is no universal .mat to .xls reformatter.

[John]

Hello Walter,

Thank you, I understand. I used the command whos -file filename to view the contents of the .mat file. There are many classes of type struct and double in the files. However, I do not need them all. Would it possible to just save the variable "ess_plant_energy_in_total_simu" of class double to the excel file only?

If that is difficult to then saving all the variables of type double would helpful also.

Thank you

Regards

John

[Walter Roberson]

vartofind = 'ess_plant_energy_in_total_simu';
for K = 1 : length(sortorder)
 thisdirnum = sortorder(K);
 thisdirname = dirinfo(thisdirnum).name;
 thisoutname = sprintf('file%d.xls', K);
 try
 thisdata = load( fullfile(thisdirname,filetofind), vartofind );
 xlswrite( thisoutname, thisdata.(vartofind) );
 catch
 fprintf(2, 'File "%s/%s" does not have variable "%s\n", thisdirname, filetofind, vartofind);
 end
end

[John]

Hello Walter,

Thank you for your time, that works perfect.

Regards

John

[John]

Hello Walter,

You you helped me with this code 2 months ago and it is working fine.

Currently it writes one variable called "ess_plant_energy_in_total_simu" to the excel file.

I'm wondering how could I write another variable, called 'sch_cycle' to the excel file also?

I've tried to add to your code, but I'm not having much luck.

Thank you

John

vartofind = 'ess_plant_energy_in_total_simu';
vartofind1 = 'sch_cycle'; 
for K = 1 : length(sortorder) 
thisdirnum = sortorder(K); 
thisdirname = dirinfo(thisdirnum).name; 
thisoutname = sprintf('file%d.xls', K); 
try 
thisdata = load( fullfile(thisdirname,filetofind), vartofind, vartofind1 ); 
xlswrite( thisoutname, thisdata.(vartofind, vartofind1 ) ); 
catch 
fprintf(2, 'File "%s/%s" does not have variable "%s\n"', thisdirname, filetofind, vartofind, vartofind1 ); 
end 
end

[John]

Hi Walter,

Apologies for coming back to you again with this. Could you give me any advice on this?

Much appreciated.

Thanks

Regards

John

[Walter Roberson]

Only one variable can be written in one call to xlswrite(). You will need to make two xlswrite() calls with different range specifications. Unless, that is, both variables are vectors of the same size, in which case:

xlswrite(thisoutname, [thisdata.(vartofind)(:), thisdata.(vartofind1)(:)]);

[Image Analyst]

I don't know what length(sortorder) is, but if you have more than about 4 or 5 calls to xlswrite, it will take a very long time and you'd find it worthwhile to learn how to do it via ActiveX. It's not hard - I've posted demo code here before on how to do it - and it will be WAY faster since you won't have to launch and shutdown Excel each time.

[John]

Hi Walter,

The variables are vectors of different sizes. The first variable is n X 1 and the the second variable is m x 2. So this means I need two xlswrite() calls. Is this correct? - Do I need two lines for "thisdata =" and "fprint" also?

Thanks for your help

John

filetofind = 'data.mat';
dirinfo = dir();
dirinfo(~[dirinfo.isdir]) = []; %remove non-directories
tf = ismember( {dirinfo.name}, {'.', '..'});
dirinfo(tf) = []; %remove current and parent directory.
numsubdir = length(dirinfo);
lastmod = inf * ones(numsubdir,1);
for K = 1 : numsubdir
 subdirinfo = dir(fullfile(dirinfo(K).name, filetofind));
 if ~isempty(subdirinfo)
 lastmod(K) = subdirinfo(1).datenum;
 end
end
[sortedmod, sortorder] = sort(lastmod);
sordorder(~isfinite(sortedmod)) = []; %directories without data.mat
vartofind = 'ess_plant_energy_in_total_simu'; 
vartofind1 = 'sch_cycle'; 
for K = 1 : length(sortorder) 
thisdirnum = sortorder(K); 
thisdirname = dirinfo(thisdirnum).name; 
thisoutname = sprintf('file%d.xls', K); 
try 
thisdata = load( fullfile(thisdirname,filetofind), vartofind); 
thisdata1 = load( fullfile(thisdirname,filetofind), vartofind1); 
xlswrite( thisoutname, thisdata.(vartofind) ); 
xlswrite( thisoutname, thisdata1.(vartofind1) ); 
catch 
fprintf(2, 'File "%s/%s" does not have variable "%s\n", thisdirname, filetofind, vartofind); 
fprintf(2, 'File "%s/%s" does not have variable "%s\n", thisdirname, filetofind, vartofind1); 
end 
end

[Walter Roberson]

You can combine the load() into a single statement:

thisdata = load( fullfile(thisdirname,filetofind), vartofind, vartofind1 ); 
xlswrite( thisoutname, thisdata.(vartofind) );
xlswrite( thisoutname, thisdata.(vartofind1) ); 

However, you need to either write to different file names or else use range specifications on the xlswrite() -- otherwise the data from the second xlswrite() will overwrite the first.

[John]

Thanks very much for your help. I got it working.

Regards

John

Answer 4

[Frederic Moisy]

You can use rdir (=recursive dir), available hire: http://www.mathworks.com/matlabcentral/fileexchange/12180-fileseries-rename-renumber-files-recursive-directories

[John]

Hello Frederic,

Would you mind helping me code this for my application. I've been trying for hours with no luck.

Thank you

John

Answer 5

[Luke Melo]

Found this article very useful. Here's the code I use to select a folder from a dialogue window and find all the files, png here for example, within all subdirectories:

path = uigetdir('/Users/user/')
files = dir(fullfile(path,'**','*.png'));

[Walter Roberson]

We do, though, recommend against using path as a variable name, as path is an important MATLAB function.

Also note that files that results here will be a directory information structure. To get the file names you would use

filenames = fullfile({files.folder}, {files.name});

[Luke Melo]

interesting, I have no issues running the code without the cell {} brackets, but I'm a fan of consistent syntax

also you're right about the variable name path

[Walter Roberson]

The line I show with fullfile() is an extra step to extract fully qualified file names from the structure that is returned by dir() . When the '**' wildcard is used with dir() each different result might come from a different directory, and the same name might show up with respect to different directories, so it becomes important to put together the folder name and file name.

[lena kappa]

@Walter Roberson is there a way to save all those files in a new folder?

Answer 6

[Matthias]

Although this does not answer the whole question it adresses the title (which is what I was looking for): to get a list of all(!) subfolders below a specified folder or drive one could use the dos command like this:

 [s, r] = dos('dir X:\folder\ /s /b /A:D');
 folders = regexpi(r, '\n', 'split')';

Using the /s option the dos dir command will go down all the way (no option to specifiy a max recursion depth here - beware of long execution times!), with attribute D (/A:D) only directories will be returned and finally the /b option gives us a nice plain list (as a very long string) that can be easily split for further use.

Edit:

 [s, r] = dos('dir X:\folder\*.mat /s /b');
 matFiles = regexpi(r, '\n', 'split')';

will obviously give you the list of all .mat files, the date of these can easily be obtained with another dir() call:

 l = cellfun(@dir, matFiles);
 changeTimes = [l.datenum]';

[Guillaume]

This obviously only works on Windows.

Note that since R2016b, the built-in matlab dir function supports recursive search, no need to delegate to the OS:

d = dir('X:\folder\**');
d = d([d.isdir]);

This can be (very) slow if the subfolders have a lot of files since the first line returns all files and folders.

[Coco]

Can you please help me with coding to put those files into the new folder?

thank you.

[Guillaume]

What files? What new folder?

It doesn't look like your question has much to do with the question being answered here, so please start your own question. Give as many details as possible about what you're trying to do.

[Walter Roberson]

https://www.mathworks.com/matlabcentral/answers/472584-how-to-put-a-struct-files-into-a-new-folder is the relevant Question.

Answer 7

[Ba Mo]

as lots of users reported above, new versions of matlab support the following command dir('**/*.mat');

However, old versions of matlab don't support this

instead of writing a large code, inspect the structure field "isfield" and so on, you could just easily ask DOS (or the command prompt) to do it for you. the output from MS-DOS isn't formatted, so you need to split the one block string to separate lines

newline = char(10); %char(10) is the character for line-break, or "enter"
[~,all_mats] = system('dir /s /b *.mat'); %you can also simply write: !dir /s /b *.mat
all_mats = strsplit(all_mats,newline)';
all_mats(cellfun(@isempty,all_mats))=[]; %the last entry/line might be an empty cell. delete it.