How to call data from files named with dynamic index

I know that a similar question has been turned down many times as we don't EVER want to use dynamic vaiable names in programming. The fact is that I'm trying to make my life easier when plotting measurement results. The files are named after the sample as in sample001.csv or SampleName_<Date>.txt (and I cannot change the export or naming format). All I want to do is:

% for each of these text files, plot a set column as x and another as y
figure
hold on
% k is a high number and the number of files to import for plotting
for n = 1:k
    plot(sample<n>(:,14),sample<n>(:,17))
end

I can think of two ways to do this:

1. Break the Rules of Matlab and somehow implement the dynamic variable name anyway (since the files per default imprt with the file name as variable name). I've tried some things with num2str or putting a char or string variable into plot but couldn't get it to work
2. Get the files to import to a multidimensional (in my case 3D) array or table from which I can call the data in a conventional method

I presume the second will be the experts' choice, therefore, how can I

• select a great number of files for importing (best from a Windows Explorer "open a file" dialog)
• forget the header in the files (it would be nice to keep the column names but it makes accessing the rest of the data so much more complicated so I'd rather scrap the header and import the actual data as numeric matrix)
• make sure the decimal separator is correct
• import them in a way that I can access the data of "sample345.txt" as array(:,:,345) or something like that
• get the whole thing to be robust so that it works if the data has different (arbitrary) height (and possibly width) in each file, contains strings as well as numbers or files are missing

Any help much appreciated.

______

PS: I've come up with something similar already but the problem is that it only works for numeric data with no header etc. and overwrites the variable every time. Sorry for the syntax, I'm not a programmer

for k=(1:3:72)
	textFileName = ['sample_' num2str(k) '.txt'];
	textData = readmatrix(textFileName);
	sz = size(textData);
	x = sz(1);
	dehnung = textData(:,4);
	spannung = textData(:,5);
	displacement = textData(:,10);
	length = textData(1,17);
	z=displacement(1);
	for i = 1:x
		displacement(i) = displacement(i) - z;
		dehnung(i) = displacement(i)/(length(1)*1000)*100;
	end
	
	plot(dehnung,spannung)
	hold on
end

[Peter Perkins]

What about (3) read each file into a field of a scalar struct named for the file?

Answer 1

[Stephen23]

"select a great number of files for importing (best from a Windows Explorer "open a file" dialog)"

If you really have a "great number" of files your user is going to dislike this way of selecting them. Most likely simply selecting the folder and using dir with a filename filter would be simpler and less fiddly to use. But each to their own...

[C,P] = uigetfile(..,'MultiSelect','on');
N = numel(C);
D = cell(1,N);
for k = 1:N
    F = fullfile(P,C{k});
    M = .. whatever file importing function you want to use
    .. whatever processing you want to do
    D{k} = M;
end

You can access the filenames and filedata using basic indexing, e.g. for the third file:

C{3} % filename
D{3} % imported data

https://www.mathworks.com/help/matlab/matlab_prog/access-data-in-a-cell-array.html

A simple alternative is to import the data into the same non-scalar structure as dir returns, e.g.:

P = 'absolute/relative path to the folder where the files are saved';
S = dir(fullfile(P,'sample_*.txt'));
for k = 1:Nnumel(S)
    F = fullfile(P,S(k).name);
    M = .. whatever file importing function you want to use
    .. whatever processing you want to do
    S(k).data = M;
end

[H Mueller]

Great, the wavy parentheses did the trick. I find it very confusing when to use which kind. Thank you!

Here is the solution I derived from your remarks. It outputs an array that contains the data and a string array with the corresponding sample names. Calculations, plotting etc. can be done at the same time.

% Select files to import
[files,path] = uigetfile('D:\filepath\*.*','MultiSelect', 'on');
% Separate file name from extension (optional, could do with "files")
i = 1;
for fn = files
[pathx,namex,extx] = fileparts(append(path,"\",fn));
sample(i) = cellstr(namex);
i = i+1;
end
% Import data to struct
for n = 1:length(files)
    data{n} = readtable([path,files{n}]);
end
% Do things with the data (in my case plot)
for k = 1:length(files)
    figure
    hold on
    plot(data{k}{:,19},data{k}{:,28:end})
end
clearvars -except sample data 

[Stephen23]

Do NOT use the variable path as this will shadow the important inbuilt path function.

You don't need to add the filepath just to remove it again with fileparts.

I recommend looping over indices rather than looping over data values, it is usually simpler.

I strongly recommend that you preallocate the cell arrays before the loops.

I strongly recommend that you use fullfile (the correct tool) instead of concatenating strings.

[C,P] = uigetfile('D:\filepath\*.*','MultiSelect', 'on');
N = numel(C);
D = cell(N,1); % preallocate
[~,S] = fileparts(C); % sample
% Import data to cell array:
for k = 1:N
    F = fullfile(P,C{k});
    D{k} = readtable(F);
end

Answer 2

[Steven Lord]

%{
% for each of these text files, plot a set column as x and another as y
figure
hold on
% k is a high number and the number of files to import for plotting
for n = 1:k
    plot(sample<n>(:,14),sample<n>(:,17))
end
%}

Where are you importing the files? Before you create the figure (where the first comment is located) or inside the for loop (omitted to make the example simpler?)

Do you need to store all the data in memory at once?

If you don't need all the data in memory at once, you don't need to store the data in different named variables. In this example I'm reusing the variable y in the loop.

% Sample data
x = 0:360;

% Set up figure
figure
axis([0 360 -1 1]);
hold on

% Y will simulate the data loaded from your files
for k = 1:5
    y = sind(k*x);
    h = plot(x, y);
end
shg

If you do need all the data at once, and each data set should be associated with its filename, consider using a struct or something similar.

x = 0:360;
axis([0 360 -1 1]);
hold on

% Create struct to hold data
allData = struct('multiplier', {}, 'yData', {}, 'handle', {});

for k = 1:5
    y = sind(k*x);
    h = plot(x, y);

    allData(k).multiplier = k;
    allData(k).yData = y;
    allData(k).handle = h;
end

disp(allData(4))

    multiplier: 4
         yData: [1×361 double]
        handle: [1×1 Line]

[H Mueller]

Doesn't really matter whether I first import all files and then have a bunch of variables to deal with or import it to the struct right away. At some point I need to extract the index and convert it into a 3rd dimension column or struct array field. And it would be nice to have all data in memory to then be able to plot one graph again with modifications without having to touch the rest (i.e. run the whole script again).

I can see that a struct (of which I didn't know it existed) would be an adequate fomat to keep the date. The problem persists that I don't know how to store the data into it.

For simplicity let's say the data is already imported into tables: data1 (27x35), data2 (303x35), and data3 (45x35). In this case the header has already been omitted (i.e. converted into the column names) and the first column is some arbitrary string; the rest is the numerical data I need. If we now want to put that into the struct we stand before the same problem (or, instead of putting it into the struct could do the plotting right away):

% Create struct to hold data
allData = struct('ColumnHeaders', {}, 'RowHandle', {}, 'Data', {});
% transfer data to struct
for K = 1:3
    allData(K).ColumnHeaders = dataK.Properties.VariableNames; %here I need to call dataK, which of course I can't do beacuse it is dynamically named
    allData(K).RowHandle = dataK{:,1};
    allData(K).Data = dataK{:,2:end};
end

I am also not sure if I understand the concept of how a structure works correctly.

[Stephen23]

"For simplicity let's say the data is already imported into tables: data1 (27x35), data2 (303x35), and data3 (45x35). "

That isn't "simplicity", that is the hard way. By having those separate variables in the workspace you force yourself into writing slow, complex, inefficient code to access them.

Which is why the MATLAB documentation shows the efficient approach of importing into one array:

https://www.mathworks.com/help/matlab/import_export/process-a-sequence-of-files.html

You should do something similar: use indexing in a loop to assign the imported data into one array (which could be a container type, e.g. a cell array, structure array, etc.) at the point when the file data is being imported. Messing around with variable names is not something you want to get into.

[Peter Perkins]

Try a using scalar struct:

filenames = ["sample001" "sample002"];
for fn = filenames
    % readtable(fn+".csv",...)
    t = array2table(rand(5,2));
    s.(fn) = t;
end

for i = filenames
    % do something to each table
    size(s.(fn))
end

You might ask, "how is this different than putting each table in the workspace?", and the answer is that you never need to eval anything. s.(fn) gets you each table. You can even iterate over them using structfun.

[Steven Lord]

You may need to use matlab.lang.makeValidName or perform some other processing to create the struct field name if the filenames are not valid MATLAB identifiers.

name = ["sample 001"; "grades (MA101 Fall 2020)"];
matlab.lang.makeValidName(name)

ans = 2×1 string array
    "sample001"
    "grades_MA101Fall2020_"

[H Mueller]

I don't understand how to feed the data into the struct. Somehow the fields need to be named beforehand and I can't name them after every entry of a string array? Here is what I have so far:

% Select files to import
[files,path] = uigetfile('D:\Path\*.*','MultiSelect', 'on');
% Separate file name from extension (is there a simpler way?)
i = 1;
for fn = files
[pathx,namex,extx] = fileparts(append(path,"\",fn));
file(i) = cellstr(namex);
i = i+1;
end
% Import Data to struct
AllData = struct
for n = (1:length(file))
    import = readtable([path,files{n}]);
    SampleName = file{n};
    ColumnHeaders = import.Properties.VariableNames;
    RowNames = (import{:,1});
    Data = table2array(import(:,2:end));
    AllData.(SampleName) = struct(ColumnHeaders{:}; RowNames{:}, Data(:)); % please fix this line
end

(Why is it an such an absolute mess if I put import into the struct? That would be way easier, but it puts every property associated with the table into a separate field.)

[EDIT]: Stephen proposed a simple way to do precisely that. Problem solved. Thanks for your answers!