How to find the index value of an element in a cell array?
Show older comments
Hello,
I have tried by all means finding an example for my case but I only got frustration, so I hope you can help me.
Context: I am using a Differential Evolution Algorithm and as a part of the results I calculate the mean, sd and also the best OF value (in this case, the minimum value) of each generation (iteration). I run the algorithm 6 times and I obtain 6 data.txt files (runde*.txt) containing the following information: iteration, OF value, parameters values (2 params.) and the run number (i.e., if it is the 1st, 2nd, 3rd etc run). Each file contains 5 columns (named before) and 50 rows. This is the code I am using:
files = dir('runde*.txt');
n=length(files);
for i = 1:n
filename = sprintf('runde%d.txt', i);
dedata{i} = importdata(filename);
end
m=length(dedata{i}.data);
fileSTATS=fopen('destats.txt', 'w');
MOF=[]; % matrix in which columns contain best OF values (S_bestval.FVr_oa)...
...per generation of every run
for j=1:m
for i=1:n
MOF(j,i)=dedata{1,i}.data(j,2);
mean_MOF(j,1)=mean(MOF(j,:));
sd_MOF(j,1)=std(MOF(j,:));
best_MOF(j,1)=min(MOF(j,:));
end
fprintf(1,'Gen.: %d,\t mean: %.6e,\t SD: %.6e,\t best: %.6e\n',j,mean_MOF(j),sd_MOF(j),best_MOF(j));
fprintf(fileSTATS,'%d\t %e\t %e\t %e\r\n',j,mean_MOF(j),sd_MOF(j),best_MOF(j));
end
fclose(fileSTATS);
MOF stores the OF values of every file (which are in column 2).
My goal is to store (besides the data indicated in the code) in the file destats.txt the data (OF parameters and run number) associated to each value of the vector best_MOF (a vector which collects the best OF value of every generation off all runs of the algorithm).
I suspect it is a question of finding the position of every best_MOF(j,1) and indicating the data to store according to the index of every element best_MOF(j,1). That is the reason for the question. I don't know if it the correct question for this problem. I hope as well I have explained my problem clearly.
I include files in case they are needed. Note (Ed.): As I explained above, the goal is to write next to best_MOF column in destats.txt file the corresponding values in runde*.txt files. Note that the array best_MOF(j,1) does not operate values, it only selects the minimun value of the 2nd column of all six runde*.txt files row by row. Therefore, the 5th, 6th and 7th columns of destats.txt will be the so-named corresponding values of 3rd, 4th and 5th columns of runde*.txt files.
I truly appreciate any advise or solution for this problem. I have little experience with programming.
Thank you very much.
Accepted Answer
Adam Drake
on 13 Mar 2023
Edited: Adam Drake
on 13 Mar 2023
Had to do quite a bit of imagining without the starting data files, but I think this will set you straight. Let me know if you have questions or if it doesn't work.
EDIT: fixed case where length of data set in runde*.txt files are different for each file. Console shows dataset length but it is not ouput to file. Easily added if needed. Add attached "runde7.txt" to directory and run to see functionality. runde7 has 52 rows instead of 50. Note that MOF will contain "NaN" on files with datasets less than max(len). Account for NaN's with 'omitnan' flag on mean, std, and min functions.
EDIT2: Added best parameters to console and file output
EDIT3: Made per run instead of per file.
clc, clear all
files = dir('runde*.txt');
n = length(files);
for i = 1:n
filename = sprintf('runde%d.txt', i);
dedata{i} = importdata(filename);
len(i) = length(dedata{i}.data);
end
m = max(len);
MOF = nan(n,m); % matrix in which columns contain best OF values (S_bestval.FVr_oa)...
...per generation of every run
for i = 1:n % number of files
for j = 1:len(i) % length of data
MOF(i,j)=dedata{i}.data(j,2); % OF Values stored in column 2 of data
end
end
mean_MOF = mean(MOF,1,'omitnan'); % row vector with mean of each column (run)
sd_MOF = std(MOF,0,1,'omitnan'); % row vector with std of each column (run)
[best_MOF,best_idx] = min(MOF,[],1,'omitnan'); % row vector with min of each column (run)
p1 = zeros(1,m);
p2 = zeros(1,m);
run = zeros(1,m);
fileSTATS=fopen('destats.txt', 'w');
for j = 1:m
p1(j) = dedata{best_idx(j)}.data(j,3);
p2(j) = dedata{best_idx(j)}.data(j,4);
run(j) = dedata{best_idx(j)}.data(j,5);
fprintf(1,'Gen.: %d,\tmean: %.6e,\tSD: %.6e,\tBest: %.6e,\tp1: %.6e\tp2: %.6e\trun: %d\n', ...
j,mean_MOF(j),sd_MOF(j),best_MOF(j),p1(j),p2(j),run(j));
fprintf(fileSTATS,'%d\t %e\t %e\t %e\t %e\t %e\t %d\r\n', ...
j,mean_MOF(j),sd_MOF(j),best_MOF(j),p1(j),p2(j),run(j));
end
fclose(fileSTATS);
17 Comments
Anibal Marrero
on 13 Mar 2023
Adam Drake
on 13 Mar 2023
Seems to work perfectly. Results are:
console
Gen.: 1, mean: 3.379283e-01, SD: 5.187342e-01, best: 1.857550e-02
Gen.: 2, mean: 2.695424e-01, SD: 1.998604e-01, best: 2.000687e-01
Gen.: 3, mean: 8.355956e-01, SD: 5.606049e-01, best: 3.792330e-01
Gen.: 4, mean: 4.188311e-01, SD: 4.805082e-01, best: 1.374399e-01
Gen.: 5, mean: 3.619961e-01, SD: 3.505464e-01, best: 2.037424e-01
Gen.: 6, mean: 3.479943e-01, SD: 1.178959e-01, best: 1.842626e-01
destats.txt
1 3.379283e-01 5.187342e-01 1.857550e-02
2 2.695424e-01 1.998604e-01 2.000687e-01
3 8.355956e-01 5.606049e-01 3.792330e-01
4 4.188311e-01 4.805082e-01 1.374399e-01
5 3.619961e-01 3.505464e-01 2.037424e-01
6 3.479943e-01 1.178959e-01 1.842626e-01
Anibal Marrero
on 13 Mar 2023
Adam Drake
on 13 Mar 2023
See edit for file length correction.
Adam Drake
on 13 Mar 2023
See edit for best parameters addition.
Anibal Marrero
on 13 Mar 2023
Adam Drake
on 13 Mar 2023
I think you need to check your numbers. try, adding "clc, clear all" to the top of your script to make sure you're running from scratch.
To clarify:
column 2 of data for each file is stored in MOF.
The mean of column 2 of data for each file is stored in mean_MOF
The std of column 2 of data for each file is stored in sd_MOF
The minumum value of column 2 of data for each file is stored in best_MOF
best_param contains column 3,4,5 of data for each file at the index where the minimum value of column 2 was found.
best_idx: the index at the minimum value for files 1-6 are
48
49
27
35
42
48
Adam Drake
on 13 Mar 2023
I see now. You don't need the data per file, you need it across files. Working on it.
Anibal Marrero
on 13 Mar 2023
Adam Drake
on 13 Mar 2023
fixed. Because you were showing 6 rows, I figured the stats were per file instead of per run across all files. Should be what you want now.
Anibal Marrero
on 13 Mar 2023
Edited: Anibal Marrero
on 13 Mar 2023
Adam Drake
on 13 Mar 2023
Edited: Adam Drake
on 13 Mar 2023
Are you scrolling to the right to see the full output? Because I have added the two parameters and run number (columns 3, 4, and 5) to the console output and file output and I'm getting the numbers you were looking for.
If you wanted to have all columns from column 3 on output to one variable you would use:
endColumn = size(dedata{j}.data,2); remainingColumns(j,:) = dedata{best_idx(j)}.data(j,3:endColumn)
Questions 1 and 2 and 3 go together. I was accounting for the possibility that you may have a different number of generations for each run (i.e. one may have 50, another may have 52). In your initial script they would have only read whatever the size of the last file was. NaN(n,m) preallocates an array that is the number of files by maximum number of data points in any file. If you have 17 files and all of your files have 50 generations then it will be 17x50 and no cell will have NaN. If one of those files has 52 generations then it will be 17x52 and every row except 1 will have 2 columns of NaNs (not a number). The omitnan flag will ignore the NaNs when computing mean, std, and min. If I had preallocated with zeros then those calculations would be incorrect since you'd be adding 2 zeros to each dataset except the one with 52 generations. It is probably the case that all of your files will have the same number of generations but this will account for them either way.
For the last question, functions like min and max can return the index of the value it finds. So [best_MOF,best_idx] = min(MOF,[],1,'omitnan'); find the minimum value across each generation and doesn't consider NaNs and returns the minimum value and the index that it was found. Since MOF is file number × generation number in size and the min function specifies the operation is done per column, what is returned is a row vector that is 1×50 where each cell is the minimum for that generation along with the index, or file number, where that value was found.
Anibal Marrero
on 14 Mar 2023
Walter Roberson
on 14 Mar 2023
for k=1:min(size(best_params))
You do not want that. You are you using k to index columns, so you want k = 1 : size(best_params,2)
Anibal Marrero
on 14 Mar 2023
You're trying to do mean and std and min before you've fully populated MOF. you have to finish that loop first. Please accept my answer.
clc, clear all
files = dir('runde*.txt');
n = length(files);
for i = 1:n
filename = sprintf('runde%d.txt', i);
dedata{i} = importdata(filename);
len(i) = length(dedata{i}.data);
end
m = length(dedata{i}.data);
MOF = []; % matrix in which columns contain best OF values (S_bestval.FVr_oa)...
...per generation of every run
% YOU HAVE TO ALLOCATE MOF WITH VALUES BEFORE YOU CAN TAKE THE MEAN,STD,MIN
for i = 1:n % number of files
for j = 1:len(i) % length of data
MOF(i,j)=dedata{i}.data(j,2); % OF Values stored in column 2 of data
end
end
mean_MOF = mean(MOF); % row vector with mean of each column (run)
sd_MOF = std(MOF); % row vector with std of each column (run)
[best_MOF,best_idx] = min(MOF); % row vector with min of each column (run)
fileSTATS=fopen('destats.txt', 'w');
for j = 1:m
best_params(j,:) = dedata{best_idx(j)}.data(j,3:end-1);
best_run(j) = dedata{best_idx(j)}.data(j,end);
fprintf(1,'Gen.: %d,\tmean: %.6e,\tSD: %.6e,\tBest: %.6e,\t',...
j,mean_MOF(j),sd_MOF(j),best_MOF(j));
fprintf(1,'Parameters:\t');
for k = 1:size(best_params,2) % 2 specifies number of columns
fprintf(1,'%.6e\t',best_params(j,k));
end
fprintf(1,'Run:\t%d\r\n',best_run(j))
fprintf(fileSTATS,'%d\t %e\t %e\t %e\t', ...
j,mean_MOF(j),sd_MOF(j),best_MOF(j));
for k = 1:size(best_params,2) % 2 specifies number of columns
fprintf(fileSTATS,'%e\t',best_params(j,k));
end
fprintf(fileSTATS,'%d\r\n',best_run(j));
end
fclose(fileSTATS);
Anibal Marrero
on 14 Mar 2023
Edited: Anibal Marrero
on 14 Mar 2023
More Answers (0)
Categories
Find more on Matrices and Arrays in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
