Data is not storage when running loop

hello community

please help me to understand why data does not show when I run a loop:

I have 2 files: @Data01.mat and @Data05.mat I want to extrac data points when conditions are met'

if I run my script with Data1 all the parameters are there:

figure;plot (Epm_nEngallc0_4, IKCtl_au16_0allc, '.');

but if I run my script with Data1&2 the output is different and I get less data points

I would expect to have all data poins from @Data01.mat plus @Data05.mat

please help me to see what am I doing wrong.

clear all;              %   
clc;                    %   

fprintf('select folder containing data files\n');

currentpath=pwd; %%

datapath=uigetdir('','Select Data Folder'); %%

if (datapath == 0)
    fprintf('Data directory was not selected...script will be terminated\n\n');
    return
end

prompt = 'Do you want to export data as .xlsx?';
save_data_excel = questdlg(prompt,'Save','Yes','No','Yes');

cd(datapath); %% 
files=dir('@*.mat'); %% 
num_files=length(files(:,1)); %% 
       
for i=1:num_files
    fprintf('Progress: %d/%d\n\n',i, num_files)
    load(files(i,1).name)
        
 for j=1:length(time)
    
     if Epm_nEng(j) > 0 && IKCtl_DiagRefLvlMinErr_au16__ls_1_rs_(j) > 0.165 && IKCtl_DiagRefLvlMinErr_au16__ls_3_rs_(j) > 0.165%% sensor signal
         Epm_nEngallc1_3(j,1) = Epm_nEng(j); %% get engine speed if: engine speed is higher than 0 and if  sensor signal > 0.165
     end
     if Epm_nEng(j) > 0 && IKCtl_DiagRefLvlMinErr_au16__ls_2_rs_(j) > 0.165 && IKCtl_DiagRefLvlMinErr_au16__ls_5_rs_(j) > 0.165
         Epm_nEngallc2_5(j,1) = Epm_nEng(j);
     end
     if Epm_nEng(j) > 0 && IKCtl_DiagRefLvlMinErr_au16__ls_0_rs_(j) > 0.165 && IKCtl_DiagRefLvlMinErr_au16__ls_4_rs_(j) > 0.165
         Epm_nEngallc0_4(j,1) = Epm_nEng(j);
     end
     
     if Epm_nEng(j) > 0 && IKCtl_DiagRefLvlMinErr_au16__ls_1_rs_(j) > 0.165 && IKCtl_DiagRefLvlMinErr_au16__ls_3_rs_(j) > 0.165
         IKCtl_au16_1allc(j,1)= IKCtl_DiagRefLvlMinErr_au16__ls_1_rs_(j); %% get sensor signal if: engine speed is higher than 0 and if  sensor signal > 0.165
         IKCtl_au16_3allc(j,1)= IKCtl_DiagRefLvlMinErr_au16__ls_3_rs_(j);
     end
     if Epm_nEng(j) > 0 && IKCtl_DiagRefLvlMinErr_au16__ls_2_rs_(j) > 0.165 && IKCtl_DiagRefLvlMinErr_au16__ls_5_rs_(j) > 0.165
         IKCtl_au16_2allc(j,1)= IKCtl_DiagRefLvlMinErr_au16__ls_2_rs_(j);
         IKCtl_au16_5allc(j,1)= IKCtl_DiagRefLvlMinErr_au16__ls_5_rs_(j);
     end
     if Epm_nEng(j) > 0 && IKCtl_DiagRefLvlMinErr_au16__ls_0_rs_(j) > 0.165 && IKCtl_DiagRefLvlMinErr_au16__ls_4_rs_(j) > 0.165
         IKCtl_au16_0allc(j,1)= IKCtl_DiagRefLvlMinErr_au16__ls_0_rs_(j);
         IKCtl_au16_4allc(j,1)= IKCtl_DiagRefLvlMinErr_au16__ls_4_rs_(j);
     end
 
     end
     
     end           

[Epsilon]

Hi, Can you please share the datafiles. It would help us understand the issue better.

[Image Analyst]

Looks like you blew right past the posting guidelines. If you have any more questions, then attach your 3 data mat files (data01, data02 and data05) with the paperclip icon after you read this:

TUTORIAL: How to ask a question (on Answers) and get a fast answer

It's not obvious from your two plots that data05 has fewer points in it than data01 combined with data02.

And how are you plotting? I don't see plot() or scatter() in your code.

And put in some comments! We have no idea what all those variables mean. Help us to help you.

And don't use cd():

https://matlab.fandom.com/wiki/FAQ#Where_did_my_file_go?_The_risks_of_using_the_cd_function.

Use fullfile() instead.

[A-Rod]

hello Thank you for your comments.

I thought I sent the files but I didn't, Now you can find them in the post.

data is comming from an engine and one of its sensors, I'm running a test with different conditons so I may have several files.

I want to process all data files with matlab and plot engine speed Vs sesnro signal,

plot (Epm_nEngallc0_4, IKCtl_au16_0allc, '.') to see how the sensor behavies.

any feedback please let me know it will be highly appreciated

Answer 1

[Epsilon]

Hi A-Rod,

On running the original code with both the data files, the number of data points generated is dependent on the highest value of ‘length(time)’. If you load onlyData01.mat, you get 5092 data points, and withData05.mat, you get 6167. However, when both files are used, the total number of data points ends up being 6167. This happens because the first 5092 datapoints are overwritten due to the same index(j) being passed in the second iteration.

Also when using just the Data01.mat file the output variables have a wider range. When both the files are used the higher values of the variables are overwritten based on the If statements, and then the variables have a narrower range. Thus, the plot using both the files has lower values on the axes. The no. of datapoints is still higher.

If you want to add all the datapoints consider appending the variable values after the existing index and not overwriting them, it can look something like this:

% Initialize arrays outside the loops to store results 
Epm_nEngallc1_3 = []; 
Epm_nEngallc2_5 = []; 
Epm_nEngallc0_4 = []; 
IKCtl_au16_1allc = []; 
IKCtl_au16_3allc = []; 
IKCtl_au16_2allc = []; 
IKCtl_au16_5allc = []; 
IKCtl_au16_0allc = []; 
IKCtl_au16_4allc = [];  
for i = 1:num_files 
    fprintf('Progress: %d/%d\n\n', i, num_files); 
    load(files(i, 1).name);  
    for j = 1:length(time) 
        if Epm_nEng(j) > 0 && IKCtl_DiagRefLvlMinErr_au16__ls_1_rs_(j) > 0.165 && IKCtl_DiagRefLvlMinErr_au16__ls_3_rs_(j) > 0.165 
            Epm_nEngallc1_3 = [Epm_nEngallc1_3; Epm_nEng(j)]; 
            IKCtl_au16_1allc = [IKCtl_au16_1allc; IKCtl_DiagRefLvlMinErr_au16__ls_1_rs_(j)]; 
            IKCtl_au16_3allc = [IKCtl_au16_3allc; IKCtl_DiagRefLvlMinErr_au16__ls_3_rs_(j)]; 
        end 
        if Epm_nEng(j) > 0 && IKCtl_DiagRefLvlMinErr_au16__ls_2_rs_(j) > 0.165 && IKCtl_DiagRefLvlMinErr_au16__ls_5_rs_(j) > 0.165 
            Epm_nEngallc2_5 = [Epm_nEngallc2_5; Epm_nEng(j)]; 
            IKCtl_au16_2allc = [IKCtl_au16_2allc; IKCtl_DiagRefLvlMinErr_au16__ls_2_rs_(j)]; 
            IKCtl_au16_5allc = [IKCtl_au16_5allc; IKCtl_DiagRefLvlMinErr_au16__ls_5_rs_(j)]; 
        end 
        if Epm_nEng(j) > 0 && IKCtl_DiagRefLvlMinErr_au16__ls_0_rs_(j) > 0.165 && IKCtl_DiagRefLvlMinErr_au16__ls_4_rs_(j) > 0.165 
            Epm_nEngallc0_4 = [Epm_nEngallc0_4; Epm_nEng(j)]; 
            IKCtl_au16_0allc = [IKCtl_au16_0allc; IKCtl_DiagRefLvlMinErr_au16__ls_0_rs_(j)]; 
            IKCtl_au16_4allc = [IKCtl_au16_4allc; IKCtl_DiagRefLvlMinErr_au16__ls_4_rs_(j)]; 
        end 
    end 
end 

This would be the output of plot (Epm_nEngallc0_4, IKCtl_au16_0allc, '.'); if you append them (more datapoints in the (500-2000, 0-5) region):

Hope this helps you understand the reasons behind what was happening and how to go ahead!

[A-Rod]

I've been learing by my self, getting help from the community helps me a lot.

Thank you for taking time to explain it. it worked

Answer 2

[Walter Roberson]

You set your various arrays conditionally. We should not be surprised if the arrays come out different lengths.

You do not clear the arrays between for i iterations. The arrays are going to stay the same size as they were set in the previous iteration, with entries going unused and maintaining their previous values, unless it just so happens that the second round happens to need more entries in the arrays.

Overall, you should expect chaos in the size of the arrays.