Loop to replace outliers with NaN extremely slow

I want to replace outliers with NaN in a large table (> 3 standard deviations from each column's mean) and my code works in principle but is incredibly slow, i.e. still not done after 10 minutes. The table size is about 2000x150. Is there a faster way, maybe without the loop, and could someone tell me what is wrong with my version?

%Version 1: loop through column names
var_list = mytable.Properties.VariableNames(4:140)
for i = 1:length(var_list)
    mytable.(var_list{i}) = filloutliers(mytable.(var_list{i}),nan,'mean','ThresholdFactor', 3)
end

%Version 2: loop through column indices
for i = 4:140
    mytable(:,i) = filloutliers(mytable(:,i),nan,'mean','ThresholdFactor', 3)
end

[Mathieu NOE]

hello Tanja

just a question : is removing the outliers the "real" need or a smoothing approach would also fit your needs ?

[TL]

Hi Mathieu, yes I do need to replace them with NaN, some values are real errors so they can be 10 times higher than the mean and need to be filtered out

Answer 1

[Star Strider]

The way the table addressing is coded is likely the problem.

I’m not certain, however using parentheses () addresses the table (or variables as individual table arrays), while curly braces {} address the variable contents themselves.

So for example

mytable(:,i) =

creates a new table as ‘mytable’ while

mytable{:,i} =

addresses only the contents of the variable.

See the documentations ection on Access Data in Tables for details.

Again, I’m not certain wht the problem is, however experimenting with changing the addressing method could provide a solution.

Also, I’m not certain if the loop is even necessary, since filloutliers appears to work on arrays as well as vectors, and operates on each column separately, according to the documentation.

.

[TL]

Thanks a lot, this was indeed much faster! I'm still having trouble with choosing the right way of accessing data with brackets etc. so the link is helpful.

The issue with dropping the loop is that I need to specify the columns where filloutliers is applied. Some columns contain text or should not be modified. If anyone knows a way to do that without looping that would be great!

[Star Strider]

As always, my pleasure!

If the column references of the non-numeric are known, the numeric variables (columns) only can be selected. See the documentation section on A and under Data Options the DataVariables information (there is no direct link to it). That will allow you to select the correct variables. And no loops necessary! The filloutliers function can do all this itself, internally.

.

[TL]

Perfect, now I could fix it, thanks so much! If anyone else has the same problem, this works without a loop and within seconds (outliers = 3 standard deviations from mean, any number can be picked here):

k = mytable.Properties.VariableNames % Then delete cells of k that should not be outlier corrected
mytable_filtered = filloutliers(mytable(:,k),nan,'mean','ThresholdFactor', 3)

[Star Strider]

As always, my pleasure!