How to remove Outliers

Hi everyone. Does anyone know how to detect and remove the outliers? I've tried man ways but I can't. I have to remove the data by Excel but I think that's not enough. Thank youuuu

Answer 1

[Mathieu NOE]

hello

here is a quick result after a few manipulations

code

clc
clearvars

T = readtable('AskingData.xlsx');

P = T.Power;
Ws = T.WingSpeed;

% remove duplicates 
[Ws_unique,IA,IC] = unique(Ws);
P_unique = P(IA);

% remove negative Power
ind = find(P_unique<0);
P_unique(ind) = [];
Ws_unique(ind) = [];

% remove outliers
[P_unique2,TF] = rmoutliers(P_unique,'movmedian',100);
Ws_unique2 = Ws_unique(~TF);

% let's add some smoothing
P_unique2S = smoothdata(P_unique2,'sgolay',151);

plot(Ws_unique,P_unique,'b*',Ws_unique2,P_unique2,'r--',Ws_unique2,P_unique2S,'g');
legend('unique','unique and outliers removed','smoothed');

[Huy Cao]

It's said

Undefined function or variable 'rmoutliers'.

Error in removeoutliers (line 14)

[P_unique2,TF] = rmoutliers(P_unique,'movmedian',100);

T.T

[Mathieu NOE]

OK

I guess you are running an older matlab release (I have R2020b)

[Huy Cao]

i use R2018a :(

[Mathieu NOE]

well maybe we can achieve a reasonnable plot even without rmoutliers

clc
clearvars

T = readtable('AskingData.xlsx');

P = T.Power;
Ws = T.WingSpeed;

% remove duplicates 
[Ws_unique,IA,IC] = unique(Ws);
P_unique = P(IA);

% remove negative Power
ind = find(P_unique<0);
P_unique(ind) = [];
Ws_unique(ind) = [];

% smooth the data
P_uniqueS = smoothdata(P_unique,'movmedian',80);

plot(Ws_unique,P_unique,'b*',Ws_unique,P_uniqueS,'r');

legend('unique','outliers removed');

[Huy Cao]

thank you but i think the outliers is the dot far from the shape ^^

[Mathieu NOE]

yes sure

now I just realised that the number of outliers are maybe not so high and that a simple smoothing was good enough

IMHO, the last result is pretty much the same as the previous (when I used rmoutliers)

now I know that I have somewhere another smoothing function with "robust" outliers removal built in; I will try this one as well

[Mathieu NOE]

this is an alternative using smoothn (from FEX : smoothn - File Exchange - MATLAB Central (mathworks.com) - attached also for convenience

the difference is quite minor , more prononced on the tail section, but that depends also what smothing factor is used in each scenario.

clc
clearvars

T = readtable('AskingData.xlsx');

P = T.Power;
Ws = T.WingSpeed;

% remove duplicates 
[Ws_unique,IA,IC] = unique(Ws);
P_unique = P(IA);

% remove negative Power
ind = find(P_unique<0);
P_unique(ind) = [];
Ws_unique(ind) = [];

% smooth the data
P_uniqueS = smoothdata(P_unique,'movmedian',80);
P_uniqueS2 = smoothn(P_unique,1e4,'robust'); % see FEX : 

plot(Ws_unique,P_unique,'b*',Ws_unique,P_uniqueS,'r',Ws_unique,P_uniqueS2,'g');

legend('unique','smoothdata','smoothn');

[Mathieu NOE]

attached the smoothn function (FYI)

[Mathieu NOE]

it's me again !

I tried another smoothing function (irlssmooth) found on the file exchange :

Robust Least-Squares Smoother - File Exchange - MATLAB Central (mathworks.com)

this one is also very good with outliers

see the benefit vs built in smoothdata from TMW

code :

clc
clearvars

T = readtable('AskingData.xlsx');

P = T.Power;
Ws = T.WingSpeed;

% remove duplicates 
[Ws_unique,IA,IC] = unique(Ws);
P_unique = P(IA);

% remove negative Power
ind = find(P_unique<0);
P_unique(ind) = [];
Ws_unique(ind) = [];

% smooth the data
P_uniqueS = smoothdata(P_unique,'movmedian',80);
P_uniqueS2 = irlssmooth(P_unique,80); % good !! % see FEX : https://fr.mathworks.com/matlabcentral/fileexchange/49788-robust-least-squares-smoother?s_tid=srchtitle_irlssmooth_1

plot(Ws_unique,P_unique,'b*',Ws_unique,P_uniqueS,'r',Ws_unique,P_uniqueS2,'g');

legend('unique','smoothdata','irlssmooth');

Answer 2

[John D'Errico]

The obvious answer is to look at the tools provided in MATLAB. Thus RMOUTLIERS, ISOUTLIER, etc. They are provided in the stats toolbox.

Or if you want to do the work yourself, use tools to do local filtering of some sort. For example, a local polynomial model that drops out the point at the center, but predicts a value there. In a time series context, this can reduce to a simple call to CONV with the correct kernel. (Not difficult to compute that kernel either.) Now compare the predicted value to the point left out. Those with large residuals are potential outliers.

You can also use tools for robust regression modeling, then identifying any points with large residuals as a possible outlier.

The data you show appears to have multiple problems though. There appears to be at least one region with a large dropout, an obvious outlier cluster, possibly caused by some sort of equiptment issues. Outlier detection schemes tend to be best at detecting single point outliers. Groups of outliers are far more difficult to detect, because these points all look like the data around them.

You might also look into clustering methods.

And that means you probably need to make an effort to clean up your data manually. Look for problems in the data that are obvious. If possible, then look back at the source of your data to see if there was some reason for the problem.

[Steven Lord]

FYI rmoutliers and isoutlier are part of MATLAB not Statistics and Machine Learning Toolbox.

which rmoutliers

/MATLAB/toolbox/matlab/datafun/rmoutliers.m

which isoutlier

/MATLAB/toolbox/matlab/datafun/isoutlier.m

Answer 3

[Sean Harvey]

Did you find a way to remove only the outliers (i.e., points away from the S-shaped curve)?