Removing columns if single value is more than threshold
7 views (last 30 days)
Show older comments
I have a very big data matrix which I am trying to filter a bit. I would like to remove whole columns, if any value is for example 10% greater or less than row average. I checked the rmoutliers function, but I don't know how I can make that work the way I need. Another matter is that some columns are fine for my use, but they are scaled up, so they would get probably filtered out too with that method. That is fine, but could that be avoided by first normalizing the data somehow, and then restoring the filtered data to original scale after that. I would appreciate the help very much
0 Comments
Accepted Answer
Adam Danz
on 24 Aug 2020
Edited: Adam Danz
on 24 Aug 2020
" I would like to remove whole columns, if any value is for example 10% greater or less than row average"
Demo:
% Create 100x10 matrix
data = rand(100,10) .* linspace(1,100,10);
% Determine which columns have at least 1 values that is
% within +/- 10% of the row's average
rowAverages = mean(data,2);
isNearAvg = abs(data - rowAverages) <= rowAverages * 0.1; % 10% threshold
replaceColumn = any(isNearAvg,1);
% Option 1: Repalce the column with NaNs, thereby preserving the original structure
data(:,replaceColumn) = NaN
% Option 2: Remove the columns (use replaceColumn to see which cols were removed)
data(:,replaceColumn) = []
2 Comments
Adam Danz
on 25 Aug 2020
Glad I could help!
Functions like rmoutliers come in handy during data exploration and if you are using a well-established method of outlier removal but it's often better to write your own functions when the method requires customization.
More Answers (0)
See Also
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!