Detrending still leaves a slope

I am attaching two line profile examples for my case. After using detrend, I obtain the profile given in 'detrend_lineprofile' photo for the original data shown in 'lineprofile' photo. Although detrending seems it is working, still there is a slope towards the end of the profile after the detrend.

I want to get rid of this slope and bring the base of the profile to an even, flat condition. Is there a way to get rid of this slope after detrending?

Thanks in advance for all the help.

[Steven Lord]

How did you call detrend? In particular, what value did you specify for the n input argument? Or did you call it with only one input?

[Bugrahan Guner]

I only have arrays of 1000 datapoints indicating the height values for the profile, so I directly tried to call it with only one input, which was the height.

[dpb]

Don't attach pictures, attach a .mat file of the raw data.

Have you just tried successively detrending?

[Bugrahan Guner]

I just attached a data set for the case

Answer 1

[Star Strider]

This is the best I can do with your data.

A reasonably robust approach is —

% imshow(imread('lineprofile.jpg'))
% imshow(imread('detrend_lineprofile.jpg'))

LD = load('profile_data.mat')

LD = struct with fields:
    topo_z_data: [1000×10 double]

Z = LD.topo_z_data

Z = 1000×10

0.0501 0.0765 0 0 0 0.2352 0 0.1216 0 0 0 0 0.4114 0.5229 0.0092 0 0.5022 0.2099 0.3771 0.0217 0.4932 0.4469 0.4332 0.3709 0.2149 0.2916 0.3531 0.1412 0.2723 0.0516 0.3805 0.6880 0.4912 0.1714 0.5189 0.2788 0.2863 0 0.5265 0.0088 0.5829 0.5417 0.6285 0.6627 0.0938 0.3920 0.4706 0.1596 0.5727 0.4251 0.5005 0.6379 0.5792 0.4328 0.2421 0.5059 0.5528 0.2265 0.8142 0.4656 0.5559 0.8270 0.6569 0.8655 0.2935 0.2508 0.8058 0.2850 0.8548 0.6962 0.6962 1.0147 0.7112 0.6039 0.3564 0.3766 4.3116 0.4459 0.7278 0.7808 0.5026 0.9304 1.0840 0.8096 0.6392 0.9144 3.8141 0.8225 0.7197 0.7080 0.9621 0.6400 1.3384 1.0655 0.7736 0.6823 3.2333 0.7367 0.6273 0.8611

figure
surf(Z)
colormap(turbo)

y = Z(:,5);

L = size(Z,1);
x = linspace(0, L-1, L).';                                                                          % Assume Column Vectors


Lva = islocalmin(y, 'FlatSelection','center');                              % All Points Meeting Criteria
NrLv = nnz(Lva)

NrLv = 188

Lvp = islocalmin(y, 'FlatSelection','center', 'MinProminence',0.5);         % Define 'Outliers'
NrLvp = nnz(Lvp)

NrLvp = 34

Lv = Lva & ~Lvp;                                                            % Delete 'Outliers'
NrLv = nnz(Lv)

NrLv = 154

xsel = x(Lv);
ysel = y(Lv);
% y = sum(x*0.02 + rand(1,5)*100.*exp(-(x-randi([100 900],1,5)).^2/1E+3) + randn(size(x))/10, 2);         % Create Data

figure
plot(x, y)
hold on
% plot(xsel, ysel, '.r')
% plot(x(Lvp), y(Lvp), '.g') 
hold off
title('Original')

B = [xsel(1) 1; xsel(end) 1] \ ysel([1 end]);                                   % Linear Regression Of End Points (Or Suitably Representative Points)

Baseline = [x ones(size(x))] * B;                                               % Calculate Baseline Cirrection

ycorr = y - Baseline;                                                           % Subtract Correction

figure
plot(x, ycorr)
title('Baseline Detrended')

EDIT — (7 Sep 2023 at 15:56)

Changed code to use provided data.

.

[Bugrahan Guner]

Thank you so much, I will try it on the data as well. In the meantime I am attaching the data set but what you did is exactly what I am looking for. Thanks.

Answer 2

[John D'Errico]

It depends on how you define detrend. Clearly, you have a different definition from me, at least, as well as the author of the function detrend.

On average, there is no slope. No overall trend. Yes, if you look at the baseline of that curve, there is an overall trend. But that is not how detrend is defined to work. Of course you can always find some subset of points on any scattered curve that will appear to have a trend.

My guess would be, IF you passed a straight line fit (using, for example, polyfit) through the detrended curve, the slope of that fit would be exactly zero.

help detrend

 DETREND Remove a polynomial trend.
    Y = DETREND(X) removes the best straight-line fit linear trend from the
    data in vector X and returns the residual in vector Y.  If X is a
    matrix, DETREND removes the trend from each column of the matrix. If
    X is a table or timetable, then DETREND operates on each variable
    separately.
 
    Y = DETREND(X,n) removes a polynomial trend with degree n. n = 1 by
    default. Setting n = 0 is equivalent to using the 'constant' option and
    setting n = 1 is equivalent to using the 'linear' option.
 
    Y = DETREND(X,n,BP) removes a continuous, piecewise polynomial trend
    according to the break points specified in BP. BP can be a vector of
    sample point values or a logical vector of length equal to the number
    of sample points, where true indicates the position of a break point.
 
    Y = DETREND(___,NANFLAG) specifies how NaN values are treated:
 
       "includemissing" / "includenan" -
                      (default) NaNs are included when calculating the
                      polynomial trend. If NaNs are present, then the
                      trend and the corresponding output will be vectors
                      of NaNs.
 
       "omitmissing" / "omitnan"       -
                      NaN values are omitted when calculating the trend.
                      NaN values in the input produce corresponding NaN
                      values in the output.
 
    Y = DETREND(___,'Continuous',TF) specifies whether the piecewise
    polynomial trend is required to be continuous. TF must be one of the
    following:
 
       true         - (default) the fitted trend is continuous everywhere
       false        - the fitted trend is not required to be continuous
 
    Y = DETREND(___,'SamplePoints',S) also specifies the sample points S
    associated with the data in X. S must be a floating-point, duration or
    datetime vector. S must be sorted and contain unique points. If the
    first input is a table, S can also specify a table variable. You can
    use S to specify time stamps for the data. By default, DETREND assumes
    the data is sampled uniformly at points S = [1 2 3 ... ].
 
    Arguments supported only when first input is table or timetable:
 
    Y = DETREND(...,'DataVariables',DV) removes a polynomial trend from the
    data only in the table variables specified by DV. Other variables in the
    table not specified by DV pass through to the output without being
    operated on. The default is all table variables in X. DV must be a table
    variable name, a cell array or string array of table variable names, a
    vector of table variable indices, a logical vector, a function handle
    that returns a logical scalar (such as @isnumeric), or a table vartype
    subscript. The output table Y has the same size as the input table X.
 
    Y = DETREND(...,'ReplaceValues',TF) specifies how the detrended data
    is returned. TF must be one of the following:
         true  - (default) replace table variables with the detrended data
         false - append the detrended data as additional table variables
    If the value of ReplaceValues is false, then the width of Y is the sum
    of the input data width and the number of data variables specified.
 
    Examples:
 
       % Remove a continuous, piecewise linear trend
       sig = [0 1 -2 1 0 1 -2 1 0];      % signal with no linear trend
       trend = [0 1 2 3 4 3 2 1 0];      % two-segment linear trend
       x = sig + trend;                    % signal with added trend
       y = detrend(x,1,5)                % break point at 5th element
 
       % Remove a nonlinear trend
       sig = repmat(sig,1,2);
       trend = ((1:18)/18).^3;
       x = sig + trend;
       y = detrend(x,3);
       plot(sig); hold on
       plot(y)
 
       % Remove discontinuous linear trend with sample points
       t = -10:10;
       x = t.^3 + 6*t.^2 + 4*t + 3;
       bp = 0;
       y = detrend(x,1,bp,'SamplePoints',t,'Continuous',false);
       plot(t,x,t,y,t,x-y,':k')
       legend('Input Data','Detrended Data','Trend','Location','northwest')
 
       % Remove quadratic trend from the selected variable in a table
        t = (-4:4)';
        trend = (t.^2 + 4*t + 3);
        sig = [0 1 -2 1 0 1 -2 1 0]';
        x = sig + trend;
        T = table(t,trend,sig,x);
        T = detrend(T,2,"DataVariables","x","SamplePoints","t","ReplaceValues",false);
        stackedplot(T,"XVariable","t")
 
    See also TRENDDECOMP, MEAN, ISCHANGE, POLYFIT, POLYVAL

    Documentation for detrend
       doc detrend

    Other uses of detrend

       gpuArray/detrend    iddata/detrend    timeseries/detrend

So, called with only one argument, detrend will remove a linear trend from the overall curve. For example, since you give no data...

t = linspace(0,1,100);
y0 = exp(t) + randn(size(t))/10;
plot(t,y0)

y1 = detrend(y0);
plot(t,y1)

So a linear trend has been removed. I might have done this instead:

P1 = polyfit(t,y0,1);
y1b = y0 - polyval(P1,t);
norm(y1-y1b)

ans = 0

As you can see, the two curves are identical (since the difference is zero.) As well, if you read the help for detrend, it says nothing at all about finding, AND removing a baseline trend.

Finally, let me test my claim that the slope of the detrended curve will be zero. The first coefficient out of polyfit should be zero, to within floating point trash.

polyfit(t,y1,1)

ans = 1×2

1.0e-15 * 0.0381 -0.1313

As well, the second coefficient is also zero, since detrending also removed any constant, DC component in the curve.

Anyway, feel free to define detrending in any way you wish. But don't expect software to read your mind. And if that day ever comes when software can read my mind, I'm heading for the hills anyway. But then, it probably knows that already.