Identifying smoothness on different plots without jumps or kinks

Hello, I am processing different signals from which I am estimating the Empirical Cumulative Function to find specific trends like the one shown in Figure 1 where some signals show a smooth plot. Meanwhile, other signals can show jumps or kinks in the Empirical Cumulative Function like the figure 2. Is there a function that help to identify these two cases using the F(x) data for each component?. This identification would say that the first case is a proper signal for having an smooth trend. On the other hand, the second case can be identified as an unsuitable signal for having jumps or kinks in Empirical Cumulative Distribution fuunction. I would appreciate the help. I attached an example of the Empirical Cumulative values F(x) for the first case (f1) and second case (f2) as txt as an example that would help identify any of those cases mentioned. I would appreciate the help.

Answer 1

[Sulaymon Eshkabilov]

Here is one solution with ftnlm():

D2 = load('f2.txt');
X2 = linspace(-25, 25, numel(D2));
% Data shows that a Nonlinear Sigmoid type nonlinear fit model is the most suitable:
S_model = @(b, x) 1./(1 + exp(-(b(1) + b(2)*x)));
% Initial guess values for b values
b0 = [0, 0.5];
% Fit the model using fitnlm
FModel = fitnlm(X2, D2, S_model, b0);
% Display the results
disp('Model Parameters:');

Model Parameters:

FModel

FModel = 
Nonlinear regression model:
    y ~ F(b,x)

Estimated Coefficients:
          Estimate       SE         tStat       pValue   
          ________    _________    _______    ___________

    b1    -0.37127     0.024842    -14.945     7.1942e-38
    b2     0.38009    0.0079024     48.098    1.6071e-140


Number of observations: 293, Error degrees of freedom: 291
Root Mean Squared Error: 0.0378
R-Squared: 0.992,  Adjusted R-Squared 0.992
F-statistic vs. zero model: 4.27e+04, p-value = 0

% Plot the data and fitted sigmoid curve
figure;
plot(X2, D2, 'bo', 'DisplayName', 'Given Data');
hold on
% Let's compute the values of the fit model to show its performannce
X_vals = linspace(min(X2), max(X2), 100);
D2_fit = predict(FModel, X_vals');

plot(X_vals, D2_fit, 'r','LineWidth', 2, 'DisplayName', 'Fitted Sigmoid Curve');
xlabel('X');
ylabel('Y');
legend('show', 'Location', 'best');
grid on

[Jorge Luis Paredes Estacio]

Thank you very much. Which of the Fmodel parameters will help me to identify the fisrt and the second case? . The f1.txt and f2.txt are the first and the second case respectively as explained in the images.

[Sulaymon Eshkabilov]

Most welcome! Thumbs up, please! :)

Answer 2

[Sulaymon Eshkabilov]

For f1.txt, the model formulation will be the best fit, e.g.:

D1 = load('f1.txt');
X1 = linspace(-25, 25, numel(D1));
% Data shows that a Nonlinear Sigmoid type nonlinear fit model is the most suitable:
S_model1 = @(b, x) 1./(1 + exp(-(b(1) + b(2)*x)));
% Initial guess values for b values
b0 = [0, 0.5];
% Fit the model using fitnlm
FModel1 = fitnlm(X1, D1, S_model1, b0);
% Display the results
disp('Model Parameters:');

Model Parameters:

FModel1

FModel1 = 
Nonlinear regression model:
    y ~ F(b,x)

Estimated Coefficients:
          Estimate        SE        tStat       pValue  
          ________    __________    ______    __________

    b1    0.017674     0.0012403    14.249    7.5132e-43
    b2     0.43796    0.00047824    915.78             0


Number of observations: 1272, Error degrees of freedom: 1270
Root Mean Squared Error: 0.00386
R-Squared: 1,  Adjusted R-Squared 1
F-statistic vs. zero model: 1.94e+07, p-value = 0

% Plot the data and fitted sigmoid curve
figure;
plot(X1, D1, 'bo', 'DisplayName', 'Given Data');
hold on
% Let's compute the values of the fit model to show its performannce
X_vals = linspace(min(X1), max(X1), 100);
D1_fit = predict(FModel1, X_vals');

plot(X_vals, D1_fit, 'r','LineWidth', 2, 'DisplayName', 'Fitted Sigmoid Curve');
xlabel('X');
ylabel('Y');
legend('show', 'Location', 'best');
grid on

[Jorge Luis]

Thank you so much. What kind of model fitting would you suggest for f3.txt case to reduce the MSE and RMSE. Thank you.

[Sulaymon Eshkabilov]

Most welcome!

This one also looks like sigmoid type of function. You can try to use the exponential function used for f1.txt and f2.txt with more parameters.

[Alex Sha]

Try the fitting function below:

n=1

Sum Squared Error (SSE): 0.198265382635044
Root of Mean Square Error (RMSE): 0.0213245660245507
Correlation Coef. (R): 0.99903177402219
R-Square: 0.998064485505925

Parameter	Best Estimate     
---------	-------------     
p0       	0.987904925538442 
a1       	-0.976370688703938
b1       	5.01866411918097  
c1       	1.73892866026451

n=2

Sum Squared Error (SSE): 0.0198251724593009
Root of Mean Square Error (RMSE): 0.00674318760195012
Correlation Coef. (R): 0.999903226259401
R-Square: 0.999806461883958

Parameter	Best Estimate    
---------	-------------    
p0       	0.340506688986253
a1       	-0.33422435435035
b1       	1.00705898018223 
c1       	0.346758083442965
a2       	0.652548080384559
b2       	-28.7898530254341
c2       	-9.96361274247002

n=3

Sum Squared Error (SSE): 0.00665429785589847
Root of Mean Square Error (RMSE): 0.0039066803101404
Correlation Coef. (R): 0.99996751905745
R-Square: 0.999935039169911

Parameter	Best Estimate      
---------	-------------      
p0       	0.320693681605882  
a1       	-0.319481310561844 
b1       	1.64165756141301   
c1       	0.575821065083703  
a2       	0.0732506083073871 
b2       	-0.224274083494237 
c2       	-0.0508621405696974
a3       	0.604308733522548  
b3       	-28.7896426138829  
c3       	-9.95835330008731 

[Jorge Luis]

Thank you so much. Happy New year :). n is the number of iterations. How do you set the this n value in the fitnlm function in MATLAB. Thank you for your response.