Warning: Equation is badly conditioned. Remove repeated data points or try centering and scaling.

I have a data set, FR, atttached. I am trying to plot the confidence levels for the same but I am encountering a warning message :"Warning: Equation is badly conditioned. Remove repeated data points or try centering and scaling.". I dont know what is the issue as well as how to solve it. Any kind of help will be very useful. I am getting a graph like this (as shown below) but I think this is not correct. Please help me.

%%% this is the code I am using

%Create fit and confidence interval
 FitVec = fit(freq,alpha,'poly9')
 pRvecConf = predint(FitVec,freq,0.95,'observation','off');
 
 %get the fitting values
 fitY=feval(FitVec,freq);
 %multiply the confidence interval with sqrt(xVec(i)).^2 .*xVec'
 ci=(fitY-pRvecConf(:,1));
 %get the weighted confidence interval
 Conf_new=[fitY-ci,fitY+ci];
 %Plot
 figure
 plot(FitVec,freq,alpha) 
 hold on
 plot(freq,Conf_new,'m--')
 legend('Data','Fitted curve','Confidence','Location','se')
 xlabel('Length')
 ylabel('Strength')

[Torsten]

You forgot to include your code.

[Ron]

Yes sir I am very sorry and was trying to do that right now.

%Create fit and confidence interval
 FitVec = fit(freq,alpha,'poly9')
 pRvecConf = predint(FitVec,freq,0.95,'observation','off');
 
 %get the fitting values
 fitY=feval(FitVec,freq);
 %multiply the confidence interval with sqrt(xVec(i)).^2 .*xVec'
 ci=(fitY-pRvecConf(:,1));
 %get the weighted confidence interval
 Conf_new=[fitY-ci,fitY+ci];
 %Plot
 figure
 plot(FitVec,freq,alpha) 
 hold on
 plot(freq,Conf_new,'m--')
 legend('Data','Fitted curve','Confidence','Location','se')
 xlabel('Length')
 ylabel('Strength')

[Bruno Luong]

This modification (normalized frequency) removes the warning. The graph and confidence still look similar to yours.

The warnning is triggered but it's still OK. The real issue is that the polynomial model cannot fit correctly your data.

data = readmatrix('https://www.mathworks.com/matlabcentral/answers/uploaded_files/1803220/FR.txt');
freq = data(:,1);
alpha = data(:,2);

s = 1/max(freq);
fn = s * freq;
%Create fit and confidence interval
 FitVec = fit(fn,alpha,'poly9');
 pRvecConf = predint(FitVec,fn,0.95,'observation','off');
 
 %get the fitting values
 fitY=feval(FitVec,fn);
 %multiply the confidence interval with sqrt(xVec(i)).^2 .*xVec'
 ci=(fitY-pRvecConf(:,1));
 %get the weighted confidence interval
 Conf_new=[fitY-ci,fitY+ci];
 %Plot
 figure
 plot(FitVec,fn,alpha) 
 hold on
 plot(fn,Conf_new,'m--')
 legend('Data','Fitted curve','Confidence','Location','se')
 xlabel('Normalized Length')
 ylabel('Strength')

[Ron]

Thankyou so much for helping me. Your solution did work very well but the nature of the graph did not change. As in the low frequency the noise is prominent hence the confidence bound should have larger width in that region and in the higher frequency region the deviation is very small so there the width should be small.

[Bruno Luong]

No the confidences are NOT smalller at the beginning. You get caught by optical illusion because the data raise sharp there. You shuld not look at the distance between two curves but the vertical separation between two curves.

data = readmatrix('https://www.mathworks.com/matlabcentral/answers/uploaded_files/1803220/FR.txt');
freq = data(:,1);
alpha = data(:,2);

s = 1/max(freq);
fn = s * freq;
%Create fit and confidence interval
FitVec = fit(fn,alpha,'poly9');
pRvecConf = predint(FitVec,fn,0.95,'observation','off');

%get the fitting values
fitY=feval(FitVec,fn);
%multiply the confidence interval with sqrt(xVec(i)).^2 .*xVec'
ci=(fitY-pRvecConf(:,1));
%get the weighted confidence interval
Conf_new=[fitY-ci,fitY+ci];
%Plot
figure
plot(freq,ci,'m')
set(gca,'ylim',[0 7000])
xlabel('Length')
ylabel('Confidence')

[Bruno Luong]

To give ou a idea here is a fit using free knot spline

[Ron]

Thankyou for answering once again. I am now trying the spline option also. the major issue is not the fit but the confidence level curve associated with the data. What I am trying to achieve looks something like the picture below. In this picture the width of the bound is large in the low frequency region because there the data is usually very noisy. My curve should have bounds like this. but I dont know how to achieve this.

Answer 1

[Bruno Luong]

I don't use MATLAB native FIT, I can't get it works correctly to fit your data, even with smoothingspline model, and predictive interval predint with spline model as input is not suported.

I rather use my file exchange BSFK here https://www.mathworks.com/matlabcentral/fileexchange/25872-free-knot-spline-approximation

data = readmatrix('https://www.mathworks.com/matlabcentral/answers/uploaded_files/1803220/FR.txt');
freq = data(:,1);
alpha = data(:,2);

options = struct('knotremoval_factor', 1, 'lambda', 2e-7);
pp = BSFK(freq, alpha, [], [], [], options);

fitY=ppval(pp,freq);
dY = (alpha-fitY);
nu = 33;
sigma = movstd(dY,nu); % sqrt(sum(dY.^2) / (n-1));
level = 0.98;
normconf = erfinv(level)*sqrt(2); % tinv((1+level)/2,nu-1) is better choice?
ci = sigma * normconf;
fi=linspace(freq(1),freq(end),400);

%Plot
close all
figure
grid on
hold on
xconf = [freq; freq(end:-1:1)] ;
yconf = [fitY+ci; fitY(end:-1:1)-ci(end:-1:1)];
p = fill(xconf,yconf,'r');
p.FaceColor = [0.8 0.8 0.8];      
p.EdgeColor = 'none';  
h1 = plot(freq, alpha,'.k');
h2 = plot(fi,ppval(pp,fi),'r','LineWidth',2);
legend([h1 h2 p], 'Data','Fitted curve', ...
    sprintf('%g%% Confidence', level*100),'Location','se')
xlabel('Length')
ylabel('Strength')

[John D'Errico]

That looks like the result of a regression spline. Long ago, I wrote one that produced nice confidence intervals, just like that. Too long by now though, as I saw some bugs in it when I just tested it out.

It would be simple to do, if you build the design matrix for a regression spline. Then tools from the stats toolbox (fitlm, or regress) can provide confidence intervals around the curve.

[Bruno Luong]

If the design matrix of linear smoothing spline is known. For free knot spline as in ly case such linearized design matrix is difficult to derive.

[Bruno Luong]

Another run with different fit option

data = readmatrix('https://www.mathworks.com/matlabcentral/answers/uploaded_files/1803220/FR.txt');
freq = data(:,1);
alpha = data(:,2);

options = struct('knotremoval', 'none', 'lambda', 2e-7);
pp = BSFK(freq, alpha, 5, 30, [], options);

fitY=ppval(pp,freq);
dY = (alpha-fitY);
sigma = movstd(dY,33); % sqrt(sum(dY.^2) / (n-1));
level = 0.98;
normconf = erfinv(level)*sqrt(2);
ci = sigma * normconf;
fi=linspace(freq(1),freq(end),400);

%Plot
close all
figure
grid on
hold on
xconf = [freq; freq(end:-1:1)] ;
yconf = [fitY+ci; fitY(end:-1:1)-ci(end:-1:1)];
p = fill(xconf,yconf,'r');
p.FaceColor = [0.8 0.8 0.8];      
p.EdgeColor = 'none';  
plot(freq, alpha,'.k')
plot(fi,ppval(pp,fi),'r','LineWidth',2)
legend(sprintf('%g%% Confidence', level*100),'Data','Fitted curve','Location','se')
xlabel('Length')
ylabel('Strength')

[Ron]

Sir, thank you so so much for putting this much efforts in solving my issue. The solution provided by you is very close to what I have been trying to achieve. I ran your code but there is an error "unknow function BSFK". Can you please let me know what is this BKSF? Once again thanking you from the bottom of my heart for helping me.

[Bruno Luong]

i gave a link to download the BSFK file exchange; please read my post again

Answer 2

[John D'Errico]

Do NOT use polynomials to model data like this In fact, do not use high order polynomials, pretty much ever.

FR = load('FR.txt');
x = FR(:,1);
y = FR(:,2);
plot(x,y,'-')

UGH.

Next, polynomials abhor a singularity, and it looks like you may have a singularity at x==0, or at least something very close to a singularity. No finite order standard polynomial ever has a singularity in it. (Don't talk about rational polynomials, which are completely different animals, but also significantly more difficult to work with.)

Next, your curve has a deep, sharp drop in it. Polynomials will never have anything in them that does this. These are things that simply do not match the expected behavior of a polynomial model.

Yes, I know, polynomials are easy to use. Easy to fit. And so people tend to use them at will. And if a low order fit is not good enough, just use a higher order fit. Heck, we learned about Taylor/Maclaurin series in some long forgotten class. And they are just polynomials, right?

I'm sorrry, but this always works out poorly for the person wanting a fit for their data.

You might decide to use a regression or smoothing spline, or you might decide to use a more general smoothing tool, or time series methods. There are many things, that with some amount of effort, you might do here. But not a polynomial.

[Bruno Luong]

"Don't talk about rational polynomials, which are completely different animals, but also significantly more difficult to work with."

I don't have the toolbox but MATLAB has the NURBS. Can it be used to fit data with singulatities?

[Ron]

Yes sir, I got your point and now I am trying other tools also but I am still unsure about how to get a confidence curve like the one I am posting here. Can you please suggest some way through which I can get something like this for my data. the variation in the low frequency zone is high and then it reduces and the confidence level should show that but in my case that is not happening or maybe I am missing something. Please help.

[John D'Errico]

@Bruno Luong I think the issue is the need for the nice pretty confidence intervals. I doubt NURBS does that. Easy enough to build the code for a basic regression spline, then get confidence intervals.

[Bruno Luong]

MATLAB fit with smoothingspline does not supportt confidence. Though it is linear regression.

I also play with smoothing parameters of FIT, I did not understand what does it do.

Answer 3

[Image Analyst]

What if you just scanned your data with movmean to get the local mean, and movstd to get the local standard deviation, and then set confidence levels 2 or 3 SDs away from the mean? If the SD array is noisier than you want, you could smooth that. Would that work for you? I'm not sure why you think you need a polynomial model in the first place. You can get a good idea of the range of data jst by doing it numerically on your actual data.

% Demo by Image Analyst.
% Initialization steps.
clc;    % Clear the command window.
fprintf('Beginning to run %s.m ...\n', mfilename);

Beginning to run LiveEditorEvaluationHelperEeditorId.m ...

close all;  % Close all figures (except those of imtool.)
clearvars;
workspace;  % Make sure the workspace panel is showing.
format long g;
format compact;
fontSize = 16;
lineWidth = 2;
markerSize = 30;
FR = load('FR.txt');
x = FR(:,1);
y = FR(:,2);
subplot(3, 2, 1);
plot(x,y,'-')
grid on;
localMeans = movmean(y, 5);
subplot(3, 2, 3);
plot(localMeans, 'b-')
title('Local means')
grid on;
localSD = movstd(y, 5);
subplot(3, 2, 5);
plot(x, localSD, 'b-')
title('Local SD')
grid on;
% Compute upper level
upperCL = localMeans + 3 * localSD;
lowerCL = localMeans - 3 * localSD;
subplot(3, 2, [2,4,6]);
plot(x, y, 'b-')
hold on;
plot(x, upperCL, 'r-')
plot(x, lowerCL, 'r-')
title('Data with Confidence Limits in red')
grid on;

[Ron]

Sir thankyou so much for investing your precious time in solving my problem. Your solution is very beautiful and it works like a charm. I want to accept this answer as the solution but the answer provided by @Bruno Luong is exactly what I need. Once again I thank you for taking out time and helping me out. I will save your code in my "Helpful code" folder.

[Image Analyst]

Thanks @Ron. If an Answer solves your original question, then could you always click the Vote icon (thumbs up) to award the answerer with "reputation points" for their efforts in helping you? They'd appreciate it. Thanks in advance. 🙂 Note: you can only accept one answer (so pick the best one) but you can click the "Vote" icon for as many Answers as you want. Voting for an answer will also award reputation points.

For full details on how to earn reputation points see: https://www.mathworks.com/matlabcentral/answers/help?s_tid=al_priv#reputation

[Ron]

Yes sir I did upvote your answer as it was realy helpful to me. I am really thankful to you for helping me out. I will surely accept the best answer it is just that I am trying the suggestions/ solutions provided by others. Once I am done I will accept the answer.

Answer 4

[Matt J]

9th order polynomials do tend to be badly conditioned. Try using a lower order, or perhaps a spline model.

[Ron]

Yes sir you are absolutely right but the documentation says that for the spline fit, the confidence levels cannot be obtained and my main motive is finding the confidence levels.

[Bruno Luong]

@Ron Your confidence shall be meaning less if you can't correctly fit the data.

[Ron]

Yes Sir very very true. I am working on that with your help only. Your solution is exactly what I wanted.