Filter regions in data with 'obvious' irregular linear trend

Hello,

I have a dataset that I am plotting but there are clear trends occurring within the data that I would like to remove. I have attached a figure of a plot of some of the data. The y variable is supposed to increase exponentially as x increases, but at some points (between 9 and 14 x values), there is a control placed on the y output, causing it to hover around certain values as x increases.

Does anyone know how I may be able to identify these regions where the data has a clear linear trend between certain x values? I would usually think to have a polynominal equation fitted on the data and then index any data points that lie outside the error margins of this fit. However, I can only fit that equation by removing these irregular trends first, so a bit of an issue there.

These linear trend regions do not usually happen in chronological sequence. For instance, there may be a few period in time where the y-output is limited along x, creating a seemingly continuous trend. Thus I cannot use the 'diff' function for instance to easily identify periods when y-output varies by a specified amount. Hope that makes sense.

Feel free to ask me if I need to explain more.

Answer 1

[John D'Errico]

This is more difficult to do than you might think. The eye/brain is very good at seeing patterns, even with a lot of noise.

The problem is, this is not a "region",where you have a trend, but a region where SOME subset of the data is spurious, that follows a trend. And it is not even a constant, but an apparently linear trend! I could as easily pick another region wherein some subset of that huge number of points also follows a linear trend, yet they represent perfectly good data in your eyes. What is "obvious" to you is rarely "obvious" to a computer. If that was true, we would all be replaced by computers. (Or is that happening already?)

Sorry, but the best solution might be to use your eye to detect that bad data, perhaps selecting the bad points with the mouse. My selectdata tool on the file exchange might work, although it is getting a bit old. I think it may have some problems with the new graphics engine. There are also brushing options in MATLAB itself now that might suffice for your problem.

[mashtine]

Thanks John. You really touched on everything I was thinking and I am surprised I even got a response as I would have thought it just wasn't possible. But you never know these days, I always stumble upon some new tool in matlab.

I will have a look at the brush options. Thanks for the suggestion!

[John D'Errico]

Your brain is just SOOOO good at seeing patterns, that in some cases it even sees trends where none exist. (Were that not true, I might have saved some money in the stock market, lol.)

But making a computer do it for you? Not as easy at all. If this is a one time job, or even a rare enough event, save that mental energy and use your mouse.

[mashtine]

haha Will do! Thanks again.