Signal filtering, smoothing and delay

Hi guys !

For further system analysis I have decided to filter some signals, obtained from an acceleration measurement of a dynamic system with multi-sinusoidal excitation signals. I'm kind of desperate to remove the noisy parts and the outliers of the sensors.

The annotated pictures speak more than words. I want to eliminate the noisy trends of the signals, especially in the lower frequency areas where the acceleration sensors' noise is considerably high. Furthermore, I want to eliminate the high outliers represented over the signal.

I have tried the filtfilt function with a butterworth filter and a cut-off frequency at around 40Hz (the excitated signals were at maximum with 20Hz frequency). Though, I have not been successful to remove what I intended to. I have also searched through matlab and tried several sgolay filters, but nothing really led to a satisfying result.

Please find attached the signal plot and the mat file. I hope you guys can help !!!

Full acceleration signal of the vertical multi-sine excitation with maximum sine at the end with frequency f=20Hz.

Outliers at higher frequencies:

Heavy noise at lower frequencies:

[Daniel M]

I have several suggestions, but since I'm not near a computer I can test them myself, which is why I'm not posting in the answer section.

First is you should remove DC effects by detrending (or high pass filter with a cutoff of 0.5 Hz) your data. The reason it hovers at around 10 is because acceleration due to gravity is 9.8. So just center it around zero.

Next thing that strikes me is your sampling rate seems pretty high for a signal with an apparent max of 20 Hz. I think just downsampling your data (or smoothing, moving average, low pass filtering, etc) would help.

Next, I would suggest trying median filtering, or mean filtering the data. 2-3 standard deviations should do it, but play with it. You could try a sliding window (see rmoutliers), or you could consider epoching your data (because of the slowly increasing amplitudes) and applying the filter to each epoch.

What does an fft of this data look like? I get the feeling the noise is much higher than 40 Hz and you can raise your cutoff frequency.

[Aladin Djuhera]

Hi Daniel !

Thank you for your comment :)

I actually want the DC effects to stay, as I need them for further processing. I have tried smoothing and it gets better, however I am looking for a really smooth signal after filtering, that would result in a sinusoidal without any noise. Do you think that this is even possible ?

[Daniel M]

Of course it's possible. What is the sampling rate of the accelerometer? Have you looked at the fft?

[Aladin Djuhera]

The sensors measured with discrete time t=0.002s. I have looked at the fft but don't know what information to gather from it.

See attached the real part plot.

[Daniel M]

You're doing great so far, follow the example on how to produce the single sided amplitude spectrum.

And 500 Hz sampling rate isn't bad, so probably don't need to down sample. But there's other time-domain smoothing to try. But the frequency domain filter will be more powerful. Use fvtool to look at what your filter is doing.

[Aladin Djuhera]

Thanks Daniel!

Here are the plots of the single sided amp spectrum:

Unscaled plot:

Scaled (zoomed) plot:

What do you think ? This is the raw signal. What filters could I apply ?

Thanks for your help !

[Daniel M]

It's hard to see because of the large spike at zero (because of the DC offset), so try semilogy.

[Aladin Djuhera]

That's the semilogy plot :)

[Image Analyst]

Please show what you think noise free signals would look like.

[Aladin Djuhera]

I would imagine a signal with less outliners that come from the sensor noise and better / clearer representation in the small frequencies of the input signals. As the first 5-6 input signals to the system are sines with small frequencies (0.5 - ... Hz), the sensor noise is considerable and influences the data massively.

I tried using medfilt1 to tackle these problems. The original signal is blue, the filtered is orange.

Better representation in the low frequencies at the beginning:

However, medfilt1 leads to a reduction in amplitude for the higher frequent input signals:

[Daniel M]

And what does a lowpass filter (cutoff 50 Hz) look like? Zoom in on just one of the high amplitude bursts.

[Aladin Djuhera]

[Daniel M]

That seemed to help. Now try downsampling.

    y = downsample(x,n);

And play with different values of n from 2-10. Again, I could be of more help, but am only on mobile.

[Aladin Djuhera]

Sadly does not help and downsampling is really a hard manipulation of data, isn't it ? I wouldn't go higher than factor 3.

However, this does not help. Can u look at the code i posted down here ?

Answer 1

[Daniel M]

I created a butterworth filter using filterDesigner, with the following parameters: Bandpass, IIR - Butterworth, specify order: 4, Fs = 500, Fc1 = 0.5, Fc2 = 40. Then exported the coefficients:

SOS =

            1            0           -1            1      -1.9911      0.99116
            1            0           -1            1      -1.3191      0.50059
G =

      0.21246
      0.21246
            1

Then applied it to the mat-file data:

            fs = 500;
            t = 0:1/fs:(length(y)-1)-1/fs;
            y = y_sine_vert;
            Y = filtfilt(SOS,G,y);
            figure
            plot(t,detrend(y),t,Y) % detrend(y) because I used a cutoff freq of 0.5 for Y. 
            % You can add the DC component back if you wish.
            residuals = detrend(y)-Y;
            figure
            plot(t,detrend(y),t,residuals)

And attached are the results. I think the zoomed in plot wonderfully shows how the underlying sine waves are captured and noise is removed. The earlier parts of the full series have small signal-to-noise ratio, so the results aren't as great. But looking at the residuals, shows that what was removed was essentially noise.

Otherwise, if that isn't sufficient, then I don't know what you really want.

[Aladin Djuhera]

Thank You Very Much!

I have obtained good results with that signal :)

Thanks for your help, patience and matlab know-how !

Answer 2

[Aladin Djuhera]

I received good results using the following code from: https://stackoverflow.com/questions/42944461/removing-spikes-from-a-signal-matlab

Code:

%moving cleaning window

y1_1= y1(1:100);%first window

x=1;

%cleaning loop

while x<= length(y1)

if(y1(x)> 1.01*(median(y1_1))||y1(x) < 0.95*(median(y1_1)))

y1(x)= median(y1_1);

end

if(x>= length(y1)-100)

y1_1= y1(length(y1)-100:length(y1));

else

y1_1 = y1(x:x+100);

end

x=x+1;

end

I obtained the following result, which I am very satisfied with:

But on the other hand, the signal amplitudes are damped quite heavily after the first third of the signal. I would like that damped part to be not manipulated that heavily but instead to apply to the same results obtained in the first third.

Thank you guys !

[Aladin Djuhera]

Up to 2.125 on the x axis, the signal is just as I want it to be. Then it gets manipulated strongly. How can I avoid this? What did I understand wrong in the code ?

[Daniel M]

This code is hard to use because it is not standalone. Make it easy for people to help you by uploading an m-file that runs without error. Otherwise, the code seems likely to be error-prone. Just use movmedian if you want to do something like this.