How to skip NaN values in cross correlation between 2 time series datasets.

Hi. I need to conduct cross correlation analysis between 2 time series datasets. Both datasets contain NaN values. In matlab, xcorr wouldn't skip wherever NaN values occurred in the time series. Please help.

~Thanks

My data look like this:

Soil moisture (SM):

SM = [ay1-1, ay1-2, ay1-3, ay1-4, NaN, NaN, NaN, NaN, ay2-1, ay2-2, ay2-3, NaN, NaN, NaN,....]

Plant growth (PG):

PG = [by1-1, by1-2, by1-3, by1-4, NaN, NaN, NaN, NaN, by2-1, by2-2, by2-3, NaN, NaN, NaN....]

NaN values in both vectors only occurred outside growing season; and that wherever there is NaN in SM, there is NaN in PG.

I want to see if there is any delayed responses in PG to changes in SM within growing season (lag analysis).

[Guillaume]

What do you mean by skipping? Remove the NaN altogether thus shortening the time series? If so, that will completely alter the whole correlation.

[Chalita]

By skipping I meant not to include NaNs in the cross correlation analysis. I can't remove NaN from the time series.

[Guillaume]

I'm not sure what you think a cross correlation analysis is. For discrete vectors, a cross correlation is just a multiplication element by element of the two vectors with a different shift followed by a sum. So, when there is a NaN what should the result of the multiplication be?

[Chalita]

Does it mean that replacing NaNs with zeros wouldn't affect the correlation?

[Guillaume]

Depends by what you mean by not affecting the correlation. It would mean that the matching non-nan element in the other vector wouldn't be taken into account for that particular shift, since the result of the multiplication would be 0. That may indeed be useful to do that as long as there aren't too many NaNs, but it will definitively affect the correlation values.

Replacing the NaNs by interpolated values may be more useful though. It's probably what Joe suggests doing with fillmissing (if he meant to use the 'linear' method).

[Chalita]

Thank you for the explanation. You're right that replacing NaNs with zeros would affect the correlation in the way you've explained. What I meant by not affecting the correlation was that the matching NaN/ non-NaN element in the other vector wouldn't be taken into account for that particular shift.

I added more details about my data and what I'm trying to do above.

~Thank you so much for your help.

Answer 1

[Joe Vinciguerra]

I encountered this same issue a few weeks ago. My solution was to use fillmissing() to patch up the NaN's.

I also used resample() to make sure my time vector was evenly distributed, but I suppose you could delete your NaN's all together, then resample.

[Chalita]

Thank you for your suggestion. "Fillmissing()" might not work in my case because my data are ranked.