Filter data according to Date and Other constraints

Hi,

I have a timetable which contains data for 4307 days, and I would like to filter it with two constraints :

1. Split data by date, so that each cell/array contains data on each single day,

e.g. cell/array1 has only data for 1996/01/01, and cell/array2 has data for 1996/01/02;

2. Filter data by limiting T to be minimum on each single day,

e.g.

I only need rows with minimum T, in this case, 11.

However, my code does not work after the second constraint, and put data for different dates into one cell:

load('TT5.mat')
unidate = unique(TT5.date_c);
for i= 1 : size(unidate)
    thisDate = unidate(i);
    indexesWithThisDate = (TT5.date_c == thisDate);
    % Extract all rows with this date and put into a cell array.
    theseData = TT5(indexesWithThisDate, 1:3); % Get XC and C
    % Assign to a new cell
    c{i} = theseData;
end

for i= 1: size(unidate)
    singledata = c(i)
    indexWithMinT = (singledata{1,1}.T == min(singledata{1,1}.T))    
    newData =  TT5(indexWithMinT, 1:3)
    call{i} = newData
end

I have attached the whole piece of data ('TT5.mat'), does anyone know how to filter twice?

Thanks for your help!

Answer 1

[jonas]

It is definitely not necessary to split the data if the purpose is to calculate the daily minimum temperature. Here's how to get the daily minimum:

TT=retime(TT5,'daily','min');

Do you still want to split the data? If so, do you want a cell array where each cell holds only the temperature on each day? To me this seems like a very bad idea, because you will lose information on when the content of the cell was recorded.

Of course, the downside here is that you do not get the other data corresponding to the minimum daily temperature, but you end up with the daily minimum of all variables. If you want the former, then I suggest this approach instead:

% Load data
data=load('TT5.mat')
TT=data.TT5;

% Split variables and time
Vars=TT{:,:};
t=TT.date_c;

% Group daily data
G=findgroups(t)

% Get rows with minimum daily T
out=splitapply(@(x)x(myfun(x(:,1)),:),Vars,G);

function [id]=myfun(x)
[~,id]=min(x);
end

I have checked so that the temperature is correctly stored!

[Peiying LI]

Thank you Jonas and your help again!

I need to split it because I need information in other columns (XC and C) for calculation at next steps, so minimum T is just a condition that has to be met.

Additionally, I will use a loop to do the calculation for every single day, so separated data by date is a must.

[jonas]

No problemo! I have updated the solution. The output is now a matrix where each row correspond to the daily minimum temperature and the corresponding XC and C (not minimum, just recorded at the same time as the minimum T).

[Peiying LI]

Thanks, this approach is clear but I think it automatically omits certain rows, because there should be at least 26 rows of XC and C within the daily minimum T. I will lose accuracy for further calculation.

[jonas]

Okay, I admit that this was clear from the original question. Change the code to the following and you will get the cell array that you requested initially:

% Get rows with minimum daily T
out=splitapply(@(x){x(myfun(x(:,1)),:)},Vars,G);

function [out]=myfun(x)
[val,id]=min(x);
out=find(x==val);
end

The vast majority of cells have more than 26 entries, but a few have less.

[Peiying LI]

Jonas you are so helpful! I was trying to change the function and didn't expect such quick response.

I would definately thank you in my dissertation acknowledgement when I finish it. Have a nice day!