How to subtract mean daily values from hourly data?

So I have flow data in an hourly time period, I was able to calculate the daily mean flow of these hourly values. Now I want to subtract the hourly flow values from the daily flow values to see how much change there is from the mean on a hourly basis.

Here is what I have so far:

Dates=datevec(DateandTime);
% Group by day
[unDates, ~, subs] = unique(Dates(:,1:3),'rows');
% Accumulate by day
AVG=[unDates accumarray(subs, Flow, [], @mean)];
f=AVG(:,4);
t=datenum(AVG(:,1:3));

figure
plot (x,y,t,f,'r')
legend('Hourly Flow','Daily Flow','location','best');
datetick ('x', 'mm-dd-yy', 'keepticks')
xlabel('Time');
ylabel('Flow [cfs]');
title('Daily Flow');
% axis tight

%%Difference from average flow
figure
n=datenum(AVG(:,1:3));
if n~=x
  DeltaFlow=Flow-AVG(:,4);
end

I keep getting an error matrix do not agree.

[Geoff Hayes]

Jen - which line throws the error? Is it

 DeltaFlow=Flow-AVG(:,4);

If it is, then check the dimensions of Flow. I'm pretty sure that AVG(:,4) will be a mx1 column (where m is the number of rows in AVG) and so Flow will also have to be an mx1 column in order for the subtraction to succeed. For an operation of this sort (subtraction) the two matrices must have the same dimension.

[jen]

The error is in the first line of the if statement. I am trying to make it match the dates to the daily values with if n~=x. n is a 348x1 and x is 33292x1.

If the dates are a match I want it to subtract the corresponding flow value of x and subtract the average of that day.

[Geoff Hayes]

So you see the problem then - you are comparing a 348x1 matrix with a 3329x1 matrix. Does n correspond to the daily flow, and x to the hourly flow? How do you wish to compare the two? If one is a subset of the other, or..?

It's not clear to me what you mean by make it match the dates to the daily values. The ~= is the "not equals" operator.

[jen]

I understand the error.

Maybe it will help if I show you some of my data. For example n= 735434 735435 735436 735437 735438 735439 numbers of days

x: 735434.708333333 735434.718750000 735434.729166667 735434.739583333 735434.750000000 735434.760416667 735434.770833333 735434.781250000 735434.791666667 735434.802083333 735434.812500000 735434.822916667 15 minute values

So in some way I want to match n (daily) values to x (15 minute time) if they are in the same day.

If the dates are a match I want it to subtract the corresponding flow value of x and subtract the average of that day.

[ccs]

Can someone please assist me on this: I have data for about 3 year (7 households electrical consumption). I am supposed to write a program in matlab that allows me to select any data (household, dates, etc) and calculate (and plot) daily, weekly average, weekend average, etc. Last time i used matlab was almost 4 years ago.......can someone please give and advise me on a good start?

Thanks everyone in advance.

[ccs]

I have data of every second (huge!)and would like to use two seconds as mean time base

[Geoff Hayes]

Sesilia - please create a new question for this problem.

[ccs]

Ok. Geoff Hayes, thanks, that was too messy. I think this one below is more clear.

I want to generate a power consumption graph. I have measured data (power, energy) for every two seconds, for say three year.

Challenge: to write a time defined program that plots every measurement (scatter) for any given day or days (between 00:00 & 24:00).

I hope this is clear....requests for more clarity are welcome.

Thank you all

[ccs]

Sorry Geoff Hayes, I get what you meant now.

I created a new question, thanks for your guide.

Appreciated. :)

Answer 1

[Joseph Cheng]

So, based on what you've done so far here is a spiffied up version:

%%generate sample data.
dateN = datenum('03-Sep-2014 15:26:10');
total15minsamps = 4*24*4; %4 days * 24 hours* 4 (15 min sec).
DATA = zeros(total15minsamps,7);
for ind = 1:total15minsamps %sample every 15 min.  
    data= 10+5*sin(2*pi/12*ind);    
    dateN = addtodate(dateN,15,'minute');
    Dvect = datevec(dateN);
    DATA(ind,:) = [Dvect data];
end
%%parse data
Dates = DATA(:,1:6);
Flow = DATA(:,end);
QuarterH_Time = datenum(DATA(:,1:end-1));
%%get hour's average flow
[unDates, HourSep, subs] = unique(Dates(:,1:4),'rows');
% Accumulate by hour
Hour_Data=[unDates accumarray(subs, Flow, [], @mean)];
Hour_AVG_flow=Hour_Data(:,5);
Hour_Time=datenum([Hour_Data(:,1:4) zeros(length(Hour_Data),2)]);

%%get day's average flow
[unDates, DaySep, subs] = unique(Dates(:,1:3),'rows');
% Accumulate by day
Day_Data=[unDates accumarray(subs, Flow, [], @mean)];
Day_AVG_flow=Day_Data(:,4);
Day_Time=datenum(Day_Data(:,1:3));
Day_Time = Day_Time + QuarterH_Time(1)-Day_Time(1); %offset so we don't start at the begining of the day

%%build subtraction matrix
DaySep = [DaySep;length(Flow)+1];
DeltaFlow = zeros(size(Flow));
for ind =1:length(DaySep)-1
    POI = DaySep(ind):DaySep(ind+1)-1;
    DeltaFlow(POI)=Flow(POI)-Day_AVG_flow(ind);
end

%%lets plot stuff
figure,plot(Hour_Time,Hour_AVG_flow,Day_Time,Day_AVG_flow,QuarterH_Time,DeltaFlow);
legend('Hourly Flow','Daily Flow','15 min data - day average','location','best');
datetick ('x', 'mm-dd-yy', 'keepticks')
xlabel('Time');
ylabel('Flow [cfs]');
title('Daily Flow');

Basically most of the stuff you can ignore because i needed to do some extra stuff to generate dummy data. But keep close attention to what i was doing with the DaySep array. Since we have two different length arrays that are based on the data i needed to use this value that is given by the unique() function to determine which indexes were deleted. With that i now know which 15 min data points correspond to which day.

[jen]

Thank you so much!!