How to optimize a function for a large data set?

Hi,

I am writing a function to perform relatively basic operations on a very large data set in a single column vector (about 32 million points) and produce a 4 column vector as an output. I was wondering if there is anything in the code that could be optimised to save processing power (and time), since it is taking me over 3 hours to run through just a tenth of the whole data set. Code below:

Thank you in advance for your help.

    function batterystatus = freqdata( freq, batpower, batenergy, batinitial , batefficiency)

    %This function takes a vector of data (large data set), the battery power, energy and initial charging status and efficiency.

    max = length(freq);

    Energy0 = batenergy * batinitial;

    uplimit = 50.015;               
    lowlimit = 49.985;              
    statuplimit = 50.5;            
    statlowlimit = 49.5;           

    batterystatus(1 ,1) = Energy0;                        

    batterystatus(1, 2) = 0;                            

    batterystatus(1, 3) = 0;                                               

    batterystatus(1,4) = freq(1);    

    for i = 2:max    

        batterystatus(i, 4) = freq(i);

         if freq(i) > uplimit

            if batterystatus(i - 1, 1) >= batenergy                                      

                batterystatus(i ,: ) = batterystatus(i -1 ,: );

            else

            batteryscaling (i, 1) = (freq(1) - uplimit) * batpower / (statuplimit - uplimit);                                    

            batterystatus(i, 1) = batterystatus(i - 1 , 1) + (batteryscaling(i, 1)  * 1 / 3600);                

            batterystatus(i, 2) = batterystatus(i - 1 , 2) + (batteryscaling(i, 1)  * 1 / 3600)/ batefficiency; 

            batterystatus(i, 3) = batterystatus(i - 1, 3);                                        

            end

        elseif freq(i) < lowlimit

            if batterystatus(i - 1, 1) <= 1                                      

                batterystatus(i ,: ) = batterystatus(i -1 ,: );

            else

            batteryscaling(i, 1) = (freq(1) - statlowlimit) * batpower / (lowlimit - statlowlimit) ;

            batterystatus(i ,1) = batterystatus(i - 1 , 1) - (batteryscaling(i, 1)  * 1 / 3600)/ batefficiency; 

            batterystatus(i, 2) = batterystatus(i - 1 , 2);

            batterystatus(i, 3) = batterystatus(i - 1 , 3) + (batteryscaling(i, 1) * 1 / 3600);

            end

        else

            batterystatus(i ,: ) = batterystatus(i -1 ,: );

         end

    end

[Adam]

doc profile

is the best start point to show you what is actually slow. There's no point just guessing what is slow, even if it seems obvious.

[Rik]

Also, a very convoluted and difficult to read method that removes the need for a for-loop can be worth it.

And don't overwrite the useful function max.

[Alexis Stavropoulos]

Hi Adam,

Thanks. I profiled the function starting on a small data set and kept increasing until I noticed the time required went up considerably. To go through 23k items it took 0.3s. To go through 230k items it took 34seconds, which seems just like a 10x increase due to 10x more data. The profiler identified the following lines as the one which took the longest.

time    calls     line
12.94   90324     38     batterystatus(i, 1) = batterystatus(i - 1 , 1) + (batteryscaling  * 1 / 3600);

Any suggestions? This seems like a simple operation, and I am not sure how I could simplify it. Or is this time normal to analyse 230000 lines?

[Adam]

batterystatus appears to be growing in the loop. I don't see any preallocation for the first dimension of this. You should have a warning about this in your code though with an orange underline.

If you don't know the size in advance it is still usually faster if you can work out an upper limit for the size, pre-size it to that and then trim it at the end based on how many elements you actually put in.

[Alexis Stavropoulos]

Hi Adam,

Thanks, that solved it. I initialised batterystatus with a length = max, and now it took only 1.4s to run 2.3m lines.

[Rik]

You should really not overwrite internal functions. It is a bad habit that can cause bugs that are really hard to find. I would advise you to un-learn it as fast as you can.

[Alexis Stavropoulos]

Sorry Rik, what internal function are you referring?

[Adam]

Especially so if you do on one line:

max = length(freq);

and then later you now say you do

length = max;

This will work in the case you have (well, I assume since you don't say you get an error), but will soon come back to bite you as Rik says.

First here you use the length function (in itself this is a bad idea - length is almost never a good function to use on an array - numel is better if you know you have a vector or size with the relevant dimension for a matrix or higher dimension array). Then you assign its result to the variable max, thus hiding the max function.

Then you overwrite the length function also with the contents of the variable max which have already obliterated the function of the same name!

So if you where to try to use the function 'length' again lower down your code (or 'max') you would get a nasty surprise and most likely an error about array indices being out of bounds or non-positive!

[Alexis Stavropoulos]

Got you. I changed it so now I am not naming any variables using Matlab functions. Thanks

[Alexis Stavropoulos]

Your answer has definitely helped, but I have added a few things to the function and it is taking a bit too long in datetime operations. See code below:

function batterystatus = freqdataV3( freq, date,FFRExcWindow1, FFRExcWindow2, REDstart, REDend, batpower, batenergy, batinitial , batefficiency)

Energy0 = batenergy * batinitial;

uplimit = 50.015;               

lowlimit = 49.985;              

statuplimit = 50.5;           

statlowlimit = 49.5;          

batterystatus(:,7) = freq;     

batterystatus(1 ,1) = Energy0; 

batterystatus(1, 2) = 0; 

batterystatus(1, 3) = 0;             

batterystatus(1, 4) = 0;            

batterystatus(1, 5) = 0;            

batterystatus(1, 6) = 0;           

FFRfinishdate = datetime(year(date), month(date), day(date),floor(FFRExcWindow1), rem(FFRExcWindow1, 1) * 60, 0);

FFRstartdate = datetime(year(date), month(date), day(date),floor(FFRExcWindow2), rem(FFRExcWindow2, 1) * 60, 0);

REDstartdate = datetime(year(date), month(date), day(date), floor(REDstart), rem(REDstart, 1) * 60,0); 

REDenddate = datetime(year(date), month(date), day(date), floor(REDend), rem(REDend, 1) * 60,0); 

for i = 2:numel(freq)    

      if weekday(date(i)) >= 2 && weekday(date(i)) <= 6 && ge(date(i),FFRfinishdate(i)) && lt(date(i),REDstartdate(i))          

               if batterystatus(i - 1, 1) >= (batenergy - 1)                               
              batterystatus(i ,1:6 ) = batterystatus(i -1 ,1:6 );

               else
                  (...)

When I run the profiler, it highlights the "if weekday(date(i))..." in RED. The whole function has a few more "if"s below, but they have similar conditions. The profiler shows that almost half the total time is due to datetime.subsref. Is this just to perform datetime operations such as "ge" and "lt"? If so, is there any clever away around it that would save me time to run it?

Thanks in advance.