How to pre-allocate an empty matrix ?

here is part of my code...

...
for l = 1:length(WtLabDt)                                               
    appnd_data = [];
    for m = 1:length(s.labs)                                          
        appnd_data = [appnd_data;e.(s.labs{m})(:,l)];                  
    end
    OVR_MEAN(:,l) = mean(appnd_data);                                 
end

Matlab is showing light red lines under 'appnd_data' (in line 4 on lhs) asking me to pre-alloacte it for speed. Does line 2 not pre-allocate it ?

[Greg]

Regardless of pre-allocation,

 appnd_data = [appnd_data;e.(s.labs{m})(:,l)];

Makes "appnd_data" larger every time it is called. It is more efficient to use indexing on the left side of the statement to fill in the appropriate portion of the pre-allocated matrix.

Run the "whos" command after line 2 and at the very end. You'll see the size in bytes of "appnd_data" grows.

[Stephen23]

"Does line 2 not pre-allocate it ?" No, it does not. Read the MATLAB documentation to know more:

https://www.mathworks.com/help/matlab/matlab_prog/preallocating-arrays.html

Answer 1

[Jan]

A pre-allocation would be:

    size1      = cellfun('size', s.labs, 1);
    appnd_data = zeros(sum(size1), 1);
    iPos       = 1;
    for m = 1:length(s.labs)                                          
       fPos = iPos + size1(m);
       appnd_data(iPos:fPos-1) = e.(s.labs{m})(:,l);
       iPos = fPos;
    end

But is there any reason to create this large vector, when all you want is to get its mean?

    allSize1 = sum(cellfun('size', s.labs, 1));
    S = 0;
    for m = 1:length(s.labs)                                          
       S = S + sum(e.(s.labs{m})(:,l));
    end
    OVR_MEAN(:, l) = S / allSize1;

Perhaps I did not understand, why the vector OVR_MEAN(:, l) is assigned, although the result is a scalar. But the ideas should got clear now:

1. Pre-allocation means reserving memory for the final array at once instead of letting the array grow iteratively
2. If you can avoid to create large temporary arrays, do this.

[Amandeep Singh]

Thanks! I have taken parts for your code and it made my code 10 times faster. I always am hesitant to pre-allocate as zero array since zeros from pre-allocation can be mistakenly read as data if you made a mistake in your code. That is why i made an empty matrix. I want to save each OVR_MEAN in a different column since each column represents an emission constituent for me and that is why I am indexing it. Now in your section, why are you not using 'm' for 'appnd_data'? why create fPos and iPos?

[Stephen23]

"That is why i made an empty matrix"

But an empty array does not preallocate the array. If you do not want to use zeros you could preallocate using a NaN array.

[Jan]

@Amandeep Singh: "Pre-allocation" and "empty matrix" are contradictions. Creating an empty matrix is not a pre-allocation and the iterative growing of an array is a severe loss of efficiency.

OVR_MEAN(:,l) = mean(appnd_data)

" I want to save each OVR_MEAN in a different column since each column represents an ..." It does not matter what the data represent. The code looks like the right hand side is a scalar, then this would be more clear:

OVR_MEAN(1, l) = mean(appnd_data)

or

OVR_MEAN(l) = mean(appnd_data)

"why are you not using 'm' for 'appnd_data'?" In:

appnd_data(iPos:fPos-1) = e.(s.labs{m})(:,l)

the right hand side is a vector. To copy it into the pre-allocated vector appnd_data you have to define a range, but m would be a scalar only. Perhaps all "e.(s.labs{m})" have the same number of elements along the 1st dimension, then the code could be simplified, but you did not explain this.

Answer 2

[Roger Stafford]

Rather than stacking your ‘appnd_data’ successively by:

   for m = 1:length(s.labs)                                          
     appnd_data = [appnd_data;e.(s.labs{m})(:,l)];                  
   end
   OVR_MEAN(:,l) = mean(appnd_data);

it should instead be done this way:

ix = 0;
for m = 1:length(s.labs) 
  t = e.(s.labs{m})(:,l);
  append_data(ix+1:ix+size(t,1)) = t; 
  ix = ix+size(t,1); 
end
OVR_MEAN(:,l) = mean(appnd_data(1:ix));

Before doing this you should allocate enough memory for ‘append_data’ for the longest of these final ix values using the ‘zeros’ function.