Out of memory error with 8GB and 64bit

Hi,

I am working with lots of memory and I am getting an out of memory error.

>> memory
Maximum possible array:     16146 MB (1.693e+10 bytes) *
Memory available for all arrays:     16146 MB (1.693e+10 bytes) *
Memory used by MATLAB:      1080 MB (1.132e+09 bytes)
Physical Memory (RAM):      8111 MB (8.505e+09 bytes)

*  Limited by System Memory (physical + swap file) available.

The model is quite large. It's a script with dozens of matrixes 50k*50k. I know I use intensive functions as copulas, kron function, nested for and if's.

But this model is being developed for a long time and I really need it to run. What can I do?

Am I being inefficient or am I simply asking the impossible? This is a 64bit version.

Thanks a lot,

[Walter Roberson]

Can you switch the large arrays to single precision, so they use half as much memory?

[Alexandra]

I don't understand what's at stake by changing to single precision. Don't understand the difference..

[Stephen23]

So named "single" floating point values take up exactly half the memory of "double" floating point values, but have a lower precision, exactly as the documentation states "Because MATLAB stores numbers of type single using 32 bits, they require less memory than numbers of type double, which use 64 bits. However, because they are stored with fewer bits, numbers of type single are represented to less precision than numbers of type double."

http://www.mathworks.com/help/matlab/matlab_prog/floating-point-numbers.html

Converting to single would be an easy way to use less memory, if the precision requirement of your algorithm and data permit this.

[Alexandra]

Ok. I see the difference is in the number of decimals ("32-bit single-precision variables represent data to about seven decimal places, which is less accurate than doubles"). Since I work with €, I have no interest in computing bellow cents and I will try single.

Thanks,

Answer 1

[Lessmann]

Hi,

considering the mentioned size of a matrix, the matrix would occupy ~20GB or ~10GB in the case of single precision. With the 16GB available memory you are asking the impossible.

 >> M = zeros(50000,50000);
 >> whos
  Name          Size                     Bytes  Class     Attributes

   M         50000x50000            20000000000  double

To the point what you cando, take a good look at your script and see if the problem can be broken down into smaller pieces (not processing all the data at once).

If the matrix is sparsely populated, you could have a look into sparse matrices.

[Alexandra]

Hi there, I am also considering to break down the problem. Maybe you can help me. If I run half the script and save the variables, then I run the rest of the problem in a different script why would this save me memory? I would still have to join all the variables in the workspace.

Thanks,

[Lessmann]

I didn't meant to break down the script but the data. Sometimes it is possible divide the data into chunks, process them independendly of each other and combine the results afterwards.

For example, process 50 times a 1000x50000 matrix.

But if such an approach is feasible depents on the problem at hand.

[Alexandra]

Ok thanks, that is more difficult to apply because I apply operations that involve all rows or columns at once, like average, sumif etc. But if I can't solve it I will think about doing that..

[Walter Roberson]

sum and average do not require having all of the data available at the same time.

Consider that sum([1 2 3 4]) is 1+2+3+4 which is the same as (1+2)+(3+4) which is sum([1 2])+sum([3 4]). Therefore you can break your data up into smaller blocks, do the sum of each block, and then add the subtotals.

Likewise, average is sum divided by the number of elements, so you can break the data up into blocks, do the sum of each block, add the subtotals, and then divide the total by the number of elements. Also, in the special case that all the blocks are the same size, you can mean() the mean()'s of the blocks (a technique that will not work if any block is a different size than the others.)