How could MATLAB store large scale of data.

Hi, All,

I have to use a dataset as large as ones(3^N1,3^N2,10^3), where N1 and N2 can be 7~14. However, matlab will report "Out of memory" even for ones(3^14,1000). Is there a way to let matlab store such a large array.

I can split my data to several parts and store the sub-data. However that is very inconvenient.

I wonder if MATLAB can extend the limit, and handle large scale data. Thanks.

Answer 1

[Walter Roberson]

If you are able to use 8 bits per value, you could use

ones(3^N1, 3^N2, 10^3, 'uint8')  %or 'int8' as appropriate.

That would occupy 4.45 gigabytes, which would be too much for any 32 bit version of MATLAB (32 bits is 4 gigabytes of address space.) It would, however, be feasible with a 64 bit MATLAB with as little as 8 gigabytes of RAM.

If you need to use double precision, then that occupies 8 bytes per location, and so requires 35.64 gigabytes for the array. You would need a 64 bit MATLAB for sure, and you would need probably 40 or so gigabytes of RAM. If that much RAM is not feasible for you, then using a double precision array of that size is not feasible for you.

This is not a matter of how MATLAB uses its memory: this is just a matter of having enough memory to store the array at all.

Note: your options are different if the array is reasonably sparse.

[C Zeng]

Thanks, Walter, my array is not sparse, I need to store them to use. So it can be feasible right? Let me try it on 64bit machine.

[C Zeng]

However, now the erros is : Maximum variable size allowed by the program is exceeded.

[C Zeng]

Hi, Walter, could you address my concern? I cannot define array like ones(10^10,1) here.

[Cedric]

The default type/class in MATLAB is double; its size in byte is 8.

 >> a = 5
 a =
     5
 >> whos
 Name      Size            Bytes  Class     Attributes
  a         1x1                 8  double

When you execute ones(10^10,1), you try to build an array of 10 giga-doubles, which means 10^10*8 = 80 gigabytes. You can specify the type when you call ones(), e.g.

 >> ones(1,1,'uint8')
 ans =
    1
 >> whos
 Name      Size            Bytes  Class     Attributes
 ans       1x1                 1  uint8

you see that uint8 objects have a size of 1 byte. Even using uint8 though, the call that you try to execute would require 10GB of RAM.

[C Zeng]

Yes, Cedric,

Thanks for showing the details. I have a 4GB RAM, now I see why Matlab reports out of memory. Let me work on server to see if it allows me to use 10GB RAM.

[Cedric]

But be aware that uint8 can store only 8 bits unsigned integers, which means integers in the range 0-255. It might not be adapted to the nature of your data, which might lead to e.g. truncation..

 >> a = uint8(200)
 a =
    200
 >> 2*a                   % Will it be 400?
 ans =
    255

I can say by experience that often, when people would need much more RAM than what is present on an average computer to perform their computation, the problem is not that computers don't have enough RAM, but that the approach is flawed.

[C Zeng]

Yes, Cedric, will be careful on that. Yes, uint8 is very limited range, and I probably use uint16. However that will exceed the 10G limit.

Answer 2

[Cedric]

If your data is sparse, you can build 1000 sparse matrices or use some FEX function that will allow you to create ND sparse matrices.

The ones(3^14, 1000) that you tried to evaluate would take more than 38GB RAM to be stored as double. It certainly indicates that your approach is not appropriate. One thing that you could do though, is to store your data using a "smaller" type/class, i.e. any decent enough type (for the nature of your data) that is stored on 1, 2, or 4 bytes. If your data were made of unsigned integers in the range 0-255, you could use uint8 that would require 1 byte per element instead of 8.

[C Zeng]

Yes, my data can be saved as uint16. Thanks.