Resolve “Out of Memory” Errors

This topic explains several strategies you can use in situations where MATLAB® runs out of memory. MATLAB is a 64-bit application that runs on 64-bit operating systems. It returns an error message whenever it requests a segment of memory from the operating system that is larger than what is available.

MATLAB has built-in protection against creating arrays that are too large. By default, MATLAB can use up to 100% of the RAM (not including virtual memory) of your computer to allocate memory for arrays, and if an array would exceed that threshold, then MATLAB returns an error. For example, this statement attempts to create an array with an unreasonable size:

A = rand(1e6,1e6);
Error using rand
Requested 1000000x1000000 (7450.6GB) array exceeds maximum array size
preference. Creation of arrays greater than this limit may take a long
time and cause MATLAB to become unresponsive. See array size limit or
preference panel for more information.
See Workspace and Variable Preferences for information on adjusting this array size limit. If you turn off the array size limit, then MATLAB returns a different error:

A = rand(1e6,1e6);
Out of memory. Type "help memory" for your options.

No matter how you run into memory limits, there are several resolutions available depending on your goals. The techniques discussed in Strategies for Efficient Use of Memory can help you optimize the available memory you have, including:

If you are already using memory efficiently and the problem persists, then the remaining sections of this page contain possible solutions.

Leverage tall Arrays

Tall Arrays for Out-of-Memory Data are designed to help you work with data sets that are too large to fit into memory. MATLAB works with small blocks of the data at a time, automatically handling all of the data chunking and processing in the background. There are two primary ways you can leverage tall arrays:

  1. If you have a large array that fits in memory, but you run out of memory when you try to perform calculations, you can cast the array to a tall array:

    B = tall(A)
    This method enables you work with large arrays that can fit in memory, but that consume too much memory to allow for copies of the data during calculations. For example, if you have 8GB of RAM and a 5GB matrix, casting the matrix to a tall array enables you to perform calculations on the matrix without running out of memory. See Convert In-Memory Arrays to Tall Arrays for an example of this usage.

  2. If you have file or folder-based data, you can create a datastore and then create a tall array on top of the datastore:

    ds = datastore('path/to/data.csv');
    tt = tall(ds);
    This method gives you the full power of tall arrays in MATLAB: the data can have any number of rows and MATLAB does not run out of memory. And because datastore works with both local and remote data locations, the data you work with does not need to be on the computer you use to analyze it. See Work with Remote Data for more information.

Leverage the Memory of Multiple Machines

If you have a cluster of computers, you can use the Parallel Computing Toolbox™ and Distributed Arrays (Parallel Computing Toolbox) to perform calculations using the combined memory of all the machines in the cluster. This enables you to operate on the entire distributed array as a single entity. However, the workers operate only on their part of the array, and automatically transfer data between themselves when necessary.

Creating a distributed array is very similar to creating a tall array:

ds = datastore('path/to/data.csv');
dt = distributed(ds);

Load Only as Much Data as You Need

Another possible way to fix memory issues is to only import into MATLAB as much of a large data set as you need for the problem you are trying to solve. This is not usually a problem when importing from sources such as a database, where you can explicitly search for elements matching a query. But this is a common problem with loading large flat text or binary files.

The datastore function enables you to work with large data sets incrementally. This function underpins Tall Arrays for Out-of-Memory Data and Distributed Arrays (Parallel Computing Toolbox), but you can use it for other purposes as well. Datastores are useful any time you want to load small portions of a data set into memory at a time.

To create a datastore, you need to provide the name of a file or a directory containing a collection of files with similar formatting. For example, with a single file:

ds = datastore('path/to/file.csv')
Or, with a collection of files in a folder:
ds = datastore('path/to/folder/')
You can also use the wildcard character * to select all files of a specific type, as in:
ds = datastore('data/*.csv')
Datastores support a wide variety of common file formats (tabular data, images, spreadsheets, and so on). See Select Datastore for File Format or Application for more information.

Aside from datastores, MATLAB also has several other functions to load parts of files, such as the matfile function to load portions of MAT-files. This table summarizes partial loading functions by file type.

File TypePartial Loading
MAT-file

Load part of a variable by indexing into an object that you create with the matfile function. See Big Data in MAT Files for an example of this usage.

Text

Use the textscan function to access parts of a large text file by reading only the selected columns and rows. If you specify the number of rows or a repeat format number with textscan, MATLAB calculates the exact amount of memory required beforehand.

Binary

You can use low-level binary file I/O functions, such as fread, to access parts of any file that has a known format. For binary files of an unknown format, try using memory mapping with the memmapfile function.

Image, HDF, Audio, and Video

Many of the MATLAB functions that support loading from these types of files allow you to select portions of the data to read. For details, see the function reference pages listed in Supported File Formats for Import and Export.

Increase System Swap Space

The total memory available to applications on your computer is composed of physical memory (RAM), plus a page file, or swap file, on disk. The swap file can be very large (for example, 512 terabytes on 64-bit Windows®). The operating system allocates the virtual memory for each process to physical memory or to the swap file, depending on the needs of the system and other processes. Increasing the size of the swap file can increase the total available memory, but also typically leads to slower performance.

Most systems enable you to control the size of your swap file. The steps involved depend on your operating system:

  • Windows Systems — Use the Windows Control Panel to change the size of the virtual memory paging file on your system. For more information, refer to the Windows help.

  • Linux® Systems — Change your swap space by using the mkswap and swapon commands. For more information, at the Linux prompt type man followed by the command name.

There is no interface for directly controlling the swap space on macOS systems.

Set the Process Limit on Linux Systems

The process limit is the maximum amount of virtual memory a single process (or application) can address. In the unlikely case you have set this preference, it must be large enough to accommodate:

  • All the data to process

  • MATLAB program files

  • The MATLAB executable itself

  • Additional state information

64-bit operating systems support a process limit of 8 terabytes. On Linux systems, see the ulimit command to view and set user limits including virtual memory.

Disable Java VM on Linux Systems

On Linux systems, if you start MATLAB without the Java® JVM™, you can increase the available workspace memory by approximately 400 megabytes. To start MATLAB without Java JVM, use the command-line option -nojvm. This option also increases the size of the largest contiguous memory block by about the same amount. By increasing the largest contiguous memory block, you increase the largest possible matrix size.

Using -nojvm comes with a penalty in that you lose many features that rely on the Java software, including the entire development environment. Starting MATLAB with the -nodesktop option does not save any substantial amount of memory.

See Also

Related Topics