MATLAB transfer speed while scripting

I am trying to read data from a series of .mat files (~120GB each). I am using a 'for' loop to iterate over the files and load them into the workspace (I'll mention matfile shortly). The first file loads fine in about 10 minutes at 1.5 Gbps. After the first file, the transfer speed tanks to Kbps. It is not the individual files because they load in separately just fine. It is when I try to script the process that the phenomena occurs.

For those wondering, I did try using matfile (as opposed to loading into the workspace), but experienced the same thing. To read in one slice of the data took less than a second, but as soon as I script the process to loop over multiple slices, there is an exponential slow down in the processing speed.

I would expect the speed to process all the data to be linear.

sample code (psuedo):

fids = dir(path\to\mat\files)
for idx = 1:length(fids)
    load(fids(idx))
    # some processing
end

[Steven Lord]

What does the memory usage of MATLAB (and the amount of available memory on your system) look like as you go through this process?

Can you provide more details behind what "# some processing" does? Does it create additional copies of the (potentially very large) matrices you loaded from the file?

As an analogy, it's relatively easy to put a book on a shelf in an empty bookcase. But as you start putting more and more books on the shelves, fitting later books in (ideally, if they're part of a series, next to the other volumes in the series) gets to be trickier. You may need to move books already placed up or down a shelf to free up space for the new book. And at some point, you may need to take books out of the bookcase and put them in storage, on another bookcase or in the attic etc., to fit new books. [Trust me, I know this all too well!]

Your computer's memory is like that bookcase. Loading and operating on that first large array may consume enough space to require MATLAB / the OS to shuffle books/memory around or shunt them into storage/swap space.

[Walter Roberson]

A common problem is that each time you plot a new figure or new axes within a figure, the old graphics information is kept. So if you are plotting into new figures each time, the stored graphics information builds up.