Why does Matlab not complete if the runtime for a computation is too long?

(edited)

I've noticed that Matlab (2012b) won't complete a simulation I run if it lasts longer than about 10 hours. Why is this and what can I do about it?

i.e

An iteration may take 3 minutes. If I run 500 iterations, it should take 25 hours to solve. Instead Matlab runs on my PC for 3 days and doesn't finish. But, if I run 150 iterations, matlab finishes it happily in about 7.5 hours.

If I don't turn Matlab off manually (the x button) it will run for a week or more just going nowhere.

The clear() command can't be used after each iteration because i'm trying to make graphs that require lengthy computation times per point.

More details:

Desktop PC, I run Windows 7, I saw this issue on windows XP as well as on earlier versions of Matlab; all on the same PC. Ram usage never goes above about 500 megs. Core 2 duo, 3.0 gig processor. Machine is about 5 years old. 8 gigs ram. When I run simulations I don't use the machine for anything else, I switch to a laptop.

Also, when this has happened, I can still use other applications without a problem.

Part of what I'm not sure on, is it normal for Matlab do not like running computations beyond a certain time limit?

edit I believe these are all the required files if anyone wants to see what my code looks like. This is for an Ensemble Kalman Filter on the Lorenz (3D) system.

[C V Ambarish]

Hey Max,

You've probably forgotten about this by now :P, but were you able to find out the root cause? Have almost exactly the same problem and I have no clue at this point. Thanks.

My question if you want to take a look. Any comments/suggestions appreciated.

Answer 1

[Image Analyst]

We don't know. How could we? What operating system? How, EXACTLY, does MATLAB "die"? Are you managing memory well? Maybe you're running out of memory, though MATLAB will usually tell you that and not just die in a fiery steaming crash. Maybe you can explicitly use clear() at the end of functions to force it to clear variables immmediately. Maybe you can use cla('reset') before plotting to axes. It's hard to say exactly since we have no information other than you saying your program crashes after a long time.

[Max]

Sorry for the lack of detail. Edited my first post. If you need more info please ask.

[Image Analyst]

Can you answer my other questions? And try my suggestions? But I suspect you may have to send the code to the Mathworks if it's a real crash: "MATLAB.exe has encountered an error and needs to close".

FAQ: http://matlab.wikia.com/wiki/FAQ#After_installation.2C_MATLAB_crashes_or_gives_an_error_message_when_I_try_to_run_MATLAB.

[Max]

It's not a real crash. It just keeps running and running far beyond the normal computation times expected. It's not a memory leak, I can run other applications without a problem if I want to.

Answer 2

[Cedric]

You should output all the information that you can, i.e. loop counters, branch of conditional statements, etc. If it's slowing down your simulation too much, start outputting it after e.g. 200 iterations (at least for inner loops). This will allow you to pinpoint the region of your code where MATLAB is stuck and refine your tests.

For example

 t_start = tic ;
 t_last  = tic ;

 for mainCnt = 1 : 500
    fprintf( ['\n=== mainCnt = %d, %.2fmin from start, last it. took', ...
              ' %.2fmin.\n\n'], mainCnt, toc(t_start)/60, toc(t_last)/60 ) ;

    if mainCnt >= 200
       whos
       [userview, systemview] = memory ;
       fprintf( ' Sys memory: phys.-> avail. = %.2e, total = %.2e\n', ...
                systemview.PhysicalMemory.Available, ...
                systemview.PhysicalMemory.Total ) ;
       fprintf( ' Usr memory: used by MATLAB = %.2e\n\n', ...
                userview.MemUsedMATLAB ) ;
    end

    t_last = tic ;

    for k = 1 : 40
       if mainCnt >= 200
          fprintf( ' - Inner loop: k = %d\n', k ) ;
       end
       ...
    end

    if ...
       fprintf( ' - Outer loop, if statement: true\n' ) ;
       ...
    else
       fprintf( ' - Outer loop, if statement: false\n' ) ;
       ...
    end
 end

[Max]

Interesting. i will try this.

[Cedric]

You can even store the time of iterations and plot the evolution, e.g.

 maxIter = 500 ;
 t_iter  = zeros( 1, maxIter ) ;
 t_start = tic ;
 t_last  = tic ;

 for mainCnt = 1 : maxIter

    t_iter(mainCnt) = toc(t_last)/60 ;

    fprintf( ['\n=== mainCnt = %d, %.2fmin from start, last it. took', ...
              ' %.2fmin.\n\n'], mainCnt, toc(t_start)/60, t_iter(mainCnt) ) ;

    plot( t_iter ) ;
    grid on ;
    xlabel( 'Iteration # (outer loop)' ) ;
    ylabel( 't_{iteration} [min]' ) ;

    ...

[Cedric]

You can also plot memory information like this actually..

[Max]

The loop is still going, but I can see that it was 3 minutes per iteration in the beginning and is now up to about 7 minutes. If the attachment thing actually works I've attached what I think are all the necessary files.

[Cedric]

Did you plot the time per iteration? If it started at 3 minutes and stayed like that for a while, before jumping to 7 minutes, then Walter was probably right (even though, as he says, the slow down associated with swapping is generally a factor greater than 10). If the time per iteration is increasing linearly with the loop index, then it's difficult to say.. you could have the span of some integration which increases with the loop index, or maybe a function integrated which becomes steeper, oscillates more, etc. And there can be quite a few additional reasons.

What I would do if I were you is probably to plot the time per iteration as I suggested, some memory information as well maybe, but also some of the args that you pass to the ODE solver(s) (in particular the span), and the size of the output of this/these solver(s), so you can see how many time steps solver(s) go through.

If you have really no idea about what takes increasing time in the main loop, you can plot multiple tocs. Here is an illustration..

 maxIter   = 100 ;
 tocBuffer = zeros( maxIter, 4 ) ;      % Store 4 tocs per iteration.

 for k = 1 : maxIter
    tic_iterStart = tic ;

    % Roughly fixed time block 1.
    pause( (2+rand(1)) / 100 ) ;
    tocBuffer(k,1) = toc( tic_iterStart ) ;

    % Roughly fixed time block 2.
    pause( (1+rand(1)) / 100 ) ;
    tocBuffer(k,2) = toc( tic_iterStart ) ;

    % Increasing time block 3.
    pause( (k+rand(1)) / 100 ) ;
    tocBuffer(k,3) = toc( tic_iterStart ) ;

    % Roughly fixed time block 4.
    pause( (3+rand(1)) / 100 ) ;
    tocBuffer(k,4) = toc( tic_iterStart ) ;

    % Plot tocs
    plot( tocBuffer ) ;
    grid on ;
    xlabel( 'Iteration #' ) ;
    ylabel( 'toc() 1-4' ) ;
 end

This example emulates 4 blocks with 3 which take roughly a constant time over iterations and 1 whose run time increases with iterations. The plot looks like:

which makes it immediately clear that the issue is with block 3.

[Max]

Matlab is still running at this point. Computation time is roughly linearly increasing as the iteration increases. Your comment makes sense though, each iteration runs the EnKF over the Lorenz system with more error/noise and then stores a bunch of results and error measurements I'm interested in.

I'll do some plots tomorrow when it's done.

[Cedric]

Ok, then add figure(1) before the tocs plot

    % Plot tocs
    figure(1) ;
    plot( tocBuffer ) ;
    grid on ;
    xlabel( 'Iteration #' ) ;
    ylabel( 'toc() 1-4' ) ;

and plot e.g. the size (1st dimension) of the output of relevant solver(s) using the same mechanism but figure(2) before the plot.

You could also test a few other solvers.

Answer 3

[Walter Roberson]

You are slowly running out of RAM and your system is set to use virtual swap (to write to harddisk). When it starts needing to write to harddisk, everything will slow down by a factor of 10 to 100, but it will keep running.

[Cedric]

The original question (before the edit) mentioned that the OP could still use other applications when MATLAB seemed to be stuck. When a MATLAB run leads to memory swap on my system, I can almost not use any other application; the UI becomes unresponsive and I can't close MATLAB by clicking on [x]. It is so annoying that I setup a memory watchdog with ProcessLasso which kills MATLAB before swap happens. I agree that it could be swap though, maybe in a context different than mine (a lot of small arrays(?) whereas I am generally dealing with a few large arrays).

[Walter Roberson]

I suppose another possibility is that with the increased iterations, the user is encountering an infinite loop, such as expecting that (X + 1) is always greater than X, which is not true for any of the numeric data types MATLAB uses.