C++ engPutVariable() memory limitation

It seems that engPutVariable has memory limitation and can't send big matrices. Withing matlab I have no problem allocating:

x = rand(259778664,3);

but the code below crashes:

#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include "engine.h"
#include <iostream>
#include <cmath>
#include <cfloat>
#include <fstream>
#include <conio.h>

using namespace std;

int main()
{
    Engine *ep;
    if (!(ep = engOpen(""))) {
        fprintf(stderr, "\nCan't start MATLAB engine\n");
        return EXIT_FAILURE;
    }

      unsigned *rowind;
      unsigned *colind;
      double *vals;
      size_t n;    
      if ( 0 ) {
      } else {
          n = 259778664; //43296444;
          rowind = new unsigned[n];
          colind = new unsigned[n];
          vals = new double[n];
      }
      cout << "n=" << n << endl;

      mxArray *M = mxCreateDoubleMatrix(n, 3, mxREAL);
      double *pM = mxGetPr(M);
      for (size_t i = 0; i < n; ++i)
      {
          pM[0*n+i] = double(rowind[i]+1);
          pM[1*n+i] = double(colind[i]+1);
          pM[2*n+i] = double(  vals[i]  );
      }

      delete[] rowind;
      delete[] colind;
      delete[] vals;

      for ( int i = 0 ; i < 4 ; i++ ) {
          char s[100];
          sprintf(s, "M%d", i);
          cout << s << endl;
          cout << "Press a key to load.." << endl;
          getch();
          int res = engPutVariable(ep, s, M);
      }

      mxDestroyArray(M);

      engClose(ep);

      return EXIT_SUCCESS;
  }

[Ken Atwell]

Does it crash on the first loop iteration, a later loop iteration, or somewhere else? In other words, how much output to you get before the crash?

And, at the risk of asking the obvious, you're using 64-bit MATLAB and a 64-bit compiler, right? What platform are you on?

[Zohar]

Yeah, there's a bit more to the story, I wanted to keep it concise. I'm using a laptop with 24GB ram, win8.1, matlab2013a, visual studio 2013, everything 64 bits, and it crashes on the first iteration.

I tried it also on our linux centos server with 64GB ram dedicated to the job, gcc 4.9.2, matlab 2014a, everything 64 bits. No crash at all. BUT, when I loaded a specific data to put in the matrix (instead of leaving the allocated memory random and uninitialized), it crashed on the second iteration with the following error:

"Error using load Can't read file stdio."

Now, this is one of the annoyances with the buggy matlab (not the first time I stumble upon this kind of thing) version for linux. It doesn't crash where it supposes to, sometimes it just continues or does half a job, and only later crashes unexpectedly, and it took me a while to find the problem origin (the matrix originated from a huge model that I ran on the server). Here is the linux version with the specific matrix, where you build it with c.sh:

https://www.dropbox.com/s/8rxuhl01c0do0v2/mat.zip?dl=0

Answer 1

[Titus Edelhofer]

Hi,

I'm not sure but I guess the engPutVariable needs to copy the matrix from your C application to MATLAB. The two applications can't share the variable. Therefore the question is, if your machine is able to have two matrices of about 6 GB in memory...?

Titus

[Zohar]

I have 64GB ram.

[Titus Edelhofer]

O.k., then this is not the problem ;-)

Answer 2

[Zohar]

I opened Technical Support Case #01197310. The reply is below. I still think it's a bug, some 32bit limitation, and there is no reason for it, and it should be resolve ASAP.

Please find code attached which can be used to test the limit of the data that can be passed to the 'engPutVariable' function. As stated earlier, there is a limit of 2GB on the 'mxArray' that can be passed to the function. The Development team is aware of the limitation and might consider working on it in the future releases.

Each element of 'mxArray' is of type 'double' i.e. 8 bytes. Therefore, 

Number of rows or value of n that can be passed = 2GB/ (8*3):
 2*1024*1024*1024/8/3 = 89478485.3

In the code, we have 2 test scenarios:
n = 89478480; //This works.
n = 89478486; //This does not work.

To check if the 'engPutVariable' was executed successfully, we check the returned boolean value.