C++ engPutVariable() memory limitation

10 views (last 30 days)
Zohar
Zohar on 14 Jan 2015
Edited: Zohar on 19 Jan 2015
It seems that engPutVariable has memory limitation and can't send big matrices. Withing matlab I have no problem allocating:
x = rand(259778664,3);
but the code below crashes:
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include "engine.h"
#include <iostream>
#include <cmath>
#include <cfloat>
#include <fstream>
#include <conio.h>
using namespace std;
int main()
{
Engine *ep;
if (!(ep = engOpen(""))) {
fprintf(stderr, "\nCan't start MATLAB engine\n");
return EXIT_FAILURE;
}
unsigned *rowind;
unsigned *colind;
double *vals;
size_t n;
if ( 0 ) {
} else {
n = 259778664; //43296444;
rowind = new unsigned[n];
colind = new unsigned[n];
vals = new double[n];
}
cout << "n=" << n << endl;
mxArray *M = mxCreateDoubleMatrix(n, 3, mxREAL);
double *pM = mxGetPr(M);
for (size_t i = 0; i < n; ++i)
{
pM[0*n+i] = double(rowind[i]+1);
pM[1*n+i] = double(colind[i]+1);
pM[2*n+i] = double( vals[i] );
}
delete[] rowind;
delete[] colind;
delete[] vals;
for ( int i = 0 ; i < 4 ; i++ ) {
char s[100];
sprintf(s, "M%d", i);
cout << s << endl;
cout << "Press a key to load.." << endl;
getch();
int res = engPutVariable(ep, s, M);
}
mxDestroyArray(M);
engClose(ep);
return EXIT_SUCCESS;
}
  2 Comments
Zohar
Zohar on 14 Jan 2015
Yeah, there's a bit more to the story, I wanted to keep it concise. I'm using a laptop with 24GB ram, win8.1, matlab2013a, visual studio 2013, everything 64 bits, and it crashes on the first iteration.
I tried it also on our linux centos server with 64GB ram dedicated to the job, gcc 4.9.2, matlab 2014a, everything 64 bits. No crash at all. BUT, when I loaded a specific data to put in the matrix (instead of leaving the allocated memory random and uninitialized), it crashed on the second iteration with the following error:
"Error using load Can't read file stdio."
Now, this is one of the annoyances with the buggy matlab (not the first time I stumble upon this kind of thing) version for linux. It doesn't crash where it supposes to, sometimes it just continues or does half a job, and only later crashes unexpectedly, and it took me a while to find the problem origin (the matrix originated from a huge model that I ran on the server). Here is the linux version with the specific matrix, where you build it with c.sh:
https://www.dropbox.com/s/8rxuhl01c0do0v2/mat.zip?dl=0

Sign in to comment.

Answers (2)

Titus Edelhofer
Titus Edelhofer on 14 Jan 2015
Hi,
I'm not sure but I guess the engPutVariable needs to copy the matrix from your C application to MATLAB. The two applications can't share the variable. Therefore the question is, if your machine is able to have two matrices of about 6 GB in memory...?
Titus

Zohar
Zohar on 16 Jan 2015
I opened Technical Support Case #01197310. The reply is below. I still think it's a bug, some 32bit limitation, and there is no reason for it, and it should be resolve ASAP.
Please find code attached which can be used to test the limit of the data that can be passed to the 'engPutVariable' function. As stated earlier, there is a limit of 2GB on the 'mxArray' that can be passed to the function. The Development team is aware of the limitation and might consider working on it in the future releases.
Each element of 'mxArray' is of type 'double' i.e. 8 bytes. Therefore,
Number of rows or value of n that can be passed = 2GB/ (8*3):
2*1024*1024*1024/8/3 = 89478485.3
In the code, we have 2 test scenarios:
n = 89478480; //This works.
n = 89478486; //This does not work.
To check if the 'engPutVariable' was executed successfully, we check the returned boolean value.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!