shared matlab-engine python connections occationally fail with an Unable to connect to MATLAB session error

On a RHEL7 platform, we are running a shared (synchronous) MATLAB session as a systemd service. We are also running 3 python systemd services that connect to the shared matlab engine using matlab_eng = matlab.engine.connect_matlab("OUR_SESSION_NAME_HERE"). After successfully connecting to the shared matlab-engine, the python service will do some MATLAB processing, then do a matlab_eng.quit() to disconnect from the shared matlab-engine. I have 4 questions that I hope somebody could answer. See below.

Regards Debbie

1. Is matlab_eng.quit() the correct way to disconnect from the shared matlab-engine?
2. Can you confirm that each of the 3 python systemd services will have it's own MATLAB workspace?
3. After our python services have run awhile, we start seeing trouble connecting to the shared matlab-engine. The stack trace looks similar to as shown below. Why would these connection errors only occationally occur and how can we fix this? We have assumed that each of our python processes can use the shared matlab-engine concurrently, is this not correct?
4. As a follow-up to question 3, why is _future.result showing up in the trace? We started a synchronous matlab-engine, the documentation mentions future results only with respect to asynchronous matlab-engines.

<class 'matlab.engine.EngineError'>

Unable to connect to MATLAB session 'OUR_SESSION_NAME_HERE'.

matlab_eng = matlab.engine.connect_matlab('OUR_SESSION_NAME_HERE')

!! File "/usr/lib/python2.7/site-packages/matlab/engine/__init__.py", line 182, in connect_matlab

eng = future.result()

!! File "/usr/lib/python2.7/site-packages/matlab/engine/futureresult.py", line 67, in result

return self.__future.result(timeout)

!! File "/usr/lib/python2.7/site-packages/matlab/engine/matlabfuture.py", line 87, in result

handle = pythonengine.getMATLAB(self._future)

Answer 1

[Rushikesh]

Hello @Deborah,

I would like to address your questions in order:

1. When using a shared MATLAB engine instance, calling “matlab_eng.quit()” will terminate the running MATLAB engine. If you wish to disconnect from the shared MATLAB engine while keeping it available for other Python scripts, you should use the “matlab_eng.disconnect()” function. This will end the current Python script's access to the engine instance without terminating the engine itself.

2. If multiple Python services (three or more) are accessing the same MATLAB shared engine instance, they will share a common MATLAB workspace. If separate MATLAB workspaces are desired for each service, you will need to create multiple shared engine instances.

3. As of MATLAB version R2024b, there is a limitation on the number of attempts to connect to the same shared instance. It is advisable to minimize the number of connect engine instance calls in your service. For more details, please refer to the following documentation:

https://www.mathworks.com/help/matlab/apiref/matlab.engine.connect_matlab.html

If reducing the connect engine calls is not feasible, an alternative workaround is to programmatically restart the shared engine instance with the same name and reconnect the services after a few connection attempts.

4. The appearance of “future.result” in the trace, even when the call is synchronous, is due to the way MATLAB handles the “connect_matlab” function. Based on my experience with MATLAB shared engines, the “FutureResult” object is involved regardless of whether the call is synchronous or asynchronous. This doesn't necessarily mean your operations are asynchronous; it's just part of how the connection logic is structured in the library.

Hope this helps.