matlab -batch on SLURM prints errors when HOME is shared across many nodes

When launching MATLAB in batch mode on a SLURM cluster, stderr shows:

terminate called after throwing an instance of 'matlabconnector::installationsregistry::InstallationsRegistryError'
what(): InstallationsRegistry:getMatlabConnectorVersion: More than one MATLAB Connector records were found

This appears during startup (even for a simple command like matlab -batch "disp(version); exit").

sbatch command (PROGLIST=/work/share/.../matlab2024a/bin/matlab):

srun "$PROGLIST" -batch "$CMD"

Find files in ~/.MATLABConnector as below:

~/.MATLABConnector/b01r2n13/LatestInstall.info

~/.MATLABConnector/b01r2n13/UpdatePending.json

~/.MATLABConnector/b01r2n13/locks

~/.MATLABConnector/b02r2n01/LatestInstall.info

~/.MATLABConnector/b02r2n01/UpdatePending.json

~/.MATLABConnector/b02r2n01/locks

...

Questions

1. Is this a known issue with MATLAB Connector on HPC systems where $HOME is shared across many nodes?
2. How to prevent MATLAB from throwing this error at startup?

Answer 1

[Umar]

Hi @Lang,

So I did a deep dive on that MATLAB Connector error you're seeing on your SLURM cluster, and yeah, this is definitely a known issue that's been causing headaches for HPC folks running R2024a.

Here's what's happening: MATLAB is creating separate MathWorks Service Host and Connector installations for every single node your jobs land on. So you're ending up with all those directories like ~/.MATLABConnector/b01r2n13/, ~/.MATLABConnector/b02r2n01/, etc. When MATLAB starts up, it looks for which Connector version to use, finds multiple records from all these different nodes, and throws that "More than one MATLAB Connector records were found" error you're seeing.

The root problem is that earlier versions of R2024a weren't designed to handle shared home directories across multiple compute nodes very well. Every node hostname got its own installation, which is obviously not great when you've got dozens of nodes all writing to the same home directory.

Good news though - MathWorks actually fixed this in R2024a Update 6 and later. The newer updates install just one Service Host per user instead of per-hostname, and more importantly, when you use the -batch flag like you're doing, the Service Host shouldn't even be starting up at all. That's kind of the whole point of batch mode.

But here's the catch: if there's a file called license_info.xml sitting in your MATLAB installation's /licenses directory, MATLAB will try to use some newer licensing technology that requires starting the Service Host even in batch mode. You should check if that file exists at /work/share/.../matlab2024a/licenses/license_info.xml and if it does, talk to your HPC admin about removing it and using a traditional network.lic file instead.

For immediate relief, you can try cleaning out those directories:

rm -rf ~/.MATLABConnector
rm -rf ~/.MathWorks/ServiceHost

But honestly, that's just a band-aid. The real fix is getting your MATLAB installation upgraded to Update 6 or later, preferably Update 7 which is the most recent. Once you're on the newer update and that license_info.xml is gone, batch mode should work cleanly without any of this Service Host nonsense.

If your admin team wants more control over how the Service Host gets deployed, MathWorks has some admin scripts available at https://github.com/mathworks-ref-arch/administer-mathworks-service-host that let you do centralized management instead of per-user installations.

Hope this helps!

[lang]

Thanks for your help. The MATLAB version provided by the cluster is not the latest one and has been deployed to the user directory. By the way, could I use an older version (e.g., R2022a) to avoid this issue?

[Umar]

Hi @Lang,now that you have provided clarification, you should be able to use R2022a to avoid all Service Host complications on your SLURM cluster with shared home.