When using the MATLAB Job Scheduler, job information is stored on the worker nodes in the form various log files. These log files sometimes cause worker licenses to get stuck in the checked out state. This could result in a License Manager Error -4 which indicates no licenses are available for the workers because they are erroneously checked out. Since the log files are stored locally on the worker nodes, this issue can occur on all the worker nodes or only affect a few of the worker nodes.
In order to resolve the issue please follow the steps below on each of the worker nodes.
- Navigate to $MATLAB\toolbox\parallel\bin
- Run MJS stop -clean
- Run MJS start -clean
Once finished, stop and then start the License Manager service. This process will restart the MATLAB Parallel Server services on each of the worker nodes and clear out any left over log files as well as reset the currently checked out licenses. Once completed, reconfigure the job scheduler and start the workers using the Admin Center. Proceed with use of the cluster. You should no longer have issues with stuck licenses.
If you are still unable to resolve the issue by the steps above, contact the Installation and Licensing Team by creating a Service Request here:
NOTE: Starting in R2019a the following changes occurred:
- MATLAB Distributed Computing Server was renamed to MATLAB Parallel Server
- mdce_def was renamed to mjs_def
- mdce binary was renamed to mjs
- mjs scripts are in $MATLAB/R20XXx/toolbox/distcomp/bin for R2019a and earlier