parfor works but parcluster fails using Cellpose in linux

Hello all,

I am trying to run parallel jobs in Matlab in linux system. I used the Matlab Add-on Cellpose, which needs Matlab versoion later than 2023b.

parfor i=1:4
    img = imread('calibrated-P01.001.tif');
    cp=cellpose;
   segmentCells2D(cp,img,ImageCellDiameter=110);
end

The parfor using Cellpose works.

But when I try to use parcluster/createJob/createTask, it falls. Here is the code

clustLocal = parcluster('Processes')
j = createJob(clustLocal,'AutoAddClientPath',true)
createTask(j,@TestCellpose,0,{},'CaptureDiary',true)
submit(j)


function TestCellpose
    
    img = imread('calibrated-P01.001.tif');
    cp=cellpose;
    segmentCells2D(cp,img,ImageCellDiameter=110);
end

The error message

Error: Unable to resolve the name 'py.MWCellposeWrapper.setModelsFolderPath'

I prefer using parcluster because it can run the code in backgound.

Does anyone know how to solve the problem?

Thanks a lot!

[Venkat Siddarth Reddy]

Hi Yunwei,

Can you share your OS details(like version,distribution etc.)?

[Yunwei]

Hi Venkat,

Thanks for your reply. Below are the OS detalis.

No LSB modules are available.

Distributor ID: Ubuntu

Description: Ubuntu 20.04.6 LTS

Release: 20.04

Codename: focal

NAME="Ubuntu"

VERSION="20.04.6 LTS (Focal Fossa)"

ID=ubuntu

ID_LIKE=debian

PRETTY_NAME="Ubuntu 20.04.6 LTS"

VERSION_ID="20.04"

HOME_URL="https://www.ubuntu.com/"

SUPPORT_URL="https://help.ubuntu.com/"

BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"

PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"

VERSION_CODENAME=focal

UBUNTU_CODENAME=focal

[Venkat Siddarth Reddy]

It seems the OS is offically supported by MATLAB R2023b( https://www.mathworks.com/content/dam/mathworks/mathworks-dot-com/support/sysreq/files/system-requirements-release-2023b-linux.pdf ). I am suspecting there might be an issue in the gateway between python and MATLAB.

Can you run the following code snippet and share the output of it?

pyenv()
string(py.sys.path)'
getenv('PATH')
mwpylib = py.importlib.import_module('MWCellposeWrapper') ;

[Yunwei]

Hi,

the output is attached. What pizzles me is in parfor the Cellpose works but not in parcluster.

I wrote path = getenv('PATH').

Answer 1

[Swastik]

Hello @Yunwei,

It appears you have encountered an issue when utilizing parcluster in conjunction with the Cellpose Library, a problem that was not present when using parfor. Your preference for parcluster stems from its ability to submit tasks that can run in the background.

I recommend exploring the batch function in MATLAB. The batch function is designed to execute commands or scripts in the background, like parcluster, but it specifically spawns parfor loops for its execution. This means you can achieve the same level of background processing that parcluster offers, potentially without the error you encountered.

To learn more about the batch function, please visit the following MATLAB Documentation page:

https://www.mathworks.com/help/parallel-computing/run-a-batch-job.html#responsive_offcanvas

Hope this will solve your issue.