Avoid serializing and unserializing when calling parfor/spmd => can I pass object by reference between matlab workers?

Hello,

I am struggling yet to have mi big data not serialized by the main worker and unserialized by parallel worker when i am entering in the parfor/spmd context...

The serialization/unserialization operation time take 10 seconds for a cell array of 2021 complex Matlab object (and thus when this data number overflow the amount of 200 000 000 (which is my case), this operation will take 100 000 seconds?) ...

Does anyone have maybe a method/process (are a "best practice") which allows communication between local worker only by object reference or/and speed up this problematic performance funnel? (I think the solution maybe obvious,because parallel computation is interessant for a large amount of data... but the time which is requested for that serialization/unserialization operation between workers is unbearable and make the Matlab the parfor/spmd solution seems irelevant ...)

Hope someone will help me

Answer 1

[Edric Ellis]

One option might be to use parallel.pool.Constant to store your large cell array. This can be of benefit if you're using the same large data across multiple parfor loops or spmd blocks, and also it allows you to avoid the transfer by building the large array directly on the workers. Here's how it might work:

% build a 1x100 cell array on the workers, where each cell contains
% a magic square array
c = parallel.pool.Constant(@() arrayfun(@magic, 1:100, 'UniformOutput', false));
out = cell(1, 10);
for idx = 1:numel(out)
  % Use the magic squares inside a parfor loop. 
  parfor jdx = 1:100
    v = c.Value;
    ranks(jdx) = rank(v{jdx});
  end
  out{idx} = ranks;
end