Why is createJob / createTask so much slower than parfor?

Hi All,

I would like to have a better control of my parallel code than what parfor allows me to do, so I switched to createJob / createTask. However, for the same simple task, the overheads are now huge. This is my test code

clust = parcluster('local');
clust.NumWorkers = 12;
saveProfile(clust);

MyPool = parpool(clust);

x = linspace(0.1,1,100000);
TParfor = tic;
parfor i = 1:length(x)
    y(i) = sin(x(i));
end
fprintf('Elapsed time (parfor)=%g\n',toc(TParfor));
delete(MyPool);

numTasks = 12;
[xSplit, numTasks] = pctdemo_helper_split_vector(x, numTasks);

job = createJob(clust);
for i = 1:numTasks
   xThis = xSplit{i};
   createTask(job, @sin, 1, {xThis});
end
TCreateJob = tic;
submit(job);
wait(job);
fprintf('Elapsed time (createJob)=%g\n',toc(TCreateJob));
y = fetchOutputs(job);

And this is the output (the time displayed does not include opening pool, creating tasks, etc).

Starting parallel pool (parpool) using the 'local' profile ... connected to 12 workers.

Elapsed time (parfor)=0.824386

Parallel pool using the 'local' profile is shutting down.

Elapsed time (createJob)=24.8891

I tried several other test codes and applications, but createJob / createTask keeps being consistently slow. When I submit the job, it takes much longer than parfor to solve.

Any idea on what can be the cause? Why doesn't it take approximately the same time with the two solutions?

Thanks! Roberto

Answer 1

[Edric Ellis]

The thing to remember about your timings is that when using createJob and createTask with the local cluster type, each task runs in a separate MATLAB process. This takes much more time to launch than a parfor loop.

I suggest looking into parfeval, which has a similar interface to createTask, but uses the workers in a parallel pool, and so is much more efficient.