PARFOR is 10X Slower than FOR

I'm trying to understand why my use of the parfor is so much slower than using a for loop.

Both parfor and for implementations require some IO, namely the reading of a png file that is 20 Kb (buildingDesign.png attached).

The code called (getSpacing_HELP.m) takes this png file, converts it into a BW matrix and does several image processing operations on the matrix. The purpose is to calculate the average spacing between objects in an image.

The code shown here is a pared-down, simplified version that only includes the meat of the code. I'm using 36 workers on a Linux machine. Images below show the histogram of times for 1000 calls to getSpacing_HELP.m.

Why would the parfor implementation be so much slower? Any suggestions on ways to speed it up? In actual use, the spacing routine could be called a ~10^8 times, so this time difference is important.

ntot = 1000; % How many runs to test

% Create a directory to place copies of the file in. This is just to
% simulate the actual use of this code for troubleshooting.
mkdir tmp1
for jj = 1:ntot
    theName = ['./tmp1/',num2str(jj),'_clipped.png'];
    copyfile('./Files/buildingDesign.png',theName);
end
%

resultsMat = zeros([ntot 1]);
timeMat = zeros([ntot 1]);

tic
%parfor k = 1:ntot
for k = 1:ntot

 fileName = ['./tmp1/',num2str(k),'_clipped.png'];

 [singleResult,theTime] = getSpace_HELP(fileName);

 resultsMat(k) = singleResult;
 timeMat(k) = theTime;

end
histogram(timeMat), xlabel('Time (s)')
timeAll = toc;
disp(['Time to do all runs: ',num2str(timeAll)])

[Carson Purnell]

You're re-using every variable within the loop, so this is not a valid test. Clear everything at the end of the loop or use unique variables at every loop iteration to fairly compare. for has optimization that parfor cannot make use of for cases like this.

[Stephen23]

"You're re-using every variable within the loop"

Which is the normal, recommended, best way to write MATLAB code. This allows the JIT-engine to work efficiently.

"Clear everything at the end of the loop or use unique variables at every loop iteration .."

DO NOT do that unless you want to pointlessly slow down all of your code for no benefit. Ignore this bad advice.

"Why would the parfor implementation be so much slower?"

Why do you think that parfor should be faster than for for your problem? It is not the case that parfor is always faster. Have you read the MATLAB documentation on this?:

https://www.mathworks.com/help/distcomp/decide-when-to-use-parfor.html

The only time it is worth using parfor is when the time saved running the operations inside the loop is greater than that of the extra time for the parfor overhead. This is computing, and nothing comes for free (something that beginners seem to forget when they expect infinitely fast computers with infinite memory, infinite-precision numerics, and overhead operations that take no time at all. Sadly reality does not meet these expectations).

Running a parfor is a much more complicated thing than running a simple for loop because the data needs to be partitioned and the results grouped again afterwards, so it is entirely possible that a parfor might be slower. This is clearly explained in the documentation: "There is overhead in calling parfor instead of for. If function evaluations are fast, this overhead could become appreciable. In particular, solving a problem in parallel can be slower than solving the problem serially."

source: https://www.mathworks.com/help/optim/ug/improving-performance-with-parallel-computing.html

So far you have not provided any actual justifcation, why you think parfor should be faster.

[Carson Purnell]

My comment is specifically to point out why parfor is slower in this case. The efficient implementation of a standard for loop lacking the overhead of the parfor loop is what OP is missing, and this will show at least part of it. It will demonstrate how the parfor loop is slowing down the code for no benefit.

[Paul Safier]

@Stephen23 I didn't expect a single interation in the parfor loop to be necessarily faster than a single iteration in the for loop, but I didn't expect a 10X slowdown. The reason I didn't expect a slowdown was because when I profiled an iteration, the lion's share of the time was with the image processing functions--I saw no evidence that the IO would be a rate-limiting step. At least that's how I interpreted the profile output.

[Edric Ellis]

Aha. I suspect that bwlookup can take advantage of MATLAB's intrinsic multi-threading. If that is the case, then your multithreaded desktop MATLAB process is already taking full advantage of all the cores on your system. The workers in a parallel pool run in single-threaded mode (by default). You can confirm this by either monitoring the processor utilisation of desktop MATLAB using top (or similar); or, you can force your desktop MATLAB into single-threaded mode for comparison purposes by using maxNumCompThreads(1).

Basically, any time your original for-loop code is dominated by stuff that is already multithreaded by MATLAB itself, there is no advantage to using a local parallel pool. You're already fully utilising your machine. In cases like this, you may see benefit from using parfor with multiple remote workers.

[Paul Safier]

Hi @Edric Ellis. I ran some tests using top and it seems like that's what's happening. The default maxNumCompThreads on my machine is 36.

Here is top when MATLAB is idle, i.e. I have not started the clipSpacing_HELP.m code. Looks like there're 151 threads available with just 1 running. Not sure why it's necessarily 151...

Here is top when running the code (with the for loop) with the default thread count of 36. It shows 37 threads are running with 113 sleeping.

Here is top when running the code (with the for loop) after setting numMaxCompThreads(1). It doesn't employ as many threads, but not identical to 1. Not sure why...

Here is top when running the code with the parfor loop. The total thread count goes up, but the running thread count is 1. I believe this is what you stated would happen.

So, I guess this test confirms what you suspected, namely that running the for loop is utilzing multithreading, whereas the parfor loop is not. What's confusing me now, is that a timing comparison of running the for loop with 36 vs 1 thread actually shows that the single thread way is a bit faster (overall for 1000 iterations) and for each iteration, it's about the same time as the multithreading result. If this is true, how can it explain the original issue which is that each iteration with a for loop is 10X faster than each iteration with a parfor loop?

Timing comparison of running the for loop with 36 vs 1 thread is below. Shouldn't the individual iteration time for the single thread test be much longer than for the 36 thread case if it's to explain the 10X delta in iteration times per the original post?

Answer 1

[Raymond Norris]

@Paul Safier I believe the problem is that you're calling nested tic/toc. In clipSpacing_HELP, you call tic on line 16 and toc on line 29. However, in the for-loop you call getSpace_HELP, which calls tic on line 6. This becomes the new start time for the call to toc on line 29 in clipSpacing_HELP.

Conversely, when you call getSpace_HELP in a parfor, since the call to tic happens in another worker, the call to toc in clipSpacing_HELP isn't aware of it, so it still uses the tic on line 16 (which is what you really want the for-loop to do as well).

The solution is to link the tic/toc together with a variable, as such (I'm using t0).

t0 = tic;
%parfor k = 1:ntot
for k = 1:ntot
    fileName = ['./tmp1/',num2str(k),'_clipped.png'];
    [singleResult,theTime] = getSpace_HELP(fileName);
    resultsMat(k) = singleResult;
    timeMat(k) = theTime;
end
histogram(timeMat), xlabel('Time (s)')
timeAll = toc(t0);

This way, the call to tic/toc in getSpace_HELP doesn't reset the toc being assigned to timeAll. Make this change and rerun it to see if that gives a more accurate run.

[Paul Safier]

Hi @Raymond Norris . Good catch. Thanks for seeing that. It seems that error messed up the final time estimate. The time of each call of the getSpace_HELP function was correct (see image). However, fixing the error, as you suggested, revealed that overall, the parfor solution for 1000 iterations takes about 3.8X less time than the serial way (using 36 workers), despite the fact that each individual call to the getSpace_HELP function takes 10X longer with the parfor loop because of the overhead.

I guess the speed up might be ~ (# of workers)/10?

[Paul Safier]

@Raymond Norris and @Steven Lord Is the 10X individual iteration slowdown because of the parfor overhead an insurmountable hit? It's just hard to swallow that conclusion since the workers don't have to communicate with each other and it would seem they only have to do what a serial worker has to do, and can do in a tenth of the time...

Answer 2

[Steven Lord]

Have you tried using the parallel profiler to determine what percentage of the time taken by the parfor code is spent on the actual computations and how much on overhead? You could try comparing comparing those parallel profiling results with the results of running the for loop version of the code in the MATLAB Profiler.

In order for your code to gain time when run in parallel, the amount of time you spend in the parallel setup and other overhead must be less than the amount of time that you save by running the iterations in parallel. If your overhead is high and/or the amount of time you save is small, your parfor loop could very well take more time to run than your for loop.

Think of grocery shopping with kids. If sending them over to the cereal aisle saves you two minutes but you have to spend five minutes searching the store to find them afterwards (eventually finding them in the candy or snacks aisle) you would be better off just going to the cereal aisle yourself.

[Paul Safier]

@Steven Lord I didn't know there was a parallel profiler. Thanks! I used the serial profiler (see comment above to Stefen23). The results of the serial profiler is what motivated my inquiry into parallel processing.

I ran the parallel profiler now and the results are below. A lot of time is spent in the remoteParallelFunction, which I guess is the 'overhead' that's being referred to.

I'm surprised that the parfor has so much extra overhead considering the workers don't need to communicate with (or wait for) each other.

So, is that all there is to it? There're no ways to optimize or adjust any parameters to improve things?

[Torsten]

Just out of curiosity:

What if you remove the settings

resultsMat(k) = singleResult;
timeMat(k) = theTime;

in the parfor loop ?

[Paul Safier]

@Torsten I just did now and profiled it. No noticable change in overall time or the proportion that's in overhead (i.e. remoteParallelFunction)...