How to efficiently use spmd to load multiple files and perform execution?

I have the following function to load and process files in parallel. I was expecting the number of cores used to 100% during run time should be limited by "maxNumAllowedByMem" argument. However, I was pleasantly surprised to see 100% on all available cores on my computer. Why is that? I thought I have only provided data to the first (maxNumAllowedByMem)th worker data?

function output = spmd_process_files(fileNameInCells, maxNumAllowedByMem)
  spmd
      if labindex<=maxNumAllowedByMem
          outputBuffer = load(fileNameInCells{labindex});
          outputBuffer = aLongProcessFunction(outputBuffer);
      end
  end
  output = outputBuffer{1};
  for nthComp = 2:maxNumAllowedByMem
      output = someCombineFunction(output, outputBuffer{nthComp});
  end
end

Answer 1

[Edric Ellis]

Under some circumstances, the "idle" workers can consume CPU resources while waiting for the other workers to get to the end of the spmd block. You could try using the optional number of workers argument to the spmd block:

spmd (maxNumAllowedByMem)
  outputBuffer = ...;
end

[Ethan]

"idle" workers consuming 100% of the CPU resources for the whole process? Why?

Also, why is optional number of workers better in this case?

Answer 2

[Ben Gislason]

Does anyone know of any good examples that show step by step of how to use spmd to load and process multiple data sets? Ethan shows the overall structure to do this but I am running into difficulty setting up the load data and run function aspects. Any help would be greatly appreciated...