MATLAB Answers

Identifying lab id within a parfor

101 views (last 30 days)
Javier
Javier on 25 Feb 2012
Commented: HiWave on 25 Sep 2020
Hello all,
I have an problem that I would like to run in parallel using a parfor. Within the parfor I have to call an external program that runs some calculations using system('external_command result_file_name'). This external program saves the result into a text file that I have to read after execution.
In order for the different iterations not overwrite the results of each other, I would like to know what is the labid running each iteration such that I can compose the file name such as file_name_labid.
I have tried the function labindex but it always returns 1 within a parfor. Is there any other way I can get the labid?
Many thanks,

  0 Comments

Sign in to comment.

Accepted Answer

Edric Ellis
Edric Ellis on 27 Feb 2012
As mentioned elsewhere, LABINDEX will not work for this purpose. You could either use
t = getCurrentTask(); t.ID
to find out which task you're running. Each worker has a unique task ID which remains the same for the duration of the MATLABPOOL.
Or, my personal preferred solution would be to use TEMPNAME to generate a unique filename to write the output to.

  3 Comments

Jason Xu
Jason Xu on 3 Jun 2019
Hi Edric:
im using parallel GA to optmize my design. In my objective function im using:
b = get(getCurrentTask(),'ID');
display(b)
to obtain which worker is running my objective function, but sometimes i got empty value for b displayed as followed (i got 4 workers so b value should be expected as 1 2 3 4, not strictly follow this sequence):
% In one generation, assuming population is 8
...
b = []
b = 1
b = 2
b = 3
b = 4
b = 3
b = 1
b = 2
b = 4
...
Can u help me to solve this issue? Thank you
Damdae
Damdae on 23 Jul 2019
It would be better if labindex returned the value of get(getCurrentTask(),'ID') when it's in a parfor loop.
HiWave
HiWave on 25 Sep 2020
I'm facing the same problem...how did you solve this?

Sign in to comment.

More Answers (6)

jiro
jiro on 26 Feb 2012
Why not save the file based on the iteration number? Wouldn't you still overwrite the results if you were using the labindex because different iterations could be running on the same worker.
As Oleg mentioned, if you want something to happen based on the labindex, use spmd instead. You would break up your loop into n groups, where n is the number of workers. Something like this:
spmd
for id = (1:100)+(labindex-1)*100
% do some computation
save(sprintf('file_name_%d', labindex));
end
end
EDIT 1: Combine SPMD and PARFOR to write to different directories
spmd
mkdir(sprintf('worker%d', labindex));
cd(sprintf('worker%d', labindex));
end
parfor id = 1:1000
fid = fopen('myresults.txt', 'a+');
fprintf(fid, '%d\n', id);
fclose(fid);
end
spmd
cd ..
end
EDIT 2:
function [] = main_function()
function [y] = obj(x)
% write parameters x to an input file
system('external_program input_file output_file');
% read output file
y = read_output();
end
spmd
folder = sprintf('worker%d', labindex);
mkdir(folder);
cd(folder);
end
% now I can use a parfor
result = [];
c = linspace(10,100,10);
parfor k=1:10
result(k) = obj( c(k) );
end
plot(c,result);
% or I can call fmincon
problem.x0 = x0;
problem.objective = @obj;
[x,fval] = fmincon(problem);
spmd
cd ..
end
end
Creating folders and cd-ing to the appropriate folder (using spmd) ensures that each worker resides in different locations. Then file writing and reading will happen in different locations.

  0 Comments

Sign in to comment.


Oleg Komarov
Oleg Komarov on 25 Feb 2012
Trying a simple parfor never uses more than 1 worker:
matlabpool(2)
parfor ii = 1:10000
if numlabs > 1
disp(labindex)
end
end
matlabpool close
But the following snippet forces the execution of labindex on all workers:
matlabpool(2)
spmd
labindex
end
matlabpool close
Somebody who can enlighten us about this behaviour?

  1 Comment

Edric Ellis
Edric Ellis on 27 Feb 2012
LABINDEX always returns the value 1 when not inside an SPMD block (or the body of a parallel job). You can check e.g. "t = getCurrentTask(); t.ID" to see that different tasks work on different PARFOR loop iterations. LABINDEX and NUMLABS are designed to show you with whom you can communicate using LABSEND and LABRECEIVE. Inside a PARFOR loop, that's no-one.

Sign in to comment.


Javier
Javier on 25 Feb 2012
Look at this other thread:
It seems that since communication is not allowed in parfor loops, numlabs and labindex are set to 1 in each worker process.
So this seems to be the expected behavior, but it does not solve my problem.
Any idea on how to identify each worker process?

  0 Comments

Sign in to comment.


Javier
Javier on 26 Feb 2012
Thanks for the suggestion.
Indeed I could use the iteration number in the for-loop, but the reason I wanted to do it with the labindex is that I expect to use the same code later with a optimization function such as fmincon. From what I tried, it seems that fmincon behaves like parfor in this regard.
In this case, I won't have access to any iteration number, and I cannot use spmd. I just call fmincon with the objective function being the function that calls the external program and reads the result, so the only way I can think of distinguish between the different workers is if I can have an identification number such as labindex.
I don't mind to overwrite the result after each iteration with the same worker, because once I read the result I save the parameters into a vector in matlab. I just don't want that a different worker overwrites the result before I read it.
Any other idea?

  1 Comment

jiro
jiro on 26 Feb 2012
With fmincon, you probably won't use parfor anyway, so I'm not sure I understand when you say you want to use the same code.
Another way I would do it is to use SPMD and PARFOR together. You can use SPMD to change the folder that each worker is in. Then you can write to a file, and the files will be unique for each worker because they're in different folders. See my edit to my answer.

Sign in to comment.


Javier
Javier on 26 Feb 2012
Maybe I didn't make myself clear, sorry. Somewhat simlified he code looks something like this. I think I don't miss anything important now
function [] = main_function()
function [y] = obj(x)
% write parameters x to an input file that will be passed to an external program
system('external_program input_file output_file');
% read output file
y = read_output();
end
% now I can use a parfor
result = [];
c = linspace(10,100,10);
parfor k=1:10
result(k) = obj( c(k) );
end
plot(c,result);
% or I can call fmincon
problem.x0 = x0;
problem.objective = @obj;
[x,fval] = fmincon(problem);
end
The code works fine when I use a regular for, or when I use fmincon with a single lab. I'm just worried that it can fail when using multiple labs.

  1 Comment

jiro
jiro on 26 Feb 2012
"spmd" would still work. See my edit (EDIT 2) to my answer.

Sign in to comment.


Javier
Javier on 1 Mar 2012
Thanks everyone for the replies.
Jiro Doke:
I did a simple test with your method, but it did not work. Maybe I did not understood you or something is missing. This is what I did:
function [] = test()
spmd
folder=sprintf('worker%d',labindex);
mkdir(folder);
cd(folder);
end
parfor k=1:10
disp(pwd)
end
spmd
cd ..
end
The problem is that once the worker changes the folder, they are not able to find the source code and throw an error.
Edric Ellis:
That's exactly what I needed! Thanks!

  0 Comments

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!