# Save inside parfor loop at a specific iteration step

24 views (last 30 days)
CalinV on 23 Feb 2015
Edited: CalinV on 24 Feb 2015
Good day, I'm having problems saving inside a parfor loop. What I want to do is to save the results (vector results) from 5000 -> 5000 iterations (parfor loop takes 1 week to complete and I don't want to lose all the temporary result in case that something bad happens with the machine, i.e. reboot, power off..) A simplified version of my code is:
parfor ii = 1:100000
[output1, output2] = MyFunction(~,~,ii)
% from 5000 to 5000 iterations I want to save the vector results
if ii == 1:5000:100000
parsave('results.mat',output1, output2)
% now I want to save the final results
elseif ii == 100000
parsave('results.mat', output1, output2)
end
end
Maybe someone can provide me with an useful parfor.m function, or a way to save it using matfile objects (I don't know how to use them)

Sean de Wolski on 23 Feb 2015
First, the if ii==vector will never return true because all values need to be true. You would need an any() around it; compare:
if 1 == 1:5,disp('hi'),end
if any(1 == 1:5),disp('hi'),end
Matt J on 23 Feb 2015
@CalinV
Saving every 5000th iteration will not help you because parfor loop iterations runs in parallel and in arbitrary order. When iteration #5000 gets executed, it does not imply that iterations 1-4999 have also been executed.

Matt J on 23 Feb 2015
Edited: Matt J on 23 Feb 2015
Since MyFunction takes as long as it does (at least 6 seconds), I might just save every iteration within a for-drange loop. It takes no time to save a 1e5x1 vector, and the local part on each worker will be smaller still.
output1=distributed.NaN(1,1e5); %pre-allocate
output2=distributed.NaN(1,1e5);
spmd
for i=drange(1:1e5)
[output1(ii), output2(ii)] = MyFunction(~,~,ii);
parsave(['TEMP', num2str(labindex)],...
getLocalPart(output1),getLocalPart(output2));
end
end

#### 1 Comment

CalinV on 24 Feb 2015
This solution is elegant and works great ! I have used it in this form:
parpool('local',10); % results will be distributed on 10 workers
output1=distributed.NaN(1,1e5); %pre-allocate
output2=distributed.NaN(1,1e5);
spmd
for i=drange(1:1e5)
[output1(ii), output2(ii)] = MyFunction(~,~,ii);
parsave(['output1', num2str(labindex)],...
getLocalPart(output1));
parsave(['output2', num2str(labindex)],...
getLocalPart(output2));
end
end
I've used 2 times the parsave function so in case that the machine shuts down before the simulation ends, 10 mat files will be saved with temporary results that I'll need to concatenate later (for output1 & output2)
The parsave function is:
function parsave(fname,data)
var_name=genvarname(inputname(2));
eval([var_name '=data'])
try
save(fname,var_name,'-append')
catch
save(fname,var_name)
end
% Written by Minjie Xu (<mailto:chokkyvista06@gmail.com chokkyvista06@gmail.com>)

Sean de Wolski on 23 Feb 2015
Edited: Sean de Wolski on 23 Feb 2015
First, the if ii==vector will never return true because all values need to be true. You would need an any() around it; compare:
if 1 == 1:5,disp('hi'),end
if any(1 == 1:5),disp('hi'),end
Second, since two workers might try to write at the same time, you'll need to save to different MAT files and concatenate after:
save(['Results' num2str(ii,'%05i') '.mat'],etc)
Other than that, looks like it should work...

Matt J on 23 Feb 2015
I can't see how the OP's initial proposal will work, or at least be beneficial. The iterations of a parfor loop are independent, so saving every 5000th iteration will discard everything else. You also don't know in what order the iterations of a parfor loop will be executed. For all one knows, the iterations 1:5000:100000 could be the final 20 that get executed at the end of the week. A power failure mid-week will lose everything.
Sean de Wolski on 23 Feb 2015
Maybe I didn't read closely enough - I thought he just wanted those iterations saved.
If he cares about order, use parfeval.