Memory issues using parfeval

7 views (last 30 days)
Andre Zeug
Andre Zeug on 22 Apr 2020
Answered: Swastik Sarkar on 16 Oct 2024
Hi everyone!
Since years I am using the nice function export_fig to save figures, which I create with my scripts and functions. To speed this up I tried to do this in background using 'parfeval'. Unfortunately, for some reason workers pile up memory consumption which can be monitored e.g. with the ProcessExplorer.
function export_figP(fname, varargin)
% same input as export_fig, but first input must be the filename, last
% input must be the figure handle (could be improved)
tParFeval = tic;
tt = []; fig_1 = [];
set(varargin{end},'Visible','off')
myEvalStr = 'tt = parfeval(@export_fig, 0, fname';
for n=1:nargin-2
myEvalStr = sprintf('%s, ''%s''',myEvalStr, varargin{n});
end
myEvalStr = sprintf('%s, fig_1);',myEvalStr);
n = nargin-1;
eval(sprintf('fig_1 = varargin{%d};',n))
eval(myEvalStr)
afterEach(tt, @(~)evalAfterAll(tt,fig_1,fname,tParFeval), 0);
function evalAfterAll(tt,hh,saveStr,tParFeval)
if ~isempty(tt.Error)
fprintf(2,'!!! Error saving Figure %s !!! \n',saveStr)
else
fprintf(2,'Figure ''%s'' saved within %.3fs (total time %.3fs)\n',saveStr,(datenum(tt.FinishDateTime)-datenum(tt.CreateDateTime))*24*60*60, toc(tParFeval))
close(hh)
tt = [];
clear all
end
end
end
The function can be tested with this lines (requires export_fig)
%% Test export_figP
if ~isfolder(fullfile(pwd, 'figs')); mkdir('figs');end
tic
for n = 1:1000
% create simple figure
fig1 = figure('Color','w','Visible','on');
imagesc(rand(1000, 1000))
colorbar
% save figure in Background
saveStr = sprintf('%04d_TestFig.png',n);
export_figP(sprintf('figs/%s',saveStr),'-q100','-nocrop','-r300',fig1);
fprintf('saving of %s started (%.3f)\n', saveStr, toc)
end
What do I do wrong?
How to clear the workers memory after the function call (except shuting down parallel pool)?
Probably my function 'export_figP' does something substantially wrong.
  1 Comment
Andre Zeug
Andre Zeug on 1 May 2020
Is there an option to clear workers memory when no further job is processed?
p = gcp('nocreate')
p =
ProcessPool with properties:
Connected: true
NumWorkers: 16
Cluster: local
AttachedFiles: {}
AutoAddClientPath: true
IdleTimeout: Inf (no automatic shut down)
SpmdEnabled: true
q = p.FevalQueue
q =
FevalQueue with properties:
Number Queued: 0
Number Running: 0
But memory consumtion is still big after several calls of parfeval:

Sign in to comment.

Answers (1)

Swastik Sarkar
Swastik Sarkar on 16 Oct 2024
The export_figP and evalAfterAll functions can be enhanced for readability and effective management of figures throughout their lifecycle. Below is the improved implementation:
function export_figP(fname, varargin)
figHandle = varargin{end};
set(figHandle, 'Visible', 'off');
tt = parfeval(@export_fig, 0, fname, varargin{1:end-1}, figHandle);
afterEach(tt, @(~) evalAfterAll(tt, figHandle, fname), 0);
end
function evalAfterAll(tt, figHandle, saveStr)
if ~isempty(tt.Error)
fprintf(2, '!!! Error saving Figure %s !!! \n', saveStr);
else
fprintf('Figure ''%s'' saved successfully.\n', saveStr);
end
% Close the figure and clear the task
close(figHandle);
delete(tt);
end
Clearing the output data of the parallel.FevalFuture object can further enhance performance. More information can be found in the following resource:
Hope this helps.

Categories

Find more on Text Data Preparation in Help Center and File Exchange

Products


Release

R2020a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!