Correct fetchNext index after clearing FevalFuture object

I am using parfeval to run code asynchronously on workers. The running time varies a lot between workers depending on some parameters that in the following code I represent with 'm'.

When I collect the results with fetchNext, I then process them with idx, in particular retrieving m(idx). At the same time, I need to clear the last retrieved FevalFuture object to save memory.

In Parfeval - Memory consumption piling up - Clear output data? it is suggest to simply assign [ ] to the FevalFuture object. However, it seems to me that doing this mix up the indices idx of the objects, which I need to further process the data. See for instance this mve

N = 10;
m = 1:10;
for idx = N:-1:1
    f(idx) = parfeval(@(m,N) m*rand(N), 1, m(idx), N);
end
total = 0;
for idx = 1:N
    [idx, data] = fetchNext(f);
    idx
    f(idx) = []; % This line here
    
    % process data with idx, m(idx) ..
end

If you comment the line indicated the code will work fine, but if I clear f(idx) the just-used idx could be reused and I would process the data with the wrong m(idx).

How is it possible to get the correct index idx with which f(idx) was called in the fist loop?

Thanks in advantage

Answer 1

[Edric Ellis]

One way you can deal with this is by making the same modification to m as you proceed. Something like this perhaps:

N = 10;
m = 1:10;
for idx = N:-1:1
    f(idx) = parfeval(@(m,N) m*rand(N), 1, m(idx), N);
end
for idx = 1:N
    [fidx, data] = fetchNext(f);
    % Get the value out of "m" corresponding to the completed future
    mval = m(fidx);
    % Delete corresponding elements of both "f" and "m"
    f(fidx) = [];
    m(fidx) = [];
    % Work with the particular value of "m" and the result
    disp([mval, sum(data(:))]);
end

[Davy Figaro]

@Edric Ellis, that looks like a good solution. Another way to handle this that avoids modifying the index referencing is to assign new empty future object to the completed index:

f(fidx) = parallel.FevalFuture; 

This has worked well for me and solved my memory problems using parfeval, and is more convenient than having to worry about updating the indexing.