wait for several system calls to finish

I am calling three external programs and can only continue in the matlab program after all programs are finished. Each of them work independently of each other and can start at the same time

[staus1,~] = system('cmd1 &');
[staus2,~] = system('cmd2 &');
[staus3,~] = system('cmd3 &');

% how to check if all of them are finished?

Using ampersand & symbol, I reached that they start nearly at the same time, but how can I check when they are all finished? In my case, the returned status variables can be zero (successful) or negative (not successful). Hence, checking status for any value does not help, I guess.

Answer 1

[Walter Roberson]

You have to use taskmgr (Windows) or ps (MacOS or Linux) to detect whether the processes are still alive.

Note: If you are using Windows then I recommend that you instead use System.Diagnostics.Process which is able to start processes independently, and allows their status to be checked.

[SA-W]

I use Linux.

How would you use the ps command there? Running a while loop with true condition and breaking out once ps says all processes are done? Or can you think of a more elegant approach?

[Walter Roberson]

You could use

ps | grep matlab | tail

and parse out the process ID. Then you could start the new process, and then

ps -g PROCESS_ID_GOES_HERE

to get the process ID of the new process.

Then you could use the java watcher https://www.mathworks.com/matlabcentral/answers/398152-listening-to-the-creation-deletion-of-a-file-with-specific-name#answer_317920 on /proc/PROCESS_ID_OF_NEW_PROCESS watching for deletion

... something like that

[SA-W]

@Walter Roberson As for this solution: I found a way to extract the process id's of interest. Also, watchting /proc/PROCESS_ID for deletion is doable. However, it is not clear to me how I can extract the exit status of my program. When I do

[status,~] = system('myprog')

When I start the system commands in background (appending &), we have status=0, but this is not the real exit status of my program which is the integer returned in the main function of myprog.

Any idea how I can get this?

Answer 2

[Mario Malic]

Ampersand means that program will be started and MATLAB will not wait for it to be terminated and will continue to execute next lines of code.

What does it mean that they are finished? Does the program exits when it's done?

You can remove ampersand, then you have to close the program for MATLAB to execute next lines.

[SA-W]

Does the program exits when it's done?

Yes, it exists with possible non-zero exit status as I said.

You can remove ampersand, then you have to close the program for MATLAB to execute next lines

Yes, this way matlab waits until program 1 exits, then starts program 2 and wait until program 2 exits, and finally starts and waits for program3 to exit.

However, I should have said this, I want to start all of them at the same time (hence the ampersand) because each of the three calls takes a lot of time but can be worked off independently of each other.

[Mario Malic]

It is not the most convenient, but you could use a parallel processing toolbox (if available) to run them in parallel. However, you would need to obtain information when the process is finished, through ps, as Walter mentioned. You would have to also have to relay that information to the main MATLAB client so it knows when to continue with execution.

And yes, you can use while loop to see if process with that particular ID exists with while loop, as you said.

[Walter Roberson]

Note: I am not certain that system() can be executed within a background pool; you might need the full Parallel Computing Toolbox

[SA-W]

@Walter Roberson system() can be executed in a parfor loop which gives perfect synchronization for free. I just tried that. In particular, the end of the parfor loop is reached after all system() commands are exited (or terminated), exactly what I wanted.

Unfortunatly, I cannot rely on this solution because I already have an outer parfor loop which I can not control (multistart optimization in parallel). Hence, the parfor loop in my objective function translates into a for loop and I gained nothing.

[Mario Malic]

I don't know if the multistart in parallel is a good idea (I don't have experience with it).

Can you bruteforce your optimization? Generate all combinations for the optimization variable values and just run it. Then hit a local (or global) optimization algorithm from the best objective value.

Otherwise, I would suggest a surrogateopt if your program takes some time to run , and it's really good, compared to others, of course, it depends on the objective function.

[SA-W]

Basically, parallel multistart runs different start vectors in a parfor loop on a local solver such as fmincon or lsqnonlin. Testing different start vectors is an approach I want to stick to. I could code this loop myself, but I would probably also do it in a parfor loop to achieve running the start vectors in parallel. Running several experiments in parallel (which is what I am doing here with the system calls) is then no longer possible with a parfor loop.

Thanks for pointing out other optimization routines.

But coming back to my question, do you have any idea to query the exit status once my process has hinished (see commen under @Walter Roberson answer)? By exit status, I mean what the main function of my c++ program returns.

[Walter Roberson]

I cannot think of any way to get the information about the process exit status.

[Mario Malic]

If I understood correctly, multistart does that already, it has multiple starting points. By running multistart in parallel, each of these runs may lead to the minimum that already has been found by some other multistart process.

If you are looking for a any (tedious) solution, you could write a file from your c++ code indicating the exit status and read it with MATLAB once process is deleted.