Is there a way to visualize simulation result (after RL agent training) at the command line?

Hi!

Is there any way to visualize the result of a simulation at the command line, given a simulation result (as on the following picture for example)?

I mean, I can run the following lines to get the simulation result, but how could I then display it? How should I for example open the Reinforcement Learning Designer app at the command line and then 'plot' the simResults?

% Perform simulations
simResults = sim(env,agentPPO,simOptions);

Is there also a way of registering the "experience1" variable in the workspace?

Answer 1

[Neha]

Hi Nicolas,

I understand you're looking to export simulation results from the Reinforcement Learning Designer app to the workspace for visualization. Here's how you can do it:

1. After obtaining the results in the app, click on "Save Session" within the "Reinforcement Learning" tab. This action saves a .mat file in your workspace.
2. Access the results through the variable: "RLDesignerSession.Data.SimResults". This variable is a struct array containing all the experiments.
3. For instance, to access the reward timeseries data for the first simulation of the first experiment, use: "RLDesignerSession.Data.SimResults(1).Data(1).Reward".
4. To visualize the results, sum up the rewards for each simulation to obtain the cumulative reward. This data can then be used to plot a bar graph similar to the one in the Reinforcement Learning Designer App.

You can refer to the following documentation link for more information on the fields of the "RLDesignerSession.Data.SimResults" struct:

https://www.mathworks.com/help/reinforcement-learning/ref/rl.env.abstractenv.sim.html#:~:text=see%20rlSimulationOptions.-,Output%20Arguments,-collapse%20all

[Nicolas CRETIN]

Thank you so much Neha!

Here is the code I finally used if someone is interrested (and the graphical result, in the case where the agent has been very poorly trained):

% -------- perform training --------
trainingStats = train(agentPPO, env, trainOpts);
    

% -------- perform the simulation to double check the training --------

% Create simulation options
simOptions = rlSimulationOptions();
simOptions.MaxSteps = 10;
simOptions.NumSimulations = 8;

% Perform simulations
simResults = sim(env,agentPPO,simOptions);

% plot reward for these simulations
sizeSimResults = size(simResults);
cumulativeReward = zeros(sizeSimResults);
for i = 1:sizeSimResults(1)
    cumulativeReward(i) = sum(simResults(1).Reward.Data);
end
simIndex = 1:sizeSimResults(1);
meanReward = mean(cumulativeReward) * ones(sizeSimResults);

figure('Name', 'Simulation results');
bar(simIndex, cumulativeReward);
hold on
plot(simIndex, meanReward,'r--')
hold off
xlabel('Simulation index');
ylabel('Cumulative reward');
title('Cumulative rewards for each simulation');
legend('Cumulative Reward', 'Mean Reward');


% let's admit that simulation ii has a bad reward
% we want to display the output speed for this episode, to get to know what
% happened
ii = 1;
figure("Name", "Investigate bad simulation reward")
plot(simResults(ii).SimulationInfo.logsout{4}.Values.Time, ...
    simResults(ii).SimulationInfo.logsout{4}.Values.Data)
hold on
plot(simResults(ii).SimulationInfo.logsout{5}.Values.Time, ...
    simResults(ii).SimulationInfo.logsout{5}.Values.Data)
hold off
xlabel('time (seconds)');
ylabel('output speed (tr/min)');
title('Investigate bad simulation reward');
legend('reference', 'output speed');