How to TRAIN further a previously trained agent?

188 views (last 30 days)
Hi,
My agent was programmed to stop after reaching an average reward of X. How do I load and extend the training further?
I did enable saving of the experiences and it has created the agent file
Rajesh

Accepted Answer

Rajesh Siraskar
Rajesh Siraskar on 11 Dec 2019
Hi Sourav, I figured it out after reading the documentation moer carefully!
I need to also set the ResetExperienceBufferBeforeTraining flag if I need to use previously saved experiences
This is my working code snippet. I must say this is a great feature and I really missed knowing about it!
USE_PRE_TRAINED_MODEL = true; % Set to true, to use pre-trained
% Set agent option parameter:
agentOpts.ResetExperienceBufferBeforeTraining = not(USE_PRE_TRAINED_MODEL);
if USE_PRE_TRAINED_MODEL
% Load experiences from pre-trained agent
sprintf('- Continue training pre-trained model: %s', PRE_TRAINED_MODEL_FILE);
load(PRE_TRAINED_MODEL_FILE,'saved_agent');
agent = saved_agent;
else
% Create a fresh new agent
agent = rlDDPGAgent(actor, critic, agentOpts);
end
% Train the agent
trainingStats = train(agent, env, trainOpts);
  3 Comments
Anh Tran
Anh Tran on 21 Feb 2020
Rajesh is correct. Currently the noise model resets when you train again. We are looking into how you can truly 'resume' training. As a workaround, you can set the noise variance option to a lower value than that of your previous train session.

Sign in to comment.

More Answers (2)

Anh Tran
Anh Tran on 21 Feb 2020
I will answer again, hopefully clear your confusion.
% Train the agent
trainingStats = train(agent, env, trainOpts);
After this line, even though the 'agent' is not returned as an output, its learnable parameters are updated. Learnable parameters, e.g. the weights and biases of the actor/critic neural networks, determines the logic behind the agent (and how it chooses action given an observation).
Now if you execute sim() or train() after this line, the 'agent' will simulate or continue training with the latest parameters.
Rajesh's workflow is very close to resume training (reuse the experiences gathered in the past, start from latest parameters). I revised the code with additional comments. Currently the noise model resets when you train again. You can consider setting the noise variance option to a lower value (still need to be > 0 because we want the agent to always explore) than that of your previous train session.
% Set to true, to resume training from a saved agent
resumeTraining = true;
% Set ResetExperienceBufferBeforeTraining to false to keep experience from the previous session
agentOpts.ResetExperienceBufferBeforeTraining = ~(resumeTraining);
if resumeTraining
% Load the agent from the previous session
sprintf('- Resume training of: %s', PRE_TRAINED_MODEL_FILE);
load(PRE_TRAINED_MODEL_FILE,'saved_agent');
agent = saved_agent;
else
% Create a fresh new agent
agent = rlDDPGAgent(actor, critic, agentOpts);
end
% Train the agent
trainingStats = train(agent, env, trainOpts);
  2 Comments
Sayak Mukherjee
Sayak Mukherjee on 23 Feb 2021
for restarting the run with saved agent, the saved agent shaould have 'SaveExperienceBufferWithAgent' parameter set to true, right?

Sign in to comment.


Sourav Bairagya
Sourav Bairagya on 10 Dec 2019
In this case, you can resume your training with the previous experience buffer as a starting point.
You have to set the 'SaveExperienceBufferWithAgent' agent option to 'true'.
For some agents, such as those with large experience buffers and image-based observations, the memory required for saving their experience buffer is large. In these cases, you must ensure that there is enough memory available for the saved agents.
For more informations you can leverage this link:
  2 Comments
mr robot
mr robot on 30 Jan 2020
How large is "large" for an experience buffer of 1e6?

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!