Set computational model for policy or value function representation
Create an environment with a continuous action space and obtain its observation and action specifications. For this example, load the environment used in the example Train DDPG Agent to Control Double Integrator System.
Load the predefined environment.
env = rlPredefinedEnv("DoubleIntegrator-Continuous")
env = DoubleIntegratorContinuousAction with properties: Gain: 1 Ts: 0.1000 MaxDistance: 5 GoalThreshold: 0.0100 Q: [2x2 double] R: 0.0100 MaxForce: Inf State: [2x1 double]
Obtain observation and action specifications.
obsInfo = getObservationInfo(env); actInfo = getActionInfo(env);
Create a PPO agent from the environment observation and action specifications.
agent = rlPPOAgent(obsInfo,actInfo);
To modify the deep neural networks within a reinforcement learning agent, you must first extract the actor and critic representations.
actor = getActor(agent); critic = getCritic(agent);
Extract the deep neural networks from both the actor and critic representations.
actorNet = getModel(actor); criticNet = getModel(critic);
To view a network, use the
plot function. For example, view the actor network.
You can modify the actor and critic networks and save them back to the agent. To modify the networks, you can use the Deep Network Designer app. To open the app for each network, use the following commands.
In Deep Network Designer, modify the networks. For example, you can add additional layers to your network. When you modify the networks, do not change the input and output layers of the networks returned by
getModel. For more information on building networks, see Build Networks with Deep Network Designer.
To export the modified network structures to the MATLAB® workspace, generate code for creating the new networks and run this code from the command line. Do not use the exporting option in Deep Network Designer. For an example that shows how to generate and run code, see Create Agent Using Deep Network Designer and Train Using Image Observations.
For this example, the code for creating the modified actor and critic networks is in
Each of the modified networks includes an additional
reluLayer in their output path. View the modified actor network.
After exporting the networks, insert the networks into the actor and critic representations.
actor = setModel(actor,modifiedActorNet); critic = setModel(critic,modifiedCriticNet);
Finally, insert the modified actor and critic representations in the actor and critic objects.
agent = setActor(agent,actor); agent = setCritic(agent,critic);
oldRep— Policy or value function representation
Policy or value function representation, specified as one of the following:
rlValueRepresentation object — Value function representation
rlQValueRepresentation object — Q-value function representation
rlDeterministicActorRepresentation object — Actor representation with
rlStochasticActorRepresentation object — Actor representation with
To create a policy or value function representation, use one of the following methods.
model— Computational model
rlTableobject | 1-by-2 cell array
Computational model, specified as one of the following:
Deep neural network defined as an array of
Layer objects, a
layerGraph object, a
DAGNetwork object, or a
dlnetwork object. The input and output layers of
model must have the same names and dimensions as the network
getModel for the same representation. Here, the
output layer is the layer immediately before the output loss layer.
rlTable object with the same dimensions as the table model
1-by-2 cell array that contains the function handle for a custom basis function and the basis function parameters.
When specifying a new model, you must use the same type of model as the one already
For agents with more than one critic, such as TD3 and SAC agents, you must call
setModel for each critic representation individually, rather
setModel for the array of returned by
critics = getCritic(myTD3Agent); % Modify critic networks. critics(1) = setModel(critics(1),criticNet1); critics(2) = setModel(critics(2),criticNet2); myTD3Agent = setCritic(myTD3Agent,critics);
newRep— New policy or value function representation
New policy or value function representation, returned as a representation object of
the same type as
oldRep. Apart from the new computational model,
newRep is the same as