rlFiniteSetSpec

Create discrete action or observation data specifications for reinforcement learning environments

Since R2019a

Description

An rlFiniteSetSpec object specifies discrete action or observation data specifications for reinforcement learning environments.

Creation

Syntax

spec = rlFiniteSetSpec(elements)

Description

spec = rlFiniteSetSpec(elements) creates a data specification with a discrete set of actions or observations, setting the Elements property.

Properties

Elements — Set of valid actions or observations

vector | cell array

Set of valid actions or observations for the environment, specified as one of the following:

• Vector — Specify valid numeric values for a single action or single observation.

• Cell array — Specify valid numeric value combinations when you have more than one action or observation. Each entry of the cell array must have the same dimensions.

Name — Name of the rlFiniteSetSpec object

string

Name of the rlFiniteSetSpec object, specified as a string. Use this property to set a meaningful name for the signal carried by this data channel. This property is used by the RL Agent block to match the bus signals with their corresponding environment channels.

Description — Description of the rlFiniteSetSpec object

string

Description of the rlFiniteSetSpec object, specified as a string. You can use this property to specify a meaningful description of the signal carried by this environment channel.

Dimension — Size of each element

vector (default)

This property is read-only.

Size of each element, specified as a vector.

If you specify Elements as a vector, then Dimension is [1 1]. Otherwise, if you specify a cell array, then Dimension indicates the size of the entries in Elements. This property is essential to construct agents and function approximators objects that work with a given environment.

DataType — Information about the type of data

"double" (default) | string

This property is read-only.

Information about the type of data, specified as a string, such as "double" or "single". The software uses this property to enforce data type consistency for observations and actions.

Object Functions

rlSimulinkEnv	Create reinforcement learning environment using dynamic model implemented in Simulink
rlFunctionEnv	Specify custom reinforcement learning environment dynamics using functions
rlValueFunction	Value function approximator object for reinforcement learning agents
rlQValueFunction	Q-Value function approximator object for reinforcement learning agents
rlVectorQValueFunction	Vector Q-value function approximator for reinforcement learning agents
rlContinuousDeterministicActor	Deterministic actor with a continuous action space for reinforcement learning agents
rlDiscreteCategoricalActor	Stochastic categorical actor with a discrete action space for reinforcement learning agents
rlContinuousGaussianActor	Stochastic Gaussian actor with a continuous action space for reinforcement learning agents

Examples

Create Reinforcement Learning Environment for Simulink Model

For this example, consider the rlSimplePendulumModel Simulink model. The model is a simple frictionless pendulum that initially hangs in a downward position.

Open the model.

mdl = "rlSimplePendulumModel";
open_system(mdl)

Create rlNumericSpec and rlFiniteSetSpec objects for the observation and action information, respectively.

The observation is a vector containing three signals: the sine, cosine, and time derivative of the angle.

obsInfo = rlNumericSpec([3 1]) 

obsInfo = 
  rlNumericSpec with properties:

     LowerLimit: -Inf
     UpperLimit: Inf
           Name: [0x0 string]
    Description: [0x0 string]
      Dimension: [3 1]
       DataType: "double"

The action is a scalar expressing the torque and can be one of three possible values, -2 Nm, 0 Nm and 2 Nm.

actInfo = rlFiniteSetSpec([-2 0 2])

actInfo = 
  rlFiniteSetSpec with properties:

       Elements: [3x1 double]
           Name: [0x0 string]
    Description: [0x0 string]
      Dimension: [1 1]
       DataType: "double"

You can use dot notation to assign property values for the rlNumericSpec and rlFiniteSetSpec objects.

obsInfo.Name = "observations";
actInfo.Name = "torque";

Assign the agent block path information, and create the reinforcement learning environment for the Simulink model using the information extracted in the previous steps.

agentBlk = mdl + "/RL Agent";
env = rlSimulinkEnv(mdl,agentBlk,obsInfo,actInfo)

env = 
SimulinkEnvWithAgent with properties:

           Model : rlSimplePendulumModel
      AgentBlock : rlSimplePendulumModel/RL Agent
        ResetFcn : []
  UseFastRestart : on

You can also include a reset function using dot notation. For this example, randomly initialize theta0 in the model workspace.

env.ResetFcn = @(in) setVariable(in,"theta0",randn,"Workspace",mdl)

env = 
SimulinkEnvWithAgent with properties:

           Model : rlSimplePendulumModel
      AgentBlock : rlSimplePendulumModel/RL Agent
        ResetFcn : @(in)setVariable(in,"theta0",randn,"Workspace",mdl)
  UseFastRestart : on

Specify Discrete Value Set for Multiple Actions

If the actor for your reinforcement learning agent has multiple outputs, each with a discrete action space, you can specify the possible discrete actions combinations using an rlFiniteSetSpec object.

Suppose that the valid values for a two-output system are [1 2] for the first output and [10 20 30] for the second output. Create a discrete action space specification for all possible input combinations.

actionSpec = rlFiniteSetSpec({[1 10],[1 20],[1 30],...
                              [2 10],[2 20],[2 30]})

actionSpec = 
  rlFiniteSetSpec with properties:

       Elements: {6x1 cell}
           Name: [0x0 string]
    Description: [0x0 string]
      Dimension: [1 2]
       DataType: "double"

Version History

Introduced in R2019a