rlFiniteSetSpec

Create specifications object for a finite-set action or observation channel

expand all in page

Description

An rlFiniteSetSpec object contains specifications for a channel that carries an action or observation belonging to a finite set.

Creation

Syntax

spec = rlFiniteSetSpec(elements)
spec = rlFiniteSetSpec(elements,Name=Value)

Description

spec = rlFiniteSetSpec(elements) creates a data specification object for a finite-set action or observation channel, setting the Elements property.

example

spec = rlFiniteSetSpec(elements,Name=Value) creates the specification object spec and sets its Properties using one or more name-value arguments.

Properties

expand all

Set of valid actions or observations for the environment, specified as one of the following:

  • Vector — Specify valid numeric values for a single action or single observation.

  • Cell array — Specify valid numeric value combinations when you have more than one action or observation. Each entry of the cell array must have the same dimensions.

Agents and policies do not check whether an observation belongs to the specified set of elements.

Example: Elements=[-2 -1 0 1 2]

Name of the rlFiniteSetSpec object, specified as a string. Use this property to set a meaningful name for the signal carried by this data channel. This property is used by the RL Agent block to match the bus signals with their corresponding environment channels.

Example: Name="Action"

Description of the rlFiniteSetSpec object, specified as a string. You can use this property to specify a meaningful description of the signal carried by this environment channel.

Example: Description="Applied force in N"

This property is read-only.

Size of each element, specified as a vector.

If you specify Elements as a vector, then Dimension is [1 1]. Otherwise, if you specify a cell array, then Dimension indicates the size of the entries in Elements. This property is essential for creating agents and function approximators objects that work with a given environment.

Example: Dimension=[1 1]

This property is read-only.

Information about the type of data, specified as a string, such as "double" or "single". The software uses this property to enforce data type consistency for observations and actions.

Example: DataType="single"

Object Functions

rlSimulinkEnvCreate environment object from a Simulink model already containing agent and environment
rlFunctionEnvCreate custom reinforcement learning environment using your reset and step functions
rlValueFunctionValue function approximator object for reinforcement learning agents
rlQValueFunction Q-Value function approximator with a continuous or discrete action space reinforcement learning agents
rlVectorQValueFunction Vector Q-value function approximator with hybrid or discrete action space for reinforcement learning agents
rlContinuousDeterministicActor Deterministic actor with a continuous action space for reinforcement learning agents
rlDiscreteCategoricalActorStochastic categorical actor with a discrete action space for reinforcement learning agents
rlContinuousGaussianActorStochastic Gaussian actor with a continuous action space for reinforcement learning agents

Examples

collapse all

This example uses:

Open Live Script

For this example, consider the rlSimplePendulumModel Simulink® model. The model is a simple frictionless pendulum that initially hangs in a downward position.

Open the model.

mdl = "rlSimplePendulumModel";
open_system(mdl)

Create rlNumericSpec and rlFiniteSetSpec objects for the observation and action specifications, respectively.

The observation is a vector containing three signals: the sine, cosine, and time derivative of the angle.

obsInfo = rlNumericSpec([3 1]) 
obsInfo = 
  rlNumericSpec with properties:

     LowerLimit: -Inf
     UpperLimit: Inf
           Name: [0×0 string]
    Description: [0×0 string]
      Dimension: [3 1]
       DataType: "double"

The action is a scalar expressing the torque and can be one of three possible values, -2 Nm, 0 Nm and 2 Nm.

actInfo = rlFiniteSetSpec([-2 0 2])
actInfo = 
  rlFiniteSetSpec with properties:

       Elements: [3×1 double]
           Name: [0×0 string]
    Description: [0×0 string]
      Dimension: [1 1]
       DataType: "double"

You can use dot notation to assign property values for the rlNumericSpec and rlFiniteSetSpec objects.

obsInfo.Name = "observations";
actInfo.Name = "torque";

Assign the agent block path information, and create the reinforcement learning environment for the Simulink model using the information extracted in the previous steps.

agentBlk = mdl + "/RL Agent";
env = rlSimulinkEnv(mdl,agentBlk,obsInfo,actInfo)
env = 
SimulinkEnvWithAgent with properties:

           Model : rlSimplePendulumModel
      AgentBlock : rlSimplePendulumModel/RL Agent
        ResetFcn : []
  UseFastRestart : on

You can also specify a reset function using dot notation. For this example, randomly initialize theta0 in the model workspace.

env.ResetFcn = @(in) setVariable(in,"theta0",randn,"Workspace",mdl)
env = 
SimulinkEnvWithAgent with properties:

           Model : rlSimplePendulumModel
      AgentBlock : rlSimplePendulumModel/RL Agent
        ResetFcn : @(in)setVariable(in,"theta0",randn,"Workspace",mdl)
  UseFastRestart : on
Open Live Script

If your environment has an observation space consisting of multiple channels, some continuous, some discrete, use a vector of specification objects (each defining a single channel) to define the observation space.

For example, define an observation space as consisting of four channels. The first one carries a single number labeled 7, 9, 19 or -2. The second one carries a vector over a continuous three-dimensional space. The third channel carries a two by two matrix that can be either the zero matrix (that is zeros(2)) or the identity matrix (that is eye(2)). Finally, the fourth channel carries a continuous matrix with four rows and three columns.

obsInfo = [  rlFiniteSetSpec([7 9 19 -2])
             rlNumericSpec([3 1])
             rlFiniteSetSpec({zeros(2), eye(2)})
             rlNumericSpec([4 3]) ]
obsInfo=4×1 heterogeneous RLDataSpec (rlFiniteSetSpec, rlNumericSpec) array with properties:
    Name
    Description
    Dimension
    DataType

You can access each channel specification using dot notation.

obsInfo(1)
ans = 
  rlFiniteSetSpec with properties:

       Elements: [4×1 double]
           Name: [0×0 string]
    Description: [0×0 string]
      Dimension: [1 1]
       DataType: "double"


obsInfo(2).Name = "Velocity";
obsInfo(2).Description = "Velocity vector in m/s in body reference frame";
obsInfo(2)
ans = 
  rlNumericSpec with properties:

     LowerLimit: -Inf
     UpperLimit: Inf
           Name: "Velocity"
    Description: "Velocity vector in m/s in body reference frame"
      Dimension: [3 1]
       DataType: "double"
Open Live Script

Within Reinforcement Learning Toolbox™ software, agents can only have one action channel. However, if your desired design involves multiple discrete action channels, you can convert them into a single action channel with a number of elements equal to the product of the number of elements of each action channel of your original design. Specifically, each element of the single action channel corresponds to a particular combination of elements in the action channels of your original design.

For example, suppose that the valid values for a two-output system are [1 2] for the first output and [10 20 30] for the second output. Create a discrete action space specification for all possible output combinations.

actionSpec = rlFiniteSetSpec({[1 10],[1 20],[1 30],...
                              [2 10],[2 20],[2 30]})
actionSpec = 
  rlFiniteSetSpec with properties:

       Elements: {6×1 cell}
           Name: [0×0 string]
    Description: [0×0 string]
      Dimension: [1 2]
       DataType: "double"

Version History

Introduced in R2019a

See Also

Functions

Objects

Blocks

Topics