rlMDPEnv

Create Markov decision process environment for reinforcement learning

Description

A Markov decision process (MDP) is a discrete-time stochastic control process in which the state and observation belong to finite spaces, and stochastic rules govern state transitions. It provides a mathematical framework for modeling decision making in situations where outcomes are partly random and partly under the control of the decision maker. MDPs are useful for studying optimization problems solved using reinforcement learning. Use rlMDPEnv to create a Markov decision process environment for reinforcement learning in MATLAB^®.

Creation

Syntax

env = rlMDPEnv(MDP)

Description

env = rlMDPEnv(MDP) creates a reinforcement learning environment env with the specified MDP model.

Input Arguments

MDP — Markov decision process model

GridWorld object | GenericMDP object

Markov decision process model, specified as one of these objects:

• GridWorld object created using the createGridWorld function

  function

• GenericMDP object created using the createMDP function

Properties

Model — Markov decision process model

GridWorld object | GenericMDP object

Markov decision process model, specified as a GridWorld object or GenericMDP object.

Example: env.Model=createMDP(3,["left";"right"])

ResetFcn — Reset function

function handle

Reset function, specified as a function handle.

Example: env.ResetFcn=@() randi(3)

Object Functions

getActionInfo	Obtain action data specifications from reinforcement learning environment, agent, or experience buffer
getObservationInfo	Obtain observation data specifications from reinforcement learning environment, agent, or experience buffer
sim	Simulate trained reinforcement learning agents within specified environment
train	Train reinforcement learning agents within a specified environment
validateEnvironment	Validate custom reinforcement learning environment

Examples

Create Grid World Environment

For this example, consider a 5-by-5 grid world with these rules:

1. A 5-by-5 grid world bounded by borders, with four possible actions: North = 1, South = 2, East = 3, West = 4.

2. The agent begins from cell [2,1] (second row, first column, indicated by the red circle in the figure).

3. The agent receives reward +10 if it reaches the terminal state at cell [5,5] (blue cell).

4. The environment contains a special jump from cell [2,4] to cell [4,4] with +5 reward (blue arrow).

5. The agent is blocked by obstacles in cells [3,3], [3,4], [3,5], and [4,3] (black cells).

6. All other actions result in –1 reward.

First, create a GridWorld object using the createGridWorld function.

GW = createGridWorld(5,5)

GW = 
  GridWorld with properties:

                GridSize: [5 5]
            CurrentState: "[1,1]"
                  States: [25×1 string]
                 Actions: [4×1 string]
                       T: [25×25×4 double]
                       R: [25×25×4 double]
          ObstacleStates: [0×1 string]
          TerminalStates: [0×1 string]
    ProbabilityTolerance: 8.8818e-16

Then set the initial, terminal, and obstacle states.

GW.CurrentState = "[2,1]";
GW.TerminalStates = "[5,5]";
GW.ObstacleStates = ["[3,3]";"[3,4]";"[3,5]";"[4,3]"];

Update the state transition matrix for the obstacle states and set the jump rule over the obstacle states.

updateStateTranstionForObstacles(GW)
GW.T(state2idx(GW,"[2,4]"),:,:) = 0;
GW.T(state2idx(GW,"[2,4]"),state2idx(GW,"[4,4]"),:) = 1;

Next, define the rewards in the reward transition matrix.

nS = numel(GW.States);
nA = numel(GW.Actions);
GW.R = -1*ones(nS,nS,nA);
GW.R(state2idx(GW,"[2,4]"),state2idx(GW,"[4,4]"),:) = 5;
GW.R(:,state2idx(GW,GW.TerminalStates),:) = 10;

Now, use rlMDPEnv to create a grid world environment using the GridWorld object GW.

env = rlMDPEnv(GW)

env = 
  rlMDPEnv with properties:

       Model: [1×1 rl.env.GridWorld]
    ResetFcn: []

You can visualize the grid world environment using the plot function.

plot(env)

Use the getActionInfo and getObservationInfo functions to extract the action and observation specification objects from the environment.

actInfo = getActionInfo(env)

actInfo = 
  rlFiniteSetSpec with properties:

       Elements: [4×1 double]
           Name: "MDP Actions"
    Description: [0×0 string]
      Dimension: [1 1]
       DataType: "double"

obsInfo = getObservationInfo(env)

obsInfo = 
  rlFiniteSetSpec with properties:

       Elements: [25×1 double]
           Name: "MDP Observations"
    Description: [0×0 string]
      Dimension: [1 1]
       DataType: "double"

You can now use the action and observation specifications to create an agent for your environment, and then use the train and sim functions to train and simulate the agent within the environment.

Version History

Introduced in R2019a