RL Agent can't inherint FiM sampling time

I am trying to experiment with this example. I replaced all logical operatios in reward and stop simulation blocks with MATLAB functions to get rid of Fixed in Minor Step (FiM) signals. This is the result:

I re-trained the agent and everything went well. Then, I made modification in agent's SampleTime. Since I want the agent to inherit environment's sampling time (decided by the solver), I set agent.AgentOptions.SampleTime = -1. During the simulation, this error message occured:

An error occurred while running the simulation for model 'rlwatertank' with the following RL agent blocks:

rlwatertank/RL Agent

Caused by:

Error using rl.env.internal.reportSimulinkSimError (line 29)

The block rlwatertank/RL Agent is not supported when inheriting a "Fixed In Minor Step" sample time. Either place a rate transition block at the block inports to inherit a discrete sample time or specify agent.SampleTime > 0.

MATLAB/Simulink said RL Agent cannot inherit FiM sample time, but there is no FiM signal in the model as we can see. How do I solve this problem? How can I make RL Agent inherit environment's sampling time? I'am using R2024b Update 5.

[Ari]

My conclusion so far: After some trials, in simulink it seems RL Agent can only inherit sample time from discrete signals.

[Ari]

@Emmanouil Tzorakoleftherakis, could you please give a comment on this matter?

[Emmanouil Tzorakoleftherakis]

Quick question before I reply: why do you want the agent to inherit the environment's sample time? This is not a requirement, and in fact it can often lead to unnecessary computational burden. For example, the solver may need a pretty small time step (if it's fixed step) to get decent simulation accuracy. The controller sample time however is completely independent of that and the environment. It may need to be smaller or larger depending on the application.

[Ari]

Consider a scenario where the reference is not stationary. For example it may look like this:

During transient period, the environment needs action update more frequently since the dynamic changes rapidly. On the other hand, during steady period the environment may receive action update less frequent. If I use low sampling rate for the agent, it could "miss" the dynamic in transient region. But, if I use high sampling rate, it may waste computational resources during steady period. I could be wrong, but I think it's a good thing if the solver and agent could have a "communication" like:

solver: hey agent, I just updated the state, give us a new action signal

agent: ok solver, here is your new action

This can be achieved in MATLAB environment using custom step function. It should be possible in Simulink environment as well, I hope.

[Emmanouil Tzorakoleftherakis]

Thanks, will reply below. Fyi, what you are describing is easily implementable in MATLAB because in MATLAB environments there is no notion of time like in Simulink - you can define what time is somewhat arbitrarily.

Answer 1

[Emmanouil Tzorakoleftherakis]

Based on the comments section above, it seems you are looking to have an "adaptive" sample time for the RL Agent block. The screenshot you have above shows that the input signals to the RL Agent block are continuous - this cannot happen. RL needs to happen at discrete sample time per definition of MDP. A couple of options I see:

1) Add a rate transition block to the output of the water tank subsystem (height signal). Set the sample time in the block to be whatever makes sense to you. The RL Agent block will inherit the same sample time. Note this will still be "fixed" time step.

2) For adaptive sample time, you need to place the RL Agent block inside a triggered subsystem. Doing that you can "call" the RL Agent block at varying sample time based on some event (could be the variable-step sample time of the environment or some other metric you define to recognize transients vs steady state).

Hope this helps