Simulink Reinforcement Learning simulation glitch during training with rate transition block and fixed step solver

So I wrote a custom RL agent, but in order for it to compile I had to add a rate transition block between the custom agent and the observation,reward,done signal as well as change the solver to a fixed step size solver. But I ran into glitches when I ran it such as my robot that i'm training, its rendering flickers during the simulation and it coincides with itself.

So to see if the glitch was caused by my code or the change in solver and addition of rate transition block I used Matlab built in TD3 agent with the fixed solver and rate transition block and reproduced similar errors of the robot with its rendering flickering again.

Can someone explain to me how the rate transition block works in Simulink specifically relating to my training environment? Like for example, if my agent has a sample time of 0.1 sec and my solver has a step size of 0.01. So my state is fed into my agent, and then my agent outputs the action. Since my solver's step size is 0.01 and my agent sample time is 0.1, would my solver run 10 times and then after the 10th iteration, then feed the result as the next state into my agent? And the rate transition block facilitates this?

Also I hypothesize that maybe since I switched a fixed step solver. I put the step size at 0.01, maybe this step size is too large and hence the dynamics are inaccurate and that's causing the glitch?

Any insight or advice is appreicated, thank you

Answer 1

[Vanita]

Hi Vincent,

I understand that sample time of the agent and the solver is matching, to fix this issue you may consider using "Variable Step Solver", please refer to the following MATLAB R2025a documentation for reference:

https://www.mathworks.com/help/simulink/ug/variable-step-solvers-in-simulink-1.html

Hope this helps!