How can I solve the NaN error during RL training process

Dear Sir/Madam,

I'm Adithya, student from TU Kaiserslautern, Germany and I'm trying to train the RL algorithm for my project work. For few policies I was getting the error in model: "Error using rl.env.internal.reportSimulinkSimError

Simulink cannot solve the algebraic loop containing 'qss_hybrid_electric_vehicle_template/Controller/Sum' at time 587.0 due to one of the following reasons: the model is ill-defined

i.e., the system equations do not have a solution; or the nonlinear equation solver failed to converge due to numerical issues.

To rule out solver convergence as the cause of this error, follow either of the suggested actions. If the error persists in spite of the above changes, then the model is likely

ill-defined and requires modification."

Is ther a way to tell the training agent to discard the current policy and conitnue with the next one without throwing the error.

With best regards,

Adithya

[Mokhtar]

hello Adithya, did you find a solution for this problem ? i am stuck at it too right now.

Answer 1

[Ayush Aniket]

Hi Adithya Ramakrishna Lingadahalli,

As per my understanding, your RL agent occasionally encounters an error during training due to an issue with the Simulink model.

The 'train()' function does not have any explicit parameter to handle this type of issue. A workaround could be to use your own custom training loop which is exactly what the 'train()' function does internally. This will allow you to figure out the error occurring in the specific step for your agent or if you want to discard that episode, you can do so by using a `try-catch` statement in your training loop. For more information for training an RL agent using custom training loop refer to the example in the following link:

https://www.mathworks.com/help/reinforcement-learning/ug/train-reinforcement-learning-policy-using-custom-training.html#TrainRLPolicyUsingCustomTrainLoopExample-4

However, it is important to note that this is not the permanent solution. It might be better to investigate why these errors are occurring and try to fix the underlying issue. The error message suggests there might be an algebraic loop in your Simulink model that Simulink cannot solve for some policies. You can refer to the following MATLAB answer link which will help you in solving them:

https://www.mathworks.com/matlabcentral/answers/95310-what-are-algebraic-loops-in-simulink-and-how-do-i-solve-them

Hope it helps.