The reward gets stuck on a single value during training or randomly fluctuates (Reinforcement Learning)

I train the reinforcement learning system, and on the reward plot I have some failures during which the reward does not change. This doesn’t look normal, especially when compared with examples (Biped Robot, etc.) I believe that some rlDDPGAgentOptions settings are responsible for this, but it seems that I changed all the possible settings, but even after several thousand episodes, the system does not learn. What can be the reason for this behavior of this graph during training?

Answer 1

[Ari Biswas]

It could mean that the training is experiencing a local minima. You can try out a few things:

1. Change the OU noise options to favor more exploration so that the robot can explore more states and get new rewards.

2. Design a different reward function that is not too dependent on sparse rewards. From the graph (flatlines) it looks like you have a sparse reward for a state that the agent is continuously visiting.

In most cases, designing better reward functions will improve training. That being said, 350 episodes might be too early to expect good results. I would let it run for a few 1000 episodes at least before coming to a conclusion that something needs to change.

[Vasiliy Polushkin]

Thank you very much. Can you still tell me. I changed the settings and the plot got better. But now, if I wait a long time, several thousand episodes, the reward is constantly growing for some time, but at a certain point it begins to fall. What could this be connected with? Could this be due to insufficient depth of the neural network?

[Ari Biswas]

It could mean that the agent is exploring a different region in the state space that it has not experienced before and it is quite common for rewards to fluctuate in that situation.

[Abd Al-Rahman Al-Remal]

Hi,

When you say to change the noise options to favour more exploration: how would this be implemented? i.e what parameters should be changed and in what manner?

My case is slightly different than OP's however as my agent just stays at the same reward value consistently (I've never tested it for more than 100 episodes or so however).

Many thanks!

[Ari Biswas]

For a DDPG agent you can tune the StandardDeviation and StandardDeviationDecayRate parameters. Please see the documentation for instructions.

https://www.mathworks.com/help/reinforcement-learning/ref/rlddpgagentoptions.html#mw_2875b71d-bfb0-4be4-b0d3-a44592c3cb30_head