Trying to train PPO RL agent

Hello,

I'm trying to train a PPO agent, but I'm encountering the following issue:

From a certain time, the agent don't learn anymore (although the agent is only in a local maximum). Let's say that for the ten first episodes the agent gets a very bad reward, since it's actually perfoming bad. Then, on the 11th episode (see graph below), the agent found a local maxmimum by updating its actions value to 30 and -30 (these are the gains coefficients of a PI controller). Finally, starting from the 12th episode (i.e. the next one), the agent don't update its action values anymore.

As a solution, I've already tried to increase the EntropyLossWeight, from 0.02 to 1. I've tried a lot of values in this range, and it seems like nothing is efficient.

Another parameter may influence the result: from different actions taken in a very wide range of values (e.g. from [1; ∞] for the first action) any system output variation is perceptible, and thus any variation in the reward can't be seen watever the action value taken within this range of values. In another words, on the picture below, the agent tried three differents values of gains, but the three actions values produced the same result, So, maybe the agent can't learn from it.

So, I would like it to continue exploring, although it just got a better reward, since it is still not the best it can achieve.

Link to PPO agent options, including EntropyLossWeight: Options for PPO agent - MATLAB - MathWorks Switzerland

Any help would be very kind!

Thanks a lot in advance!

Nicolas

Answer 1

[Karan Singh]

Hi Nicolas,

The problem you are facing is a common scenario, and in my view, the only way to proceed forward is through trial and error of various parameters. Here are some which I have tried in my personal projects that may be useful for you:

1. Modify the "learning rate" to see if it helps the agent to escape the local maximum.
2. The "clip factor" in PPO controls how much the new policy can deviate from the old policy. Adjusting this parameter could help the agent explore more.
3. Adding noise to the actions can encourage exploration. For example, you can add Gaussian noise to the action values. Here is an interesting article by OpenAI on the same: https://openai.com/index/better-exploration-with-parameter-noise/
4. Lastly, consider the "Entropy Loss Weight," which you have already tried.

[Nicolas CRETIN]

Thank you Karan for your help!

[Nicolas CRETIN]

But, what does exactly parameter noise refer to ?

Is it noise that we apply to the observation signal, which then goes into the actor/critic network ?

Is it noise that we apply to the output of the actor net ?

Or does it refer to increasing the range of values of the standard deviation branch of the actor net?

Actually, I don't really understand the picture below (which comes from the article above):