What if I provide just 1 critic instead of 2 to a rlSACAgent algorithm?

I have runned well my rlSACAgent and is still running, but today I realized that I had to create 2 critics in order to implement well this algorithm.

Now, it have been running for days, and I already know this is a really hard problem that I have already solved with a DDPG agent (and it also took days to converge), but I think this is a big mistake and I am afraid that my rlSACAgent will never be able to converge to a solution due this... Matlab have never shown me any warning or error for not having two critics...

I am really concerned because right now I am running out of time... What is your opinion about this?

Thanks..

Answer 1

[Neha]

Hi Manuel,

I understand that you want to know if it is advisable to retrain the "rISACAgent" with two critics.

If you have been running your agent for days without encountering any errors or warnings, it is possible that the algorithm is still making progress towards convergence, despite not having two critics. However, having two critics is an essential part of the Soft Actor-Critic (SAC) algorithm and is recommended for optimal performance.

The purpose of having two critics in SAC is to estimate the state-action value function more accurately by using the minimum value between the two critics. This helps to reduce overestimation bias, which can be problematic in value-based reinforcement learning algorithms.

Not having two critics may result in slower convergence or suboptimal performance compared to the standard SAC implementation. If you believe that using two critics is crucial for your problem and you want to ensure optimal performance, it would be advisable to stop the current training and modify your implementation to include the second critic. Although it may require additional time to retrain the agent, it will likely lead to better results in the long run.

For more information about SAC algorithm, please refer to Soft Actor-Critic (SAC) Agents - MATLAB & Simulink (mathworks.com)

I hope this helps!