What is the fcnLossLayer in the generatePolicyFunction network and how can I implement it myself?

I wish to use my trained SAC agent in Matlab only as a policy function. For this, I tried the generatePolicyFunction option. The code and the network works as intended, but I'm having a hard time figuring out the last layer of the network. It says it's a fcnLossLayer, although I have no idea what does it mean or how does it work. It's description is very vauge:

>> policy.Layers(11, 1)

ans = 

  FcnLossLayer with properties:

              LossFcn: []
    IsNetworkStateful: 0
                 Name: 'RepresentationLoss'
        ResponseNames: {}
          Description: ''
                 Type: "GenericLossLayer"

This is the only information I get from the DAGNetwork object and it makes no sense to me. Also I couldn't find any relevant information in the documentation, or in the referenced articles, or anywhere on the internet. I found it's type is rl.layer.FcnLossLayer, but searching for this I still got nothing. This seams to me like a focalLossLayer, but even if it is, I don't know its parameters.

This is important to me because I want to use this policy in a previous Matlab version (R2020a), and it doesn't seem to support this layer, so I want to try and implement it as a custom layer. Thank you for your help in advance!

[Szilard Hunor Toth]

I found a solution: for exploitation, one can just simply use a regression layer instead of this loss layer, it seems to work exactly the same. As for what this loss layer represents, I'm guessing it applies the actor loss function, but I'm still not sure about that.

Answer 1

[Shubham]

Hi Szilard,

Your approach to using a regression layer in place of the FcnLossLayer for exploitation purposes in your Soft Actor-Critic (SAC) agent's policy network is a creative solution. Indeed, for the exploitation phase where the objective is to use the trained policy to make decisions (actions) given states, the exact loss computation during training (which involves learning and optimization) is less critical. The key during exploitation is to accurately predict the action values based on the current state, for which a regression layer can be suitable.

Understanding the Role of the FcnLossLayer

Your guess about the FcnLossLayer representing the actor loss function in the SAC framework is on point. In reinforcement learning, especially in actor-critic methods like SAC, the actor's job is to propose actions given states, while the critic evaluates these actions by estimating the value function. The loss layer for the actor, therefore, is crucial during the training phase for adjusting the policy towards more rewarding actions. The actor loss typically encourages the policy to maximize the expected return, adjusted for the entropy term in SAC to encourage exploration.

This loss computation is quite specific and involves gradients of the policy network with respect to action values, adjusted by the critic's feedback and possibly entropy terms for exploration efficiency. Hence, during training, this FcnLossLayer plays a critical role in learning by applying the specific SAC actor loss function, which might not be straightforwardly replicated by standard neural network layers.

Exploitation with a Regression Layer

For exploitation, where the goal is to use the policy network to predict the best action given the current state (without further learning), replacing the FcnLossLayer with a regression layer makes sense. The regression layer can output the action values directly from the input states, bypassing the need for the specific loss computation used during training.

This approach simplifies the use of the trained model in environments or MATLAB versions that do not support the specialized FcnLossLayer. It aligns with the exploitation goal of making the best decision based on the learned policy, without the complexity of adjusting the policy further.

Final Thoughts

Your solution illustrates a practical way to adapt and utilize a trained reinforcement learning model for exploitation, even when facing compatibility issues with certain layers across different MATLAB versions. It's a good example of understanding the underlying principles of the model and the framework (SAC in this case) and applying that understanding to effectively use the model within the constraints of your tools and environment.

Keep in mind, though, that should you wish to further train or fine-tune the policy under different conditions or with additional data, reintegrating or emulating the functionality of the original FcnLossLayer might become necessary to accurately apply the SAC algorithm's actor loss function.