satlin and purelin in deep learning toolbox

I am just getting started working with a student on the neural network toolboxes available from Matlab, and i am unclear on what is the relationship between the various parts. To give a specific example of my confusion, according to the documentation, the function trainAutoEncoder allows specification of activation functions including logsig, satlin, and purelin. However, this list of activation functions is seemingly completely disjoint from the activation functions available in the deep learning toolbox, which include relu, leakyrelu, etc. Are these totally different products (autoencoder vs deep learning)? What if I want a satlin or purelin activation function in a deep neural network? Or if I want relu in an autoencoder? Thanks.

[Stephen Vavasis]

Thanks very much! After several more weeks of deliberation after the original post, my student and I eventually decided to pursue a solution in Python instead. However, this answer clarifies the situation if we embark on another neural net project using Matlab.

Answer 1

[Kausthub]

Hi Stephen Vavasis,

I understand that you are facing a confusion regarding the Deep Learning Toolbox as there are different functions available in MATLAB for neural networks and deep learning and have multiple questions regarding the activation functions that it offers.

I have addressed your questions below:

• Relationship between Autoencoder and Deep Learning Toolbox and are these totally different products?

No, Autoencoder and Deep Learning Toolbox are not different products. Autoencoder is a class in Deep Learning Toolbox. Autoencoder is a part of the Deep Learning Toolbox which is used to create an autoencoder network, which consists of an encoder and a decoder.

• Activation functions in Autoencoder and in Deep Learning Toolbox.

Autoencoder offers a range of activation functions like `logsig`, `satlin`, `purelin`, etc whereas the Deep Learning Toolbox provides a different set of activation functions like `relu`, `leakyrelu`, `sigmoid`, etc.

• What if I want a satlin or purelin activation function in a deep neural network? Or if I want relu in an autoencoder?

If you want to use `satlin` or `purelin` activation functions in a deep neural network or use `relu`, `leakyrelu` or other activation functions in an autoencoder you can achieve this by customizing the network architecture using the Deep Learning Toolbox. You can create a custom and define your desired activation function.

Here are some resources for creating custom layers:

• Define Custom Deep Learning Layers
• Customizing ReluLayer
• A MATLAB Answer related to creating customized activation functions

I hope this clarifies the relationship between the various parts of the Deep Learning Toolbox and addresses your questions regarding activation functions.