Get input/output gradient of neural network

Matlab's built in functions in the NN toolbox seem to provide a good set of options for getting the gradient of the network performance wrt the network parameters. Is there a way to get the gradient of the network output with respect to the network input?

Answer 1

[Mahesh Taparia]

Hi

In general, in any neural network, the network tries to learn the weights which can reduce the cost/ loss function. The gradients are updated iteratively by using the derivative of loss function with respect to weights.

Usually for a fix input, calculating gradients of loss with respect to input is not meaningful because if input is fix, then d(loss)/d(Input) is not defined. If the network is feed with 2 different input sequence, in this case you can find the gradient by calculating (Loss2-Loss1)/(X2-X1), where Loss is the value of network loss with respect to input X. There is no use of this while training the network.

Hope it will helps!

[Aaron Kandel]

I'm actually not looking to find any derivative with respect to the loss function. I simply want the input/output derivative of the network. E.g. if y = f(x), where y is the network output, f is the neural network model, and x is the network input, I want dy/dx. Apologies if that was unclear.

[Aaron Kandel]

The documentation indicates that function still only provides derivatives with respect to the weights and biases, I'm confused.

[Mahesh Taparia]

Hi

Mostly the neural network is highly non linear which consists of several activation functions.As of now there is no direct function which can calculate the gradient of the network output with respect to the network input. The approach is to derive the equations of derivative of the network. However, you can use the trained network which contains the trained weights and the used activations function. It can help in evaluation. For example, let F1 and F2 be the activation function and W1, W2 be the trained weights and B1 and B2 be the trained bias of each layer, then

Y=F2(W2*F1(W1*X+B1)+B2);

In this case, Y' (derivative with respect to X) can be written as:

Y'=W2*F2'{W2*(F1(W1*X+B1))+B2}*W1*{F1'(W1*X+B1)};

So, the nodes output are passed through the derivative of activation and the weights are getting multiplied. The nodes outputs and weights are stored in the trained network and for each layer. The derivatives of activation can be made as a separate function for simplicity.

Hope it will helps!

[Aaron Kandel]

Thank you for your input and help! I wish Matlab had the full functionality to provide analytic gradients with built in functions, but hard coding them should work for the project I'm working on.

[David Leather]

This seems like an oversight. When applying the trained neural network to other applications, it is essential to be able to evaluate the gradient wrt to the output of the neural network, and not the loss function....

[Ruyue Yang]

Get the gradient dy/dx can be really trick for the trained multi-layer neural network (not that deep, maybe 3 or 4 layer). Such function can help a lot for the network's various application.