MATLAB Answers

How to compute gradients using the Neural Network Toolbox software through a backpropagation process?

9 views (last 30 days)
Jungyeon Baek
Jungyeon Baek on 15 Jan 2019
Edited: Jungyeon Baek on 17 Jan 2019
I've been reading Neural_Network_Toolbox_Users_Guide and I have a question about below section.
As what it said in 3-19,
In fact, the gradients and Jacobians for any network that has differentiable transfer functions, weight functions and net input functions can be computed using the Neural Network Toolbox software through a backpropagation process. You can even create your own custom networks and then train them using any of the training functions in the table above. The gradients and Jacobians will be automatically computed for you.
But I don't understand how exactly this works.
Here is what I'm trying to demonstrate.
  1. We use neural networks as a non-linear function approximator to estimate the real function.
  2. We have inputs(x) and refer the real function(y) with weights that we try to estimate using neural networks.
  3. A neural network can be trained by minimizing a loss function that changes at each iteration.
  4. Loss function can be the average of square of (y'-y) , where y' is the target function. Briefly speaking, the gradient of loss function can be represented with (y'-y)▽y.
  5. Here, note that the target function(y') depends on weights this is different from general learning (e.x. supervised learning), which are fixed before training, so that the target function is changed at each iteration.
  6. Therefore, what I'm looking for is how to calculate ▽y at each iteration to update weights.
Could you help me to understand and give an example about this?


Sign in to comment.

Answers (1)

Greg Heath
Greg Heath on 16 Jan 2019
You are confused
The target function is constant and independent of the weights.
Hope this helps.
*Thank you for formally accepting my answer*

  1 Comment

Jungyeon Baek
Jungyeon Baek on 17 Jan 2019
I know the target need to be fixed but it can be depended on previous iteration weights as I mentioned this is in contrast with supervised learning. Therefore, it isn't affected when we differentiate the loss function with regard to the weights because it doesn't depend on the weights that we try to find this iterations, but the previsous weights.

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!