what do pruningActivations and pruningGradients refer to?

In pruning using Taylor scores example, I can't understand what pruningActivations and pruningGradients refer to

 [loss,pruningActivations, pruningGradients, netGradients, state] = ...
            dlfeval(@modelLossPruning, prunableNet, X, T);

[Matt J]

This is not an AI site. Only humans on here.

[muhamed ibrahim]

Thank you for your answer but I don’t think it’s relevant to my question.

[Matt J]

No, it's not an answer to your question. It's to tell you that no human would be able to understand the context of your question from just one line of code taken from somewhere in the documentation, so maybe you thought you were on a GPT engine. If you are looking for an AI interface, you can find one here,

https://www.mathworks.com/matlabcentral/playground/new

Otherwise, providing a link to the full code and example would be a good start.

[Walter Roberson]

The discussion appears to be related to https://www.mathworks.com/help/deeplearning/ref/deep.prune.taylorprunablenetwork.updateprunables.html

Answer 1

[Shantanu Dixit]

Hi Muhamed, If I correctly understood the query you're referring to the example: https://www.mathworks.com/help/deeplearning/ug/prune-image-classification-network-using-taylor-scores.html which details on pruning networks for resource efficient inferencing.

In this context 'pruningActivations' refers to the outputs of the layers designated as prunable (ex: in convolution layers, each activation corresponds to the output of a filter), similarly 'pruningGradients' refers to the gradients of the loss with respect to the 'pruningActivations' (measures the sensitivity: larger the gradient, greater the impact on loss).

Correspondingly, 'Taylor Score' is computed as the element-wise product of 'pruningActivations' and 'pruningGradients' using 'updateScore'. Higher scores imply the activation and associated parameters is more critical (for more information you can refer to References Section: Article 2, Section 2.2 which details on 'criteria for pruning').

% Compute first-order Taylor scores and accumulate the score across
% previous mini-batches of data.
prunableNet = updateScore(prunableNet, pruningActivations, pruningGradients);

You can refer to the below pruning network related references for more information:

Pruning network using Taylor Scores: https://in.mathworks.com/help/deeplearning/ug/prune-image-classification-network-using-taylor-scores.html#PruningUsingTaylorPrunableNetworkExample-3

updateScore: https://www.mathworks.com/help/deeplearning/ref/deep.prune.taylorprunablenetwork.updatescore.html

Pruning CNNs for resource efficient inference: Molchanov, Pavlo, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. “Pruning Convolutional Neural Networks for Resource Efficient Inference.” Preprint, submitted June 8, 2017. https://arxiv.org/abs/1611.06440.

Hope this helps.