dlnetwork

Deep learning neural network

Description

A dlnetwork object specifies a deep learning neural network architecture.

Tip

For most deep learning tasks, you can use a pretrained neural network and adapt it to your own data. For an example showing how to use transfer learning to retrain a convolutional neural network to classify a new set of images, see Retrain Neural Network to Classify New Images. Alternatively, you can create and train neural networks from scratch using the trainnet and trainingOptions functions.

If the trainingOptions function does not provide the training options that you need for your task, then you can create a custom training loop using automatic differentiation. To learn more, see Train Network Using Custom Training Loop.

If the trainnet function does not provide the loss function that you need for your task, then you can specify a custom loss function to the trainnet as a function handle. For loss functions that require more inputs than the predictions and targets (for example, loss functions that require access to the neural network or additional inputs), train the model using a custom training loop. To learn more, see Train Network Using Custom Training Loop.

If Deep Learning Toolbox™ does not provide the layers you need for your task, then you can create a custom layer. To learn more, see Define Custom Deep Learning Layers. For models that cannot be specified as networks of layers, you can define the model as a function. To learn more, see Train Network Using Model Function.

For more information about which training method to use for which task, see Train Deep Learning Model in MATLAB.

Creation

Syntax

net = dlnetwork

net = dlnetwork(layers)

net = dlnetwork(layers,OutputNames=names)

net = dlnetwork(layers,Initialize=tf)

net = dlnetwork(layers,X1,...,XN)

net = dlnetwork(layers,X1,...,XN,OutputNames=names)

net = dlnetwork(prunableNet)

net = dlnetwork(mdl)

Description

Empty Network

net = dlnetwork creates a dlnetwork object with no layers. Use this syntax to create a neural network from scratch. (since R2024a)

Network with Input Layers

net = dlnetwork(layers) creates neural network using the specified layers and initializes any unset learnable and state parameters. This syntax uses the input layer in layers to determine the size and format of the learnable and state parameters of the neural network.

Use this syntax when layers defines a complete single-input neural network, has layers arranged in series, and has an input layer.

net = dlnetwork(layers,OutputNames=names) also sets the OutputNames property. The OutputNames property specifies the layers or layer outputs that correspond to network outputs.

Use this syntax when layers defines a complete single-input multi-output neural network, has layers arranged in series, and has an input layer.

net = dlnetwork(layers,Initialize=tf) specifies whether to initialize the learnable and state parameters of the neural network. When tf is 1, (true), this syntax is equivalent to net = dlnetwork(layers). When tf is 0 (false), this syntax is equivalent to creating an empty network and then adding layers using the addLayers function.

Network With Unconnected Inputs

net = dlnetwork(layers,X1,...,XN) creates a neural network using the specified layers and initializes any unset learnable and state parameters. This syntax uses the network data layout objects or example inputs X1,...,XN to determine the size and format of the learnable parameters and state values of the neural network, where N is the number of network inputs.

Use this syntax when layers defines a complete neural network, has layers arranged in series, and has inputs that are not connected to input layers.

net = dlnetwork(layers,X1,...,XN,OutputNames=names) also sets the OutputNames property.

Use this syntax when layers defines a complete neural network, has multiple outputs, has layers arranged in series, and has inputs that are not connected to input layers.

Conversion

net = dlnetwork(prunableNet) converts a TaylorPrunableNetwork to a dlnetwork object by removing filters selected for pruning from the convolution layers of prunableNet and returns a compressed dlnetwork object that has fewer learnable parameters and is smaller in size.

net = dlnetwork(mdl) converts a Statistics and Machine Learning Toolbox™ machine learning model to a dlnetwork object.

Input Arguments

layers — Network layers

Layer array

Network layers, specified as a Layer array.

The software connects the layers in series.

For a list of supported layers, see List of Deep Learning Layers.

X1,...,XN — Example network inputs or data layouts

formatted dlarray object | formatted networkDataLayout object

Example data or data layouts to use to determine the size and formats of learnable and state parameters, specified as formatted dlarray objects or formatted networkDataLayout objects. The software propagates X1,...XN through the network to determine the appropriate sizes and formats of the learnable and state parameters of the dlnetwork object and initializes any unset learnable or state parameters.

The order of X1,...,XN must match the order of the layers that require inputs in layers.

Note

Automatic initialization uses only the size and format information of the input data. For initialization that depends on the values on the input data, you must initialize the learnable parameters manually.

tf — Flag to initialize learnable and state parameters

1 (true) (default) | 0 (false)

Flag to initialize learnable and state parameters, specified as one of these values:

• 1 (true) — Initialize the learnable and state parameters. The software uses the input layer in layers to determine the sizes of the learnable and state parameters.

• 0 (false) — Do not initialize the learnable and state parameters. Use this option when:

  • You expect to make further edits to the neural network. For example, when you expect to add or remove layers and connections.

  • You use the network in a custom layer and you want to use a custom initialize function.

Neural network prediction and custom training loops requires an initialized network. To initialize an uninitialized network, use the initialize function.

prunableNet — Network for pruning by using first-order Taylor approximation

TaylorPrunableNetwork object

Network for pruning by using first-order Taylor approximation, specified as a TaylorPrunableNetwork object.

Pruning a deep neural network requires the Deep Learning Toolbox Model Quantization Library support package. This support package is a free add-on that you can download using the Add-On Explorer. Alternatively, see Deep Learning Toolbox Model Quantization Library.

mdl — Classification or regression neural network from Statistics and Machine Learning Toolbox

ClassificationNeuralNetwork | RegressionNeuralNetwork | CompactClassificationNeuralNetwork | CompactRegressionNeuralNetwork

Since R2024b

Classification or regression neural network, specified as a ClassificationNeuralNetwork (Statistics and Machine Learning Toolbox), RegressionNeuralNetwork (Statistics and Machine Learning Toolbox), CompactClassificationNeuralNetwork (Statistics and Machine Learning Toolbox), or CompactRegressionNeuralNetwork (Statistics and Machine Learning Toolbox) object.

The fitcnet (Statistics and Machine Learning Toolbox) and fitrnet (Statistics and Machine Learning Toolbox) functions return ClassificationNeuralNetwork and RegressionNeuralNetwork objects, respectively. The compact (Statistics and Machine Learning Toolbox) function returns CompactClassificationNeuralNetwork and CompactRegressionNeuralNetwork objects.

Properties

Layers — Network layers

Layer array

Network layers, specified as a Layer array.

Connections — Layer connections

table

Layer connections, specified as a table with two columns.

Each table row represents a connection in the neural network. The first column, Source, specifies the source of each connection. The second column, Destination, specifies the destination of each connection. The connection sources and destinations are either layer names or have the form "layerName/IOName", where "IOName" is the name of the layer input or output.

Data Types: table

Learnables — Network learnable parameters

table

Network learnable parameters, specified as a table with three columns:

• Layer — Layer name, specified as a string scalar.

• Parameter — Parameter name, specified as a string scalar.

• Value — Value of parameter, specified as a dlarray object.

The network learnable parameters contain the features learned by the network. For example, the weights of convolution and fully connected layers.

The learnable parameter values can be complex-valued (since R2024a).

Data Types: table

State — Network state

table

Network state, specified as a table.

The network state is a table with three columns:

• Layer – Layer name, specified as a string scalar.

• Parameter – State parameter name, specified as a string scalar.

• Value – Value of state parameter, specified as a dlarray object.

Layer states contain information calculated during the layer operation to be retained for use in subsequent forward passes of the layer. For example, the cell state and hidden state of LSTM layers, or running statistics in batch normalization layers.

For recurrent layers, such as LSTM layers, with the HasStateInputs property set to 1 (true), the state table does not contain entries for the states of that layer.

During training or inference, you can update the network state using the output of the forward and predict functions.

The state values can be complex-valued (since R2024a).

Data Types: table

InputNames — Names of network inputs

cell array of character vectors

Names of the network inputs, specified as a cell array of character vectors.

Network inputs are the input layers and the unconnected inputs of layers.

For input layers and layers with a single input, the input name is the name of the layer. For layers with multiple inputs, the input name is "layerName/inputName", where layerName is the name of the layer and inputName is the name of the layer input.

For networks with multiple inputs, training and prediction functions use this property to determine the order of the inputs. For example, for in-memory inputs X1,...,XM to the predict function, the order of the inputs must match the order of the corresponding inputs in the InputNames property of the network.

To customize the order, set InputNames to the desired order. (since R2024b)

Before R2024b: This property is read-only. Adjust your code such that calls to the predict function and similar has input arguments that match the order specified by InputNames.

Note

If you customize the InputNames property and then make edits to the neural network, for example, by adding or removing layers, then the InputNames property does not change. This behavior means that if you add or remove layers that correspond to network inputs, then you must also update the InputNames property manually.

Data Types: cell

OutputNames — Names of network outputs

cell array of character vectors

Names of the network outputs, specified as a cell array of character vectors.

For layers with a single output, the output name is the name of the layer. For layers with multiple outputs, the output name is "layerName/outputName", where layerName is the name of the layer and outputName is the name of the layer output.

If you do not specify the output names, then when you create the network, the software sets the OutputNames property to the layers with unconnected outputs.

For networks with multiple outputs, training and prediction functions use this property to determine the order of the outputs. For example, the outputs Y1,...,YN of the predict function correspond to the outputs specified by the OutputNames property of the network.

Note

If you customize the OutputNames property and then make edits to the neural network, for example, by adding or removing layers, then the OutputNames property does not change. This behavior means that if you add or remove layers that correspond to network outputs, then you must also update the OutputNames property manually.

Data Types: cell

Initialized — Flag for initialized network

0 (false) | 1 (true)

This property is read-only.

Flag for initialized network, specified as one of these values:

• 1 (true) — Network is initialized and is ready for prediction and custom training loops. If you change the values of the learnable or state parameters, then the network remains initialized.

• 0 (false) — Network is not initialized and is not ready prediction or custom training loops. To initialize an uninitialized network, use the initialize function.

Data Types: logical

Object Functions

addInputLayer	Add input layer to network
addLayers	Add layers to neural network
removeLayers	Remove layers from neural network
connectLayers	Connect layers in neural network
disconnectLayers	Disconnect layers in neural network
replaceLayer	Replace layer in neural network
getLayer	Look up a layer by name or path
expandLayers	Expand network layers
groupLayers	Group layers into network layers
summary	Print network summary
plot	Plot neural network architecture
initialize	Initialize learnable and state parameters of a dlnetwork
predict	Compute deep learning network output for inference
forward	Compute deep learning network output for training
resetState	Reset state parameters of neural network
setL2Factor	Set L2 regularization factor of layer learnable parameter
setLearnRateFactor	Set learn rate factor of layer learnable parameter
getLearnRateFactor	Get learn rate factor of layer learnable parameter
getL2Factor	Get L2 regularization factor of layer learnable parameter

Examples

Create Neural Network from Scratch

Define a two-output neural network that predicts both categorical labels and numeric values given 2-D images as input.

Specify the number of classes and responses.

numClasses = 10;
numResponses = 1;

Create an empty neural network.

net = dlnetwork;

Define the layers of the main branch of the network and the softmax output.

layers = [
    imageInputLayer([28 28 1],Normalization="none")

    convolution2dLayer(5,16,Padding="same")
    batchNormalizationLayer
    reluLayer(Name="relu_1")

    convolution2dLayer(3,32,Padding="same",Stride=2)
    batchNormalizationLayer
    reluLayer
    convolution2dLayer(3,32,Padding="same")
    batchNormalizationLayer
    reluLayer

    additionLayer(2,Name="add")

    fullyConnectedLayer(numClasses)
    softmaxLayer(Name="softmax")];

net = addLayers(net,layers);

Add the skip connection.

layers = [
    convolution2dLayer(1,32,Stride=2,Name="conv_skip")
    batchNormalizationLayer
    reluLayer(Name="relu_skip")];

net = addLayers(net,layers);
net = connectLayers(net,"relu_1","conv_skip");
net = connectLayers(net,"relu_skip","add/in2");

Add the fully connected layer for the regression output.

layers = fullyConnectedLayer(numResponses,Name="fc_2");
net = addLayers(net,layers);
net = connectLayers(net,"add","fc_2");

View the neural network in a plot.

figure
plot(net)

Convert Layer Array to Neural Network

If you have a layer that defines a complete single-input neural network, has layers arranged in series, and has an input layer, then you can convert the layer array to a dlnetwork object directly.

Specify an LSTM network as a layer array.

layers = [
    sequenceInputLayer(12)
    lstmLayer(100)
    fullyConnectedLayer(9)
    softmaxLayer];

Convert the layer array to a dlnetwork object. Because the layer array has an input layer and no other inputs, the software initializes the neural network.

net = dlnetwork(layers)

net = 
  dlnetwork with properties:

         Layers: [4x1 nnet.cnn.layer.Layer]
    Connections: [3x2 table]
     Learnables: [5x3 table]
          State: [2x3 table]
     InputNames: {'sequenceinput'}
    OutputNames: {'softmax'}
    Initialized: 1

  View summary with summary.

Freeze Learnable Parameters

Load a pretrained network.

net = imagePretrainedNetwork;

The Learnables property of the dlnetwork object is a table that contains the learnable parameters of the network. The table includes parameters of nested layers in separate rows. View the first few rows of the learnables table.

learnables = net.Learnables;
head(learnables)

          Layer           Parameter           Value       
    __________________    _________    ___________________

    "conv1"               "Weights"    {3x3x3x64  dlarray}
    "conv1"               "Bias"       {1x1x64    dlarray}
    "fire2-squeeze1x1"    "Weights"    {1x1x64x16 dlarray}
    "fire2-squeeze1x1"    "Bias"       {1x1x16    dlarray}
    "fire2-expand1x1"     "Weights"    {1x1x16x64 dlarray}
    "fire2-expand1x1"     "Bias"       {1x1x64    dlarray}
    "fire2-expand3x3"     "Weights"    {3x3x16x64 dlarray}
    "fire2-expand3x3"     "Bias"       {1x1x64    dlarray}

To freeze the learnable parameters of the network, loop over the learnable parameters and set the learn rate to 0 using the setLearnRateFactor function.

factor = 0;

numLearnables = size(learnables,1);
for i = 1:numLearnables
    layerName = learnables.Layer(i);
    parameterName = learnables.Parameter(i);
    
    net = setLearnRateFactor(net,layerName,parameterName,factor);
end

To use the updated learn rate factors when training, you must pass the dlnetwork object to the update function in the custom training loop. For example, use the command

[net,velocity] = sgdmupdate(net,gradients,velocity);

Extended Capabilities

C/C++ Code Generation

Generate C and C++ code using MATLAB® Coder™.

Usage notes and limitations:

• Code generation supports only the InputNames and OutputNames properties.

• The Initialized property of the dlnetwork object must be 1 (true).

• Code generation supports tuning the variable Value of the State property. Code generation does not support modifying variables Layer and Parameter of the State property.

• Code generation supports these functions for the State property:

  • height

  • isscalar

  • isempty

  • istable

  • istabular

  • ndims

  • numel

  • size

  • width

• For Simulink simulation, code generation does not support extracting and updating the State of a dlnetwork in a MATLAB Function Block. Instead, use a Stateful Predict or a Stateful Classify block.

• Code generation supports multi-input multi-output dlnetwork objects with heterogeneous input layers.

  • For recurrent neural networks, multiple inputs are not supported.

  • For Intel^® MKL-DNN, all input layers must be sequence input layers.

  • For ARM^® Compute, the dlnetwork can have sequence and non-sequence input layers.

• Code generation supports dlarray inputs with these limitations for data formats.

  • For generic C/C++ that does not depend on any third-party libraries, the dlnetwork can have input layers with any number of spatial dimensions.

  • For ARM Compute and Intel MKL-DNN, code generation supports dlarray objects containing zero or two spatial dimensions. For example, code generation supports

    • Vector sequences that have "CT" or "CBT" data formats

    • Image sequences that have "SSCT" or "SSCBT" data formats

  • For code generation, only the "T" (time) dimension can be variable-sized, all other dimensions must have fixed sizes.

• Code generation supports only the predict object function. The dlarray input to the predict method must be a single datatype.

• To create a dlnetwork object for code generation, see Load Pretrained Networks for Code Generation (MATLAB Coder).

GPU Code Generation

Generate CUDA® code for NVIDIA® GPUs using GPU Coder™.

Usage notes and limitations:

• Code generation supports only the InputNames and OutputNames properties.

• For code generation, the Initialized property of the dlnetwork object must be true.

• Code generation supports tuning the variable Value of the State property. Code generation does not support modifying variables Layer and Parameter of the State property.

• Code generation supports these functions for the State property:

  • height

  • isscalar

  • isempty

  • istable

  • istabular

  • ndims

  • numel

  • size

  • width

• For Simulink simulation, code generation does not support extracting and updating the State of a dlnetwork in a MATLAB Function Block. Instead, use a Stateful Predict or a Stateful Classify block.

• Code generation supports multi-input multi-output dlnetwork objects with heterogeneous input layers. For recurrent neural networks, multiple inputs are not supported.

• Code generation supports dlnetwork objects for cuDNN and TensorRT targets.

• Code generation supports dlarray inputs with these limitations for data formats.

  • For plain CUDA that does not depend on any third-party libraries, the dlnetwork can have input layers with any number of spatial dimensions.

  • For cuDNN and TensorRT targets, code generation supports dlarray objects containing zero or two spatial dimensions. For example, code generation supports

    • Vector sequences that have "CT" or "CBT" data formats

    • Image sequences that have "SSCT" or "SSCBT" data formats

  • For code generation, only the "T" (time) dimension can be variable-sized, all other dimensions must have fixed sizes.

• Code generation supports only the predict object function. You must pass single data type dlarray input to the predict method.

• When targeting TensorRT with INT8 precision, the last layer(s) of the network must be a softmaxLayer layer.

• To create a dlnetwork object for code generation, see Load Pretrained Networks for Code Generation (GPU Coder).

Version History

Introduced in R2019b

R2024b: Convert Statistics and Machine Learning Toolbox machine learning models to dlnetwork

Convert these objects to dlnetwork objects:

• ClassificationNeuralNetwork (Statistics and Machine Learning Toolbox)

• RegressionNeuralNetwork (Statistics and Machine Learning Toolbox)

• CompactClassificationNeuralNetwork (Statistics and Machine Learning Toolbox)

• RegressionNeuralNetwork (Statistics and Machine Learning Toolbox)

R2024b: Customize order of neural network inputs

Customize the order of neural network inputs by setting the InputNames property.

R2024a: Create empty neural network

Create an empty neural network using the dlnetwork function with no input arguments. Use empty neural networks as a starting point for building neural networks from scratch.

R2024a: Complex-valued learnables and state

The values in the Learnables and State properties can be complex-valued.

R2024a: Layer graphs are not recommended

Starting in R2024a, LayerGraph objects are not recommended, use dlnetwork objects instead.

This recommendation means that these syntaxes are also not recommend for layer graph input:

• net = dlnetwork(lgraph)

• net = dlnetwork(lgraph,X1,...,XN)

• net = dlnetwork(__,Initialize=tf)

• net = dlnetwork(__,OutputNames=names)

To specify a neural network with a graph structure, create a dlnetwork object and add and connect layers using the addLayers and connectLayers functions, respectively. To update your code, in most cases, you can use these replacements:

Layer Graph Syntax	Replacement
lgraph = layerGraph;	net = dlnetwork;
lgraph = layerGraph(layers);	net = dlnetwork(layers,Initialize=false);

Most LayerGraph object functions such as addLayers and connectLayers also support dlnetwork and should not need updating.

Depending on the neural network architecture, the resulting dlnetwork object can be uninitialized. Neural network prediction and custom training loops require initialized neural networks. To initialize an uninitialized neural network, use the initialize function.

Existing code using LayerGraph objects still runs. You can still convert LayerGraph objects to dlnetwork objects by using net = dlnetwork(lgraph) and specifying the Initialize or X1,...,XN arguments if necessary.

R2023b: Initialize networks containing input layers with unset normalization statistics

Input layers such as imageInputLayer and sequenceInputLayer contain properties that networks use for data normalization. These properties are Mean, StandardDeviation, Min, and Max. The software uses these properties to apply the data normalization method defined by the Normalization property of the layer.

Starting in R2023b, when you initialize a network by creating an initialized dlnetwork or by using the initialize function, the software initializes the Mean, StandardDeviation, Min, and Max properties of input layers if you do not set them when you create the layer and if the normalization method requires them. For normalization methods that use two properties, for example, zscore, the software initializes those properties only if you do not set either property when you create the layer.

• For zerocenter normalization, Mean is initialized to 0.

• For zscore normalization, Mean is initialized to 0 and StandardDeviation is initialized to 1.

• For rescale-symmetric normalization, Min is initialized to -1 and Max is initialized to 1.

• For rescale-zero-one normalization, Min is initialized to 0 and Max is initialized to 1.

By default, the software automatically calculates the normalization statistics during training. To customize the normalization, set the Mean, StandardDeviation, Min, and Max properties of input layers manually.

In previous releases, the software errors when you initialize a network containing an input layer that uses a normalization method requiring properties that you do not specify when you create the layer.

R2021a: dlnetwork state values are dlarray objects

The State of a dlnetwork object is a table containing the state parameter names and values for each layer in the network.

Starting in R2021a, the state values are dlarray objects. This change enables better support when using AcceleratedFunction objects. To accelerate deep learning functions that have frequently changing input values, for example, an input containing the network state, the frequently changing values must be specified as dlarray objects.

In previous versions, the state values are numeric arrays.

In most cases, you will not need to update your code. If you have code that requires the state values to be numeric arrays, then to reproduce the previous behavior, extract the data from the state values manually using the extractdata function with the dlupdate function.

state = dlupdate(@extractdata,net.State);