neuralODELayer

Neural ODE layer

Since R2023b

expand all in page

    Description

    A neural ODE layer outputs the solution of an ODE.

    Creation

    Syntax

    layer = neuralODELayer(net,tspan)
    layer = neuralODELayer(net,tspan,Name=Value)

    Description

    layer = neuralODELayer(net,tspan) creates a neural ODE layer and sets the Network and TimeInterval properties.

    example

    layer = neuralODELayer(net,tspan,Name=Value) specifies additional properties using one or more name-value arguments.

    example

    Properties

    expand all

    Neural network characterizing neural ODE function, specified as a dlnetwork object.

    If Network has one input, then predict(net,Y) defines the ODE system, where net is the network. If Network has two inputs, then predict(net,T,Y) defines the ODE system, where T is a time step repeated over the batch dimension.

    The network size and format of the network inputs and outputs must match.

    When GradientMode is "adjoint", the network State property must be empty. To use a network with a nonempty State property, set GradientMode to "direct".

    Interval of integration, specified as a numeric vector with two or more elements. The elements in TimeInterval must be all increasing or all decreasing.

    The solver imposes the initial conditions given by Y0 at the initial time TimeInterval(1), then integrates the ODE function from TimeInterval(1) to TimeInterval(end).

    • If TimeInterval has two elements, [t0 tf], then the solver returns the solution evaluated at point tf.

    • If TimeInterval has more than two elements, [t0 t1 ... tf], then the solver returns the solution evaluated at the given points [t1 ... tf]. The solver does not step precisely to each point specified in TimeInterval. Instead, the solver uses its own internal steps to compute the solution, then evaluates the solution at the points specified in TimeInterval. The solutions produced at the specified points are of the same order of accuracy as the solutions computed at each internal step.

      Specifying several intermediate points has little effect on the efficiency of computation, but for large systems it can negatively affect memory management.

    Since R2025a

    Solver for neural ODE operation, specified as one of these values:

    • "ode45" — Solver for nonstiff differential equations using explicit Runge-Kutta (4,5) formula, the Dormand-Prince pair. The "ode45" solver is well suited for most tasks and can be faster and more accurate than other solvers.

    • "ode1" — Solver for nonstiff differential equations using Euler method. The "ode1" solver uses a fixed step size and can be better suited for code generation and Simulink tasks.

    If Solver is "ode1", then the RelativeTolerance and AbsoluteTolerance properties have no effect.

    If you specify the SolverOptions property, then the default value matches the corresponding solver. Otherwise, the default value is "ode45".

    Since R2025a

    Solver options object, specified as a deep.ode.options.ODE45 or a deep.ode.options.ODE1 object.

    To set the solver options of the layer, use dot notation. For example, to set the relative tolerance to 1e-4, use layer.SolverOptions.RelativeTolerance = 1e-4, where layer is an instance of the neural ODE layer.

    In most cases, you do not need to create the options object directly. If you specify the SolverOptions property, then you must not specify the RelativeTolerance, AbsoluteTolerance, and GradientMode properties.

    To see which options the "ode45" and "ode1" solvers support, see deep.ode.options.ODE45 and deep.ode.options.ODE1, respectively.

    Method to compute gradients with respect to the initial conditions and parameters when using the dlgradient function, specified as one of these values:

    • "direct" — Compute gradients by backpropagating through the operations undertaken by the numerical solver. This option best suits large mini-batch sizes or when TimeInterval contains many values.

    • "adjoint" — Compute gradients by solving the associated adjoint ODE system. This option best suits small mini-batch sizes or when TimeInterval contains a small number of values.

    Tip

    To customize the neural ODE solver options, use the Solver and SolverOptions properties. These properties are recommended because they provide additional control over the neural ODE solver.

    When GradientMode is "adjoint", the network State property must be empty. To use a network with a nonempty State property, set GradientMode to "direct".

    The dlaccelerate function does not support accelerating forward passes of networks with neural ODE layers with the GradientMode property set to "direct". To accelerate forward passes of networks with neural ODE layers, set the GradientMode property to "adjoint" or "adjoint-seminorm", or accelerate parts of your code that do not perform forward passes of the network.

    Warning

    When GradientMode is "adjoint", all layers in Network must support acceleration. Otherwise, the software can return unexpected results.

    When GradientMode is "adjoint", the software traces the ODE function input to determine the computation graph used for automatic differentiation. This tracing process can take some time and can end up recomputing the same trace. By optimizing, caching, and reusing the traces, the software can speed up the gradient computation.

    For more information on deep learning function acceleration, see Deep Learning Function Acceleration for Custom Training Loops.

    The NeuralODELayer object stores this property as a character vector.

    Relative error tolerance, specified as a positive scalar. This tolerance measures the error relative to the magnitude of each solution component. Roughly speaking, it controls the number of correct digits in all solution components, except those smaller than the absolute tolerance AbsoluteTolerance.

    At each step, the ODE solver estimates the local error e in the ith component of the solution. To be successful, the step must have acceptable error, as determined by both the relative and absolute error tolerances:

    |e(i)| <= max(RelativeTolerance*abs(y(i)),AbsoluteTolerance(i))

    If Solver is "ode1", then the RelativeTolerance property has no effect.

    Tip

    To customize the neural ODE solver options, use the Solver and SolverOptions properties. These properties are recommended because they provide additional control over the neural ODE solver.

    Data Types: single | double

    Absolute error tolerance, specified as a positive scalar or vector. This tolerance is a threshold below which the value of the solution becomes unimportant. If the solution |y| is smaller than AbsoluteTolerance, then the solver does not need to obtain any correct digits in |y|. For this reason, the value of AbsoluteTolerance should take into account the scale of the solution components.

    If AbsoluteTolerance is a vector, then it must be the same length as the solution. If AbsoluteTolerance is a scalar, then the value applies to all solution components.

    At each step, the ODE solver estimates the local error e in the ith component of the solution. To be successful, the step must have acceptable error, as determined by both the relative and absolute error tolerances:

    |e(i)| <= max(RelativeTolerance*abs(y(i)),AbsoluteTolerance(i))

    If Solver is "ode1", then the AbsoluteTolerance property has no effect.

    Tip

    To customize the neural ODE solver options, use the Solver and SolverOptions properties. These properties are recommended because they provide additional control over the neural ODE solver.

    Data Types: single | double

    Examples

    collapse all

    Open Live Script

    Create a neural ODE layer. Specify an ODE network containing a convolution layer followed by a tanh layer. Specify a time interval of [0, 1].

    inputSize = [14 14 8];

layersODE = [
    imageInputLayer(inputSize)
    convolution2dLayer(3,8,Padding="same")
    tanhLayer];

netODE = dlnetwork(layersODE);

tspan = [0 1];
layer = neuralODELayer(netODE,tspan)
    layer = 
  NeuralODELayer with properties:

                 Name: ''
         TimeInterval: [0 1]

   Learnable Parameters
              Network: [1×1 dlnetwork]

   Solver properties
         GradientMode: 'direct'
    RelativeTolerance: 1.0000e-03
    AbsoluteTolerance: 1.0000e-06
               Solver: ode45

  Show all properties

    Create a neural network containing a neural ODE layer.

    layers = [
    imageInputLayer([28 28 1])
    convolution2dLayer([3 3],8,Padding="same",Stride=2)
    reluLayer
    neuralODELayer(netODE,tspan)
    fullyConnectedLayer(10)
    softmaxLayer];

net = dlnetwork(layers)
    net = 
  dlnetwork with properties:

         Layers: [6×1 nnet.cnn.layer.Layer]
    Connections: [5×2 table]
     Learnables: [6×3 table]
          State: [0×3 table]
     InputNames: {'imageinput'}
    OutputNames: {'softmax'}
    Initialized: 1

  View summary with summary.

    Since R2025a

    Open Live Script

    Create a neural ODE layer. Specify an ODE network containing a convolution layer followed by a tanh layer. Specify a time interval of [0, 1].

    inputSize = [14 14 8];

layersODE = [
    imageInputLayer(inputSize)
    convolution2dLayer(3,8,Padding="same")
    tanhLayer];

netODE = dlnetwork(layersODE);

tspan = [0 1];
layer = neuralODELayer(netODE,tspan)
    layer = 
  NeuralODELayer with properties:

                 Name: ''
         TimeInterval: [0 1]

   Learnable Parameters
              Network: [1×1 dlnetwork]

   Solver properties
         GradientMode: 'direct'
    RelativeTolerance: 1.0000e-03
    AbsoluteTolerance: 1.0000e-06
               Solver: ode45

  Show all properties

    Specify an initial step size of 1e-3 and a maximum step size of 1e-2.

    layer.SolverOptions.InitialStep = 1e-3;
layer.SolverOptions.MaxStep = 1e-2;

    View the solver options.

    layer.SolverOptions
    ans = 
  ODE45 with properties:

          InitialStep: 1.0000e-03
              MaxStep: 0.0100
    RelativeTolerance: 1.0000e-03
    AbsoluteTolerance: 1.0000e-06
         GradientMode: 'direct'

    Tips

    • To apply the neural ODE operation in deep learning models defined as functions or in custom layer functions, use dlode45.

    Algorithms

    expand all

    Version History

    Introduced in R2023b

    expand all

    See Also

    | | | |

    Topics