exponentialLearnRate

Exponential learning rate schedule

Since R2024b

Description

An exponential learning rate schedule decays the learning rate by a specified factor.

Tip

To easily use an exponential learning rate schedule with the default options, specify the LearnRateSchedule argument of the trainingOptions function as "exponential".

Creation

Description

schedule = exponentialLearnRate creates an exponentialLearnRate object.

This syntax is equivalent to setting the LearnRateSchedule argument of the trainingOptions function to "exponential".

schedule = exponentialLearnRate(Name=Value) specifies optional properties using one or more name-value arguments. For example, DropFactor=0.5 specifies halving of the learning rate.

Properties

DropFactor — Scaling factor used to drop learning rate

0.1 (default) | scalar in the range (0,1)

Scaling factor used to drop the learning rate, specified as a scalar in the range (0,1).

At each step, the software scales the learning rate using a factor of DropFactor^(1/Period), that is, by DropFactor after Period steps.

If FrequencyUnit is "epoch", then the software updates the learning rate every epoch and each iteration of the epoch uses the same learning rate. If FrequencyUnit is "iteration", then the software updates the learning rate every iteration.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

Period — Number of steps before scaling learning rate by DropFactor

1 (default) | positive integer

Number of steps before scaling the learning rate by DropFactor, specified as a positive integer.

At each step, the software scales the learning rate using a factor of DropFactor^(1/Period), that is, by DropFactor after Period steps.

If FrequencyUnit is "epoch", then the software updates the learning rate every epoch and each iteration of the epoch uses the same learning rate. If FrequencyUnit is "iteration", then the software updates the learning rate every iteration.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

FrequencyUnit — Frequency unit

"epoch" (default) | "iteration"

Frequency unit, specified as "epoch" or "iteration".

If FrequencyUnit is "epoch", then the software updates the learning rate every epoch and each iteration of the epoch uses the same learning rate. If FrequencyUnit is "iteration", then the software updates the learning rate every iteration.

NumSteps — Number of steps to drop learning rate

Inf (default)

This property is read-only.

Number of steps to drop learning rate, specified as Inf.

exponentialLearnRate objects are infinite learning rate schedules, so NumSteps is Inf.

Data Types: double

Examples

Create Exponential Learning Rate Schedule

Create an exponential learning rate schedule with the default settings.

schedule = exponentialLearnRate

schedule = 
  exponentialLearnRate with properties:

       DropFactor: 0.1000
           Period: 10
    FrequencyUnit: "epoch"
         NumSteps: Inf

Specify this schedule as a training option.

options = trainingOptions("adam",LearnRateSchedule=schedule);

As an alternative to creating an exponential learning rate schedule object, to easily use an exponential learning rate schedule with the default options, specify the LearnRateSchedule argument of the trainingOptions function as "exponential".

options = trainingOptions("adam",LearnRateSchedule="exponential");

Customize Exponential Learning Rate Schedule

Create an exponential learning rate schedule that drops the learning rate by a factor of 2 every 100 iterations.

schedule = exponentialLearnRate( ...
    DropFactor=0.5, ...
    Period=100, ...
    FrequencyUnit="iteration")

schedule = 
  exponentialLearnRate with properties:

       DropFactor: 0.5000
           Period: 100
    FrequencyUnit: "iteration"
         NumSteps: Inf

Specify this schedule as a training option.

options = trainingOptions("adam",LearnRateSchedule=schedule);

Algorithms

Exponential Learning Rate Schedule

An exponential learning rate schedule decays the learning rate by a specified factor.

The software uses this formula to calculate the learning rate:

$$\alpha ={\gamma }^{(k-1)/P}{\alpha }_0,$$

where:

• α₀ is the base learning rate, specified by the InitialLearnRate option of the trainingOptions function.

• γ is the scaling factor, specified by the DropFactor argument.

• P is the period, specified by the Period argument.

If FrequencyUnit is "iteration", then the variable k denotes the schedule iteration number. The value of k can be different from the training iteration number. For example, if you train using {"warmup",schedule}, where schedule is a exponentialLearnRate object with FrequencyUnit set to "iteration", then at training iteration 15, the exponentialLearnRate object uses k with a value of 10 because the warm-up learning rate schedule runs for 5 training iterations.

If FrequencyUnit is "epoch", then the variable k denotes the schedule epoch number. In this case, the schedule uses the same learning rate for each iteration of the epoch. The value of k can be different from the training epoch number. For example, if you train using {warmupLearnRate(FrequencyUnit="epoch"),schedule}, where schedule is a exponentialLearnRate object with FrequencyUnit set to "epoch", then at training epoch 15, the exponentialLearnRate object uses k with a value of 10 because the warm-up learning rate schedule processes the first 5 epochs.

This plot shows an overview of the exponential learning rate schedule with the default settings.

Version History

Introduced in R2024b