SGD#

class empulse.optimizers.SGD(lr=0.01, momentum=0.0, nesterov=False, dampening=0.0, lr_schedule=None, alpha_schedule=None, batch_size=None, random_state=None, max_iter=1000, tolerance=1e-06, patience=20)[source]#

Stochastic Gradient Descent with optional Nesterov momentum.

Although called “stochastic”, this optimizer operates on the full dataset (as required by cost-sensitive logit objectives) and is therefore a full-batch gradient descent with SGD-style update rules.

The update rule without momentum is simply:

\[w_{t+1} = w_t - \text{lr} \cdot g_t\]

With momentum (momentum > 0):

\[\begin{split}v_t &= \text{momentum} \cdot v_{t-1} + (1 - \text{dampening}) \cdot g_t \\ w_{t+1} &= w_t - \text{lr} \cdot v_t\end{split}\]

With Nesterov momentum (nesterov=True):

\[w_{t+1} = w_t - \text{lr} \cdot (g_t + \text{momentum} \cdot v_t)\]
Parameters:
lrfloat, default=0.01

Learning rate (used when no lr_schedule is given).

momentumfloat, default=0.0

Momentum factor. 0.0 disables momentum.

nesterovbool, default=False

If True, use Nesterov momentum (requires momentum > 0).

dampeningfloat, default=0.0

Dampening applied to the velocity update (only used when momentum > 0 and nesterov=False).

lr_scheduleBaseSchedule, optional

If given, overrides the constant lr each step. schedule(t) receives the 0-based step index and returns a float.

alpha_scheduleBaseSchedule, optional

If given, calls objective.set_alpha(schedule(t)) before each gradient computation. Has no effect on objectives that do not expose set_alpha.

batch_sizeint, optional

Number of samples per gradient step. None (default) uses all samples. The objective must support with_indices for mini-batching to work.

random_stateint or numpy.random.Generator, optional

Seed or random number generator used for mini-batch shuffling.

max_iterint, default=1000

Maximum number of gradient steps.

tolerancefloat, default=1e-6

Convergence tolerance on gradient infinity-norm and loss plateau.

patienceint, default=20

Number of consecutive steps with loss improvement smaller than tolerance before declaring convergence.

Examples

from empulse.models import CSLogitClassifier
from empulse.optimizers import SGD, ExponentialSchedule

lr_schedule = ExponentialSchedule(start_value=0.05, gamma=0.99, min_value=1e-5)
model = CSLogitClassifier(optimizer=SGD(lr=0.05, momentum=0.9, nesterov=True,
                                        lr_schedule=lr_schedule))
__call__(objective, X, **kwargs)#

Run the optimization and return an OptimizeResult.

On this page