RMSProp

class empulse.optimizers.RMSProp(lr=0.01, alpha=0.99, eps=1e-08, momentum=0.0, lr_schedule=None, alpha_schedule=None, batch_size=None, random_state=None, max_iter=1000, tolerance=1e-06, patience=20)

RMSProp optimizer.

Divides the learning rate by a running average of recent gradient magnitudes:

\[\begin{split}v_t &= \alpha \cdot v_{t-1} + (1 - \alpha) \cdot g_t^2 \\ w_{t+1} &= w_t - \frac{\text{lr}}{\sqrt{v_t} + \varepsilon} \cdot g_t\end{split}\]

When momentum > 0, a momentum buffer is added:

\[\begin{split}b_t &= \text{momentum} \cdot b_{t-1} + \frac{\text{lr}}{\sqrt{v_t} + \varepsilon} \cdot g_t \\ w_{t+1} &= w_t - b_t\end{split}\]

Parameters:

lrfloat, default=0.01

Learning rate (used when no lr_schedule is given).

alphafloat, default=0.99

Smoothing constant for the squared-gradient running average.

epsfloat, default=1e-8

Term added to the denominator for numerical stability.

momentumfloat, default=0.0

Momentum factor. 0.0 disables momentum.

lr_scheduleBaseSchedule, optional

If given, overrides the constant lr each step.

alpha_scheduleBaseSchedule, optional

If given, calls objective.set_alpha(schedule(t)) before each gradient computation.

batch_sizeint, optional

Number of samples per gradient step. None uses all samples.

random_stateint or numpy.random.Generator, optional

Seed or random number generator for mini-batch shuffling.

max_iterint, default=1000

Maximum number of gradient steps.

tolerancefloat, default=1e-6

Convergence tolerance.

patienceint, default=20

Early-stopping patience (loss plateau window).

Examples

from empulse.models import CSLogitClassifier
from empulse.optimizers import RMSProp

model = CSLogitClassifier(optimizer=RMSProp(lr=0.005, alpha=0.9))

__call__(objective, X, **kwargs)

Run the optimization and return an OptimizeResult.