Adam

class empulse.optimizers.Adam(lr=0.001, beta1=0.9, beta2=0.999, eps=1e-08, amsgrad=False, lr_schedule=None, alpha_schedule=None, batch_size=None, random_state=None, max_iter=1000, tolerance=1e-06, patience=20)

Adam optimizer with optional AMSGrad correction.

Combines momentum (first moment) with adaptive per-parameter learning rates (second moment):

\[\begin{split}m_t &= \beta_1 m_{t-1} + (1 - \beta_1) g_t \\ v_t &= \beta_2 v_{t-1} + (1 - \beta_2) g_t^2 \\ \hat m_t &= m_t / (1 - \beta_1^t) \\ \hat v_t &= v_t / (1 - \beta_2^t) \\ w_{t+1} &= w_t - \text{lr} \cdot \hat m_t / (\sqrt{\hat v_t} + \varepsilon)\end{split}\]

When amsgrad=True the maximum of all past \(\hat v_t\) is used instead of \(\hat v_t\) (AMSGrad variant).

Parameters:

lrfloat, default=0.001

Learning rate (used when no lr_schedule is given).

beta1float, default=0.9

Exponential decay rate for the first moment.

beta2float, default=0.999

Exponential decay rate for the second moment.

epsfloat, default=1e-8

Numerical stability term added to the denominator.

amsgradbool, default=False

If True, use the AMSGrad variant of Adam.

lr_scheduleBaseSchedule, optional

If given, overrides the constant lr each step. schedule(t) receives the 0-based step index and returns a float.

alpha_scheduleBaseSchedule, optional

If given, calls objective.set_alpha(schedule(t)) before each gradient computation. Has no effect on objectives that do not expose set_alpha.

batch_sizeint, optional

Number of samples per gradient step. None (default) uses all samples. The objective must support with_indices for mini-batching to work.

random_stateint or numpy.random.Generator, optional

Seed or random number generator used for mini-batch shuffling.

max_iterint, default=1000

Maximum number of gradient steps.

tolerancefloat, default=1e-6

Convergence tolerance.

patienceint, default=20

Early-stopping patience (loss plateau window).

Examples

from empulse.models import CSLogitClassifier
from empulse.optimizers import Adam, CosineAnnealingSchedule, LinearSchedule

# Cosine-anneal learning rate and linearly grow alpha
lr_schedule = CosineAnnealingSchedule(max_value=1e-2, min_value=1e-5, t_max=500)
alpha_schedule = LinearSchedule(start_value=0.5, end_value=20.0, n_steps=200)
model = CSLogitClassifier(
    optimizer=Adam(lr=1e-2, lr_schedule=lr_schedule, alpha_schedule=alpha_schedule)
)

__call__(objective, X, **kwargs)

Run the optimization and return an OptimizeResult.