ElasticNetPenalty

class empulse.metrics.ElasticNetPenalty(C, l1_ratio, objective_scale, n_samples, start_coef)

Elastic-net penalty on the non-intercept coefficients of a logit objective.

The penalty added to the (sample-averaged) data loss is

\[\lambda \left( \rho \sum_j |w_j| + \frac{1 - \rho}{2} \sum_j w_j^2 \right)\]

with \(\rho\) the l1_ratio and \(\lambda = \mathrm{objective\_scale} / (C \cdot n\_samples)\).

Dividing by n_samples is what puts C on scikit-learn’s footing: scikit-learn adds an unnormalized penalty to a summed data loss, which is the same thing as adding 1 / (C * n_samples) to a mean one. Multiplying by objective_scale additionally makes the regularization path invariant to rescaling the cost matrix.

See also

objective_scale_from_costs : Computes the objective_scale term.

Parameters:

Cfloat

Inverse regularization strength; smaller means stronger regularization.

l1_ratiofloat

Elastic-net mixing parameter in [0, 1]. 0 is pure L2, 1 is pure L1.

objective_scalefloat

Magnitude of the objective’s per-sample training signal, from objective_scale_from_costs.

n_samplesint

Number of training samples the data loss is averaged over.

start_coefint

Index at which the penalized coefficients begin. 1 when an intercept is fitted, since the intercept is never penalized.

add_to(loss, gradient, weights)

Add the penalty to a data loss and its gradient in one pass.

Parameters:

lossfloat

Data loss.

gradientndarray

Data gradient; modified in place.

weightsndarray

Full coefficient vector, intercept included.

Returns:

lossfloat

Penalized loss.

gradientndarray

Penalized gradient.

classmethod from_costs(*, y_true, tp_benefit, tn_benefit, fp_cost, fn_cost, C, l1_ratio, fit_intercept, n_samples)

Build a penalty whose scale is derived from a cost matrix.

Parameters:

y_truendarray

Binary labels, recoded to 0/1.

tp_benefitfloat or ndarray

Benefit of true positives.

tn_benefitfloat or ndarray

Benefit of true negatives.

fp_costfloat or ndarray

Cost of false positives.

fn_costfloat or ndarray

Cost of false negatives.

Cfloat

Inverse regularization strength.

l1_ratiofloat

Elastic-net mixing parameter.

fit_interceptbool

Whether an (unpenalized) intercept is fitted.

n_samplesint

Number of training samples.

Returns:

penaltyElasticNetPenalty

The configured penalty.

classmethod from_scale(*, objective_scale, C, l1_ratio, fit_intercept, n_samples)

Build a penalty from an already-computed objective scale.

Parameters:

objective_scalefloat

Magnitude of the objective’s per-sample training signal.

Cfloat

Inverse regularization strength.

l1_ratiofloat

Elastic-net mixing parameter.

fit_interceptbool

Whether an (unpenalized) intercept is fitted.

n_samplesint

Number of training samples.

Returns:

penaltyElasticNetPenalty

The configured penalty.

gradient(weights)

Compute the penalty’s contribution to the gradient.

Parameters:

weightsndarray

Full coefficient vector, intercept included.

Returns:

gradientndarray of numpy.float64

Same shape as weights, zero in the unpenalized intercept slots. For l1_ratio > 0 this is a subgradient: it is zero at w_j = 0, where the penalty is not differentiable.

property is_active

Whether the penalty contributes anything to the objective.

property is_nonsmooth

Whether the penalty has a kink at zero, which a smooth solver cannot minimize.

property l1_weight

Weight of the L1 term, lambda_ * l1_ratio.

l2_gradient(coef)

Compute the gradient of the smooth (L2) half for already-sliced coefficients.

Parameters:

coefndarray

Penalized coefficients only, i.e. weights[start_coef:].

Returns:

gradientndarray of numpy.float64

L2 contribution, same shape as coef.

l2_value(coef)

Compute the smooth (L2) half of the penalty for already-sliced coefficients.

Used by the split-variable solver, which handles the L1 half separately as a linear term.

Parameters:

coefndarray

Penalized coefficients only, i.e. weights[start_coef:].

Returns:

valuefloat

L2 contribution.

property l2_weight

Weight of the L2 term, lambda_ * (1 - l1_ratio).

property lambda_

Overall penalty weight, objective_scale / (C * n_samples).

prox(weights, step)

Apply the proximal operator of the penalty to weights.

Soft-thresholds by step * l1_weight and then shrinks by 1 / (1 + step * l2_weight). This is the exact proximal step of a proximal-gradient method, and it is what makes such a method produce exact zeros.

Parameters:

weightsndarray

Full coefficient vector, intercept included.

stepfloat

Step size of the gradient step this prox follows.

Returns:

weightsndarray of numpy.float64

A new array; weights is left untouched. The intercept is passed through unchanged.

value(weights)

Compute the penalty’s contribution to the objective.

Parameters:

weightsndarray

Full coefficient vector, intercept included.

Returns:

valuefloat

Penalty value.

with_n_samples(n_samples)

Return a copy of this penalty rescaled to a different sample count.

Parameters:

n_samplesint

The new number of samples.

Returns:

penaltyElasticNetPenalty

A new penalty; this one is left untouched.