2.2. Working with metric objects#

Pairing a CostMatrix with a MetricStrategy produces a Metric. It behaves like an ordinary scoring function, but it carries the cost matrix with it, which lets it do several things a plain function cannot: derive an operating point, serve as a training objective, and tell you what it needs.

import numpy as np
from empulse.metrics import Cost, CostMatrix, Metric

matrix = (
    CostMatrix()
    .add_fp_cost('c_fp')
    .add_fn_cost('c_fn')
    .set_default(c_fp=1.0, c_fn=5.0)
)
expected_cost = Metric(matrix, Cost())

y_true = np.array([0, 1, 0, 1, 0, 1, 0, 1])
y_score = np.array([0.1, 0.2, 0.3, 0.4, 0.5, 0.7, 0.8, 0.9])

print(expected_cost(y_true, y_score))

2.2.1. Supplying parameters#

Any symbol in the cost matrix becomes a keyword argument. Empulse resolves each one in order: an explicit keyword argument wins, otherwise the value from set_default is used, and if neither exists the call fails naming the missing parameter.

A symbol can be supplied under its own name or under any alias registered for it — but not both. Passing both raises a ValueError rather than letting whichever came last silently win.

Scalars apply to every row; arrays supply one value per row, and must match the length of y_true. A mismatch raises a ValueError naming the parameter, rather than surfacing later as an unrelated broadcasting error.

A keyword the metric does not recognise raises a warning and is ignored, so a typo in a parameter name does not silently fall back to a default.

2.2.2. Finding the operating point#

Because the metric knows the cost matrix, it can say where the break-even point between acting and not acting lies.

print(expected_cost.optimal_threshold(y_true, y_score))
print(expected_cost.optimal_rate(y_true, y_score))

optimal_threshold returns a score cut-off; optimal_rate returns the fraction of the population to act on. They are two views of the same decision, and classification_threshold converts between them. Threshold Tuning covers when each is the more useful form.

With instance-dependent costs, optimal_threshold returns an array: each row has its own break-even point, because each row has its own costs.

per_row_costs = np.array([1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0])
print(expected_cost.optimal_threshold(y_true, y_score, c_fn=per_row_costs))

2.2.3. Using a metric as a scikit-learn scorer#

A metric is callable with the (y_true, y_score, **kwargs) signature that make_scorer expects, so it wraps directly:

from sklearn.metrics import make_scorer

scorer = make_scorer(
    expected_cost,
    response_method='predict_proba',
    greater_is_better=False,
)

Set greater_is_better to match the strategy: False for Cost and LogCost, True for the rest. Rather than hard-coding it, read it off the metric:

greater_is_better = expected_cost.direction.name == 'MAXIMIZE'
print(greater_is_better)

Every metric also has a settable __name__, which is what appears in GridSearchCV results and scorer error messages:

expected_cost.__name__ = 'campaign_cost'
print(expected_cost.__name__)

Metrics are picklable, including ones built from symbolic and stochastic cost matrices, so n_jobs > 1 works without rebuilding them per worker.

2.2.4. The prebuilt metrics are metric objects#

empc_score, mpc_score, expected_cost_loss and the rest are not functions — they are Metric (or MixtureMetric) instances built from the standard cost matrix of their domain. Everything on this page therefore applies to them too:

from empulse.metrics import empc_score

print(empc_score(y_true, y_score, clv=200))
print(empc_score.optimal_rate(y_true, y_score, clv=200))

They can also be passed straight to a model as loss, so the metric you report is the metric you train on.

2.2.5. Mixtures of metrics#

Some domain measures assume a parameter whose distribution mixes point masses with a continuous piece — the recovery fraction on a defaulted loan is 0 with some probability, 1 with some probability, and spread over the interval otherwise. sympy.stats cannot express that as a single random variable.

MixtureMetric sidesteps it. Each MixtureComponent pairs a weight with an ordinary metric and any parameter values fixed for that piece, and the mixture is their weighted sum. Worked cost matrices has a worked example.

This is exact, not an approximation. The score, its gradients, and the optimal predicted-positive rate are all linear functionals of the assumed density, and integration and differentiation are linear, so a mixture’s value is exactly the weighted sum of its components’ values.

optimal_threshold is the one exception, because a threshold is a non-linear function of a rate: combining the components’ thresholds would be wrong. MixtureMetric therefore combines the rates first and converts the result to a threshold once.

A mixture behaves like a plain metric everywhere else. It satisfies the same BaseMetric interface, so it can be a model’s loss, and its direction and strategy are read from its components — which must agree, or accessing them raises.

2.2.6. Where next#