2.3. Tree-Based Cost-Sensitive Models

Empulse provides four tree-based classifiers that incorporate cost information directly into the learning process. Three of them (CSTreeClassifier, CSForestClassifier, and CSBaggingClassifier) use a cost-sensitive splitting criterion, so each tree node is grown by maximising cost reduction rather than a purity measure like Gini impurity. The fourth (ProfTreeClassifier) takes a different approach and uses an evolutionary genetic algorithm to evolve trees that directly maximise a user-defined profit metric.

Model	Base algorithm	Key characteristic
CSTreeClassifier	Decision tree	Single cost-sensitive tree; interpretable
CSForestClassifier	Random forest	Ensemble of cost-sensitive trees; feature importances
CSBaggingClassifier	Bagging / Pasting / Random Patches	Flexible ensemble; custom base estimator
ProfTreeClassifier	Genetic algorithm	Evolves trees directly against a profit metric

2.3.1. Quick Start

import numpy as np
from sklearn.datasets import make_classification
from empulse.models import (
    CSTreeClassifier,
    CSForestClassifier,
    CSBaggingClassifier,
    ProfTreeClassifier,
)

X, y = make_classification(n_samples=1_000, random_state=0)

# Cost-sensitive decision tree
tree = CSTreeClassifier(fp_cost=5, fn_cost=1)
tree.fit(X, y)

# Cost-sensitive random forest
forest = CSForestClassifier(n_estimators=100, fp_cost=5, fn_cost=1)
forest.fit(X, y)

# Cost-sensitive bagging
bagging = CSBaggingClassifier(n_estimators=10, fp_cost=5, fn_cost=1)
bagging.fit(X, y)

# Profit-maximising evolutionary tree
proftree = ProfTreeClassifier(tp_cost=300, fp_cost=10)
proftree.fit(X, y)

y_proba = forest.predict_proba(X)[:, 1]

2.3.2. Cost Matrix

All four models accept the same four cost terms:

• tp_cost — benefit / cost of a true positive

• tn_cost — benefit / cost of a true negative

• fp_cost — cost of a false positive

• fn_cost — cost of a false negative

2.3.2.1. Constant costs

Pass a scalar to apply the same cost to every sample:

from empulse.models import CSTreeClassifier

model = CSTreeClassifier(fp_cost=5, fn_cost=1, tp_cost=0, tn_cost=0)

2.3.2.2. Instance-dependent costs

Pass a 1-D array of length n_samples to fit to assign a unique cost to each individual observation:

import numpy as np
from sklearn.datasets import make_classification
from empulse.models import CSForestClassifier

X, y = make_classification(n_samples=500, random_state=0)
clv = np.random.default_rng(0).uniform(100, 1_000, size=len(y))
contact_cost = 10

model = CSForestClassifier(fn_cost=1)
model.fit(X, y, tp_cost=clv - contact_cost, fp_cost=contact_cost)

Note

Costs passed to fit take priority over costs passed to __init__. It is best practice to pass instance-dependent costs through fit rather than the constructor, because scikit-learn cloners do not carry sample arrays.

2.3.3. Cost-Sensitive Decision Tree (CSTreeClassifier)

CSTreeClassifier is a single decision tree whose splitting criterion directly maximises cost savings at each node [1]. It wraps scikit-learn’s DecisionTreeClassifier and exposes the same tree structure, pruning utilities, and feature importances.

2.3.3.1. Split criterion

The criterion parameter controls how the cost signal is weighted at each split:

criterion	Description
"cost" (default)	Pure cost reduction — splits are evaluated by the expected-cost gain.
"gini"	Cost gain is weighted by Gini impurity — blends class separation with cost.
"entropy" / "log_loss"	Cost gain is weighted by Shannon information gain.

from empulse.models import CSTreeClassifier

# Default: use the raw cost impurity
tree = CSTreeClassifier(fp_cost=5, fn_cost=1)

# Weight by Gini impurity
tree_gini = CSTreeClassifier(fp_cost=5, fn_cost=1, criterion="gini")

# Weight by entropy
tree_entropy = CSTreeClassifier(fp_cost=5, fn_cost=1, criterion="entropy")

2.3.3.2. Controlling tree size

Use the standard scikit-learn parameters to regularise the tree:

tree = CSTreeClassifier(
    fp_cost=5,
    fn_cost=1,
    max_depth=5,
    min_samples_leaf=20,
    min_samples_split=50,
)

2.3.3.2.1. Post-training pruning via ccp_alpha

Minimal Cost-Complexity Pruning is available through the ccp_alpha parameter. To find a good value, inspect the pruning path first:

from empulse.models import CSTreeClassifier
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=500, random_state=0)
tree = CSTreeClassifier(fp_cost=5, fn_cost=1).fit(X, y)

path = tree.cost_complexity_pruning_path(X, y)
print(path.ccp_alphas)    # candidate alpha values
print(path.impurities)    # impurity of the corresponding subtrees

# Apply pruning with a chosen alpha
pruned_tree = CSTreeClassifier(fp_cost=5, fn_cost=1, ccp_alpha=0.01).fit(X, y)

2.3.3.3. Inspecting the tree

CSTreeClassifier exposes the underlying sklearn tree object and several inspection helpers:

tree.fit(X, y)

print(tree.get_depth())          # maximum depth reached
print(tree.get_n_leaves())       # number of leaf nodes
print(tree.feature_importances_) # impurity-based importances
print(tree.tree_)                # the raw sklearn Tree object

# Leaf indices for each sample
leaf_idx = tree.apply(X)

# Indicator matrix: which nodes does each sample pass through?
path = tree.decision_path(X)

2.3.4. Cost-Sensitive Random Forest (CSForestClassifier)

CSForestClassifier builds an ensemble of CSTreeClassifier trees using bootstrap sampling and random feature subsets, identical to scikit-learn’s RandomForestClassifier except each tree is grown with a cost-sensitive splitting criterion [1].

2.3.4.1. Number of estimators

from empulse.models import CSForestClassifier

# Larger forests are more stable but slower to train
forest = CSForestClassifier(n_estimators=200, fp_cost=5, fn_cost=1)

2.3.4.2. Combining predictions

The combination parameter controls how individual tree predictions are aggregated into a single ensemble prediction:

combination	Description
"majority_voting" (default)	Each tree casts one vote; the majority class wins.
"weighted_voting"	Trees are weighted by their out-of-bag (OOB) score; requires oob_score=True.

forest = CSForestClassifier(
    n_estimators=100,
    fp_cost=5,
    fn_cost=1,
    combination="weighted_voting",
    oob_score=True,
)
forest.fit(X, y)
print(forest.oob_score_)

2.3.4.3. Parallelism and memory

Like RandomForestClassifier, fitting and prediction can be parallelised across CPU cores with n_jobs:

forest = CSForestClassifier(
    n_estimators=500,
    fp_cost=5,
    fn_cost=1,
    n_jobs=-1,  # use all available cores
)

2.3.4.4. Warm-start incremental training

Set warm_start=True to add more trees to an already-fitted forest without starting from scratch:

forest = CSForestClassifier(n_estimators=50, fp_cost=5, fn_cost=1, warm_start=True)
forest.fit(X, y)

forest.n_estimators = 100    # grow the forest to 100 trees
forest.fit(X, y)

2.3.4.5. Feature importances

forest.fit(X, y)
importances = forest.feature_importances_   # shape (n_features,)

# For a more reliable estimate use permutation importances
from sklearn.inspection import permutation_importance
result = permutation_importance(forest, X, y, n_repeats=10, random_state=0)

2.3.5. Cost-Sensitive Bagging (CSBaggingClassifier)

CSBaggingClassifier is the most flexible of the ensemble models. It is an ensemble meta-estimator that fits copies of a base classifier on random subsets of the dataset [1]. By default the base estimator is CSTreeClassifier, but any compatible classifier can be used.

2.3.5.1. Sampling strategies

The four classical bagging variants are all available through combinations of bootstrap, bootstrap_features, max_samples, and max_features:

Variant	Parameter settings
Pasting	bootstrap=False, bootstrap_features=False
Bagging (default)	bootstrap=True, bootstrap_features=False
Random Subspaces	bootstrap=False, bootstrap_features=True, max_features < 1.0
Random Patches	bootstrap=True, bootstrap_features=True, max_features < 1.0

from empulse.models import CSBaggingClassifier

# Standard bagging (default)
bagging = CSBaggingClassifier(n_estimators=20, fp_cost=5, fn_cost=1)

# Random Patches: subsample both samples and features
patches = CSBaggingClassifier(
    n_estimators=50,
    fp_cost=5,
    fn_cost=1,
    max_samples=0.8,
    max_features=0.7,
    bootstrap=True,
    bootstrap_features=True,
)

2.3.5.2. Custom base estimator

Any classifier that accepts cost arrays in its fit method can be used:

from empulse.models import CSBaggingClassifier, CSTreeClassifier

# Shallow cost-sensitive base trees
base = CSTreeClassifier(max_depth=3)
bagging = CSBaggingClassifier(
    estimator=base,
    n_estimators=50,
    fp_cost=5,
    fn_cost=1,
)
bagging.fit(X, y)

2.3.5.3. Out-of-bag evaluation

bagging = CSBaggingClassifier(
    n_estimators=20,
    fp_cost=5,
    fn_cost=1,
    oob_score=True,
)
bagging.fit(X, y)
print(bagging.oob_score_)

2.3.5.4. Inspecting sub-estimators

bagging.fit(X, y)

# List of fitted base estimators
print(len(bagging.estimators_))

# Bootstrap sample indices used for each estimator
print(bagging.estimators_samples_[0])

# Feature subset used for each estimator
print(bagging.estimators_features_[0])

2.3.6. Profit-Driven Evolutionary Tree (ProfTreeClassifier)

ProfTreeClassifier takes a fundamentally different approach: instead of growing a tree greedily by splitting nodes, it uses a genetic algorithm to evolve a population of complete trees over many generations. At each generation, trees are selected, crossed over, and mutated; their fitness is measured by a profit metric (default: MaxProfit). Because the search is gradient-free, it can optimise non-smooth and non-convex objectives that are intractable for gradient-based methods [2].

2.3.6.1. Basic usage

from empulse.models import ProfTreeClassifier
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=500, random_state=0)

proftree = ProfTreeClassifier(
    tp_cost=300,
    fp_cost=10,
    max_depth=5,
    max_iter=500,
    random_state=42,
)
proftree.fit(X, y)

2.3.6.2. Controlling the genetic algorithm

The GA is configured through five complementary variation operators; their probabilities must sum to exactly 1.0:

Parameter	Description
crossover_rate	Probability of crossing sub-trees between two parent trees.
grow_rate	Probability of attaching a new random split to a leaf.
prune_rate	Probability of removing a split (collapse an internal node to a leaf).
mutate_split_rate	Probability of replacing a split’s feature and threshold.
mutate_value_rate	Probability of replacing only a split’s threshold.

proftree = ProfTreeClassifier(
    tp_cost=300,
    fp_cost=10,
    crossover_rate=0.3,
    grow_rate=0.2,
    prune_rate=0.2,
    mutate_split_rate=0.15,
    mutate_value_rate=0.15,    # must sum to 1.0
    population_size=200,
    max_iter=1_000,
)

2.3.6.3. Tree size constraints

Computation time scales exponentially with depth, so it is important to constrain the tree size appropriately:

proftree = ProfTreeClassifier(
    tp_cost=300,
    fp_cost=10,
    max_depth=6,            # default 10; be careful above 8
    min_samples_split=30,   # default 20
    min_samples_leaf=10,    # default 7
)

2.3.6.4. Early stopping

The GA stops early if no improvement greater than tolerance is observed for patience consecutive generations:

proftree = ProfTreeClassifier(
    tp_cost=300,
    fp_cost=10,
    patience=200,         # wait 200 generations without improvement
    tolerance=1e-5,       # minimum relative improvement to count
    max_iter=2_000,
)

2.3.6.5. Complexity regularisation

Set alpha > 0 to penalise trees with many nodes, which can help reduce overfitting on small datasets:

proftree = ProfTreeClassifier(
    tp_cost=300,
    fp_cost=10,
    alpha=0.01,
)

2.3.6.6. Parallelising the GA

Set n_jobs to the number of CPU cores to use when evaluating the population in parallel:

proftree = ProfTreeClassifier(
    tp_cost=300,
    fp_cost=10,
    n_jobs=4,
)

2.3.6.7. Custom fitness metric

Any Metric from empulse.metrics can be used as the fitness function:

from empulse.metrics import Metric, CostMatrix, Savings
from empulse.models import ProfTreeClassifier

savings_metric = Metric(
    cost_matrix=CostMatrix().add_fp_cost('fp').add_fn_cost('fn'),
    strategy=Savings(),
)

proftree = ProfTreeClassifier(loss=savings_metric)
proftree.fit(X, y, fp=5, fn=1)

2.3.7. sklearn Integration

All four models are fully scikit-learn compatible: they can be embedded in Pipeline, evaluated with cross_val_score, and tuned with GridSearchCV. When instance-dependent costs are used, metadata routing must be enabled.

2.3.7.1. Pipeline with cross-validation

import numpy as np
from sklearn import set_config
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from empulse.models import CSForestClassifier

set_config(enable_metadata_routing=True)

X, y = make_classification(n_samples=500, random_state=0)
fp_cost = np.random.default_rng(0).uniform(1, 10, size=len(y))

pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('model', CSForestClassifier(n_estimators=50, fn_cost=1)
                .set_fit_request(fp_cost=True)),
])

scores = cross_val_score(pipeline, X, y, params={'fp_cost': fp_cost})

2.3.7.2. Hyperparameter search

import numpy as np
from sklearn import set_config
from sklearn.datasets import make_classification
from sklearn.metrics import make_scorer
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from empulse.metrics import expected_cost_loss
from empulse.models import CSForestClassifier

set_config(enable_metadata_routing=True)

X, y = make_classification(n_samples=200, random_state=0)
fp_cost = np.random.default_rng(0).uniform(1, 10, size=len(y))
fn_cost = 1.0

pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('model', CSForestClassifier().set_fit_request(fp_cost=True)),
])

scorer = make_scorer(
    expected_cost_loss,
    response_method='predict_proba',
    greater_is_better=False,
    normalize=True,
    fn_cost=fn_cost,
).set_score_request(fp_cost=True)

grid_search = GridSearchCV(
    pipeline,
    param_grid={'model__n_estimators': [50, 100, 200]},
    scoring=scorer,
)
grid_search.fit(X, y, fp_cost=fp_cost)
print(f"Best n_estimators: {grid_search.best_params_['model__n_estimators']}")

2.3.8. Choosing the Right Model

Situation	Recommended model
Interpretability is important	CSTreeClassifier — a single, visualisable tree
Best predictive performance	CSForestClassifier — the most accurate option in most cases
Non-standard base estimator needed	CSBaggingClassifier — fully customisable ensemble
Objective is non-smooth or non-convex	ProfTreeClassifier — gradient-free evolutionary search
Small dataset	ProfTreeClassifier with a small population_size or CSTreeClassifier with ccp_alpha pruning

2.3.9. References

[1] (1,2,3)

Correa Bahnsen, A., Aouada, D., & Ottersten, B. “Example-Dependent Cost-Sensitive Decision Trees.” Expert Systems with Applications, 42(19), 6609–6619, 2015. https://doi.org/10.1016/j.eswa.2015.04.042

[2]

Höppner, S., Stripling, E., Baesens, B., Broucke, S. V., & Verdonck, T. (2017). Profit driven decision trees for churn prediction. arXiv preprint arXiv:1712.08101.