3.3. Tree-Based Cost-Sensitive Models

Empulse provides four tree-based classifiers that incorporate cost information directly into the learning process. Three of them (CSTreeClassifier, CSForestClassifier, and CSBaggingClassifier) use a cost-sensitive splitting criterion, so each tree node is grown by maximising cost reduction rather than a purity measure like Gini impurity. The fourth (ProfTreeClassifier) takes a different approach and uses an evolutionary genetic algorithm to evolve trees that directly maximise a user-defined profit metric.

Model	Base algorithm	Key characteristic
CSTreeClassifier	Decision tree	Single cost-sensitive tree; interpretable
CSForestClassifier	Random forest	Ensemble of cost-sensitive trees; feature importances
CSBaggingClassifier	Bagging / Pasting / Random Patches	Flexible ensemble; custom base estimator
ProfTreeClassifier	Genetic algorithm	Evolves trees directly against a profit metric

3.3.1. Quick Start

import numpy as np
from sklearn.datasets import make_classification
from empulse.models import (
    CSTreeClassifier,
    CSForestClassifier,
    CSBaggingClassifier,
    ProfTreeClassifier,
)

X, y = make_classification(n_samples=1_000, random_state=0)

# Cost-sensitive decision tree
tree = CSTreeClassifier(fp_cost=5, fn_cost=1)
tree.fit(X, y)

# Cost-sensitive random forest
forest = CSForestClassifier(n_estimators=100, fp_cost=5, fn_cost=1)
forest.fit(X, y)

# Cost-sensitive bagging
bagging = CSBaggingClassifier(n_estimators=10, fp_cost=5, fn_cost=1)
bagging.fit(X, y)

# Profit-maximising evolutionary tree
proftree = ProfTreeClassifier(tp_cost=300, fp_cost=10)
proftree.fit(X, y)

y_proba = forest.predict_proba(X)[:, 1]

3.3.2. Specifying costs

All four models accept costs the same two ways as every other cost-sensitive model in Empulse: as plain tp_cost/tn_cost/fp_cost/fn_cost values, scalar or per-sample, or as a Metric passed as loss. Handing costs to an estimator has the rules; Costs that differ per row covers getting per-sample arrays through cross-validation.

import numpy as np
from sklearn.datasets import make_classification
from empulse.models import CSForestClassifier

X, y = make_classification(n_samples=500, random_state=0)
clv = np.random.default_rng(0).uniform(100, 1_000, size=len(y))
contact_cost = 10

model = CSForestClassifier(fn_cost=1)
model.fit(X, y, tp_cost=clv - contact_cost, fp_cost=contact_cost)

Unlike the linear and boosting models, the tree-based models can train on any of the six strategies, including the two ranking-based ones — a tree only needs a scalar fitness for a split, not a gradient. See Models by supported strategy.

3.3.3. Cost-Sensitive Decision Tree (CSTreeClassifier)

CSTreeClassifier is a single decision tree whose splitting criterion directly maximises cost savings at each node [1]. It wraps scikit-learn’s DecisionTreeClassifier and exposes the same tree structure, pruning utilities, and feature importances.

3.3.3.1. Split criterion

The criterion parameter controls how the cost signal is weighted at each split:

criterion	Description
"cost" (default)	Pure cost reduction — splits are evaluated by the expected-cost gain.
"gini"	Cost gain is weighted by Gini impurity — blends class separation with cost.
"entropy" / "log_loss"	Cost gain is weighted by Shannon information gain.

from empulse.models import CSTreeClassifier

# Default: use the raw cost impurity
tree = CSTreeClassifier(fp_cost=5, fn_cost=1)

# Weight by Gini impurity
tree_gini = CSTreeClassifier(fp_cost=5, fn_cost=1, criterion="gini")

# Weight by entropy
tree_entropy = CSTreeClassifier(fp_cost=5, fn_cost=1, criterion="entropy")

3.3.3.2. Controlling tree size

Use the standard scikit-learn parameters to regularise the tree:

tree = CSTreeClassifier(
    fp_cost=5,
    fn_cost=1,
    max_depth=5,
    min_samples_leaf=20,
    min_samples_split=50,
)

3.3.3.2.1. Post-training pruning via ccp_alpha

Minimal Cost-Complexity Pruning is available through the ccp_alpha parameter. To find a good value, inspect the pruning path first:

from empulse.models import CSTreeClassifier
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=500, random_state=0)
tree = CSTreeClassifier(fp_cost=5, fn_cost=1).fit(X, y)

path = tree.cost_complexity_pruning_path(X, y)
print(path.ccp_alphas)    # candidate alpha values
print(path.impurities)    # impurity of the corresponding subtrees

# Apply pruning with a chosen alpha
pruned_tree = CSTreeClassifier(fp_cost=5, fn_cost=1, ccp_alpha=0.01).fit(X, y)

3.3.3.3. Inspecting the tree

CSTreeClassifier exposes the underlying sklearn tree object and several inspection helpers:

tree.fit(X, y)

print(tree.get_depth())          # maximum depth reached
print(tree.get_n_leaves())       # number of leaf nodes
print(tree.feature_importances_) # impurity-based importances
print(tree.tree_)                # the raw sklearn Tree object

# Leaf indices for each sample
leaf_idx = tree.apply(X)

# Indicator matrix: which nodes does each sample pass through?
path = tree.decision_path(X)

3.3.4. Cost-Sensitive Random Forest (CSForestClassifier)

CSForestClassifier builds an ensemble of CSTreeClassifier trees using bootstrap sampling and random feature subsets, identical to scikit-learn’s RandomForestClassifier except each tree is grown with a cost-sensitive splitting criterion [1].

3.3.4.1. Number of estimators

from empulse.models import CSForestClassifier

# Larger forests are more stable but slower to train
forest = CSForestClassifier(n_estimators=200, fp_cost=5, fn_cost=1)

3.3.4.2. Combining predictions

The combination parameter controls how individual tree predictions are aggregated into a single ensemble prediction:

combination	Description
"majority_voting" (default)	Each tree casts one vote; the majority class wins.
"weighted_voting"	Trees are weighted by their out-of-bag (OOB) score; requires oob_score=True.

forest = CSForestClassifier(
    n_estimators=100,
    fp_cost=5,
    fn_cost=1,
    combination="weighted_voting",
    oob_score=True,
)
forest.fit(X, y)
print(forest.oob_score_)

3.3.4.3. Parallelism and memory

Like RandomForestClassifier, fitting and prediction can be parallelised across CPU cores with n_jobs:

forest = CSForestClassifier(
    n_estimators=500,
    fp_cost=5,
    fn_cost=1,
    n_jobs=-1,  # use all available cores
)

3.3.4.4. Warm-start incremental training

Set warm_start=True to add more trees to an already-fitted forest without starting from scratch:

forest = CSForestClassifier(n_estimators=50, fp_cost=5, fn_cost=1, warm_start=True)
forest.fit(X, y)

forest.n_estimators = 100    # grow the forest to 100 trees
forest.fit(X, y)

3.3.4.5. Feature importances

forest.fit(X, y)
importances = forest.feature_importances_   # shape (n_features,)

# For a more reliable estimate use permutation importances
from sklearn.inspection import permutation_importance
result = permutation_importance(forest, X, y, n_repeats=10, random_state=0)

3.3.5. Cost-Sensitive Bagging (CSBaggingClassifier)

CSBaggingClassifier is the most flexible of the ensemble models. It is an ensemble meta-estimator that fits copies of a base classifier on random subsets of the dataset [1]. By default the base estimator is CSTreeClassifier, but any compatible classifier can be used.

3.3.5.1. Sampling strategies

The four classical bagging variants are all available through combinations of bootstrap, bootstrap_features, max_samples, and max_features:

Variant	Parameter settings
Pasting	bootstrap=False, bootstrap_features=False
Bagging (default)	bootstrap=True, bootstrap_features=False
Random Subspaces	bootstrap=False, bootstrap_features=True, max_features < 1.0
Random Patches	bootstrap=True, bootstrap_features=True, max_features < 1.0

from empulse.models import CSBaggingClassifier

# Standard bagging (default)
bagging = CSBaggingClassifier(n_estimators=20, fp_cost=5, fn_cost=1)

# Random Patches: subsample both samples and features
patches = CSBaggingClassifier(
    n_estimators=50,
    fp_cost=5,
    fn_cost=1,
    max_samples=0.8,
    max_features=0.7,
    bootstrap=True,
    bootstrap_features=True,
)

3.3.5.2. Custom base estimator

Any classifier that accepts cost arrays in its fit method can be used:

from empulse.models import CSBaggingClassifier, CSTreeClassifier

# Shallow cost-sensitive base trees
base = CSTreeClassifier(max_depth=3)
bagging = CSBaggingClassifier(
    estimator=base,
    n_estimators=50,
    fp_cost=5,
    fn_cost=1,
)
bagging.fit(X, y)

3.3.5.3. Out-of-bag evaluation

bagging = CSBaggingClassifier(
    n_estimators=20,
    fp_cost=5,
    fn_cost=1,
    oob_score=True,
)
bagging.fit(X, y)
print(bagging.oob_score_)

3.3.5.4. Inspecting sub-estimators

bagging.fit(X, y)

# List of fitted base estimators
print(len(bagging.estimators_))

# Bootstrap sample indices used for each estimator
print(bagging.estimators_samples_[0])

# Feature subset used for each estimator
print(bagging.estimators_features_[0])

3.3.6. Profit-Driven Evolutionary Tree (ProfTreeClassifier)

ProfTreeClassifier takes a fundamentally different approach: instead of growing a tree greedily by splitting nodes, it uses a genetic algorithm to evolve a population of complete trees over many generations. At each generation, trees are selected, crossed over, and mutated; their fitness is measured by a profit metric (default: MaxProfit). Because the search is gradient-free, it can optimise non-smooth and non-convex objectives that are intractable for gradient-based methods [2].

3.3.6.1. Basic usage

from empulse.models import ProfTreeClassifier
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=500, random_state=0)

proftree = ProfTreeClassifier(
    tp_cost=300,
    fp_cost=10,
    max_depth=5,
    max_iter=500,
    random_state=42,
)
proftree.fit(X, y)

3.3.6.2. Controlling the genetic algorithm

The GA is configured through five complementary variation operators; their probabilities must sum to exactly 1.0:

Parameter	Description
crossover_rate	Probability of crossing sub-trees between two parent trees.
grow_rate	Probability of attaching a new random split to a leaf.
prune_rate	Probability of removing a split (collapse an internal node to a leaf).
mutate_split_rate	Probability of replacing a split’s feature and threshold.
mutate_value_rate	Probability of replacing only a split’s threshold.

proftree = ProfTreeClassifier(
    tp_cost=300,
    fp_cost=10,
    crossover_rate=0.3,
    grow_rate=0.2,
    prune_rate=0.2,
    mutate_split_rate=0.15,
    mutate_value_rate=0.15,    # must sum to 1.0
    population_size=200,
    max_iter=1_000,
)

3.3.6.3. Tree size constraints

Computation time scales exponentially with depth, so it is important to constrain the tree size appropriately:

proftree = ProfTreeClassifier(
    tp_cost=300,
    fp_cost=10,
    max_depth=6,            # default 10; be careful above 8
    min_samples_split=30,   # default 20
    min_samples_leaf=10,    # default 7
)

3.3.6.4. Early stopping

The GA stops early if no improvement greater than tolerance is observed for patience consecutive generations:

proftree = ProfTreeClassifier(
    tp_cost=300,
    fp_cost=10,
    patience=200,         # wait 200 generations without improvement
    tolerance=1e-5,       # minimum relative improvement to count
    max_iter=2_000,
)

3.3.6.5. Complexity regularisation

Set alpha > 0 to penalise trees with many nodes, which can help reduce overfitting on small datasets:

proftree = ProfTreeClassifier(
    tp_cost=300,
    fp_cost=10,
    alpha=0.01,
)

3.3.6.6. Parallelising the GA

Set n_jobs to the number of CPU cores to use when evaluating the population in parallel:

proftree = ProfTreeClassifier(
    tp_cost=300,
    fp_cost=10,
    n_jobs=4,
)

3.3.6.7. Custom fitness metric

Any Metric from empulse.metrics can be used as the fitness function:

from empulse.metrics import Metric, CostMatrix, Savings
from empulse.models import ProfTreeClassifier

savings_metric = Metric(
    cost_matrix=CostMatrix().add_fp_cost('fp').add_fn_cost('fn'),
    strategy=Savings(),
)

proftree = ProfTreeClassifier(loss=savings_metric)
proftree.fit(X, y, fp=5, fn=1)

3.3.7. sklearn Integration

All four models are ordinary scikit-learn estimators and drop into Pipeline, cross_val_score and GridSearchCV unchanged. Per-sample costs reach each fold through metadata routing — see Costs that differ per row.

from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('model', CSForestClassifier(n_estimators=10, fp_cost=5, fn_cost=1)),
])

grid_search = GridSearchCV(pipeline, {'model__max_depth': [3, 5]}, cv=3)
grid_search.fit(X, y)
print(grid_search.best_params_['model__max_depth'])

3.3.8. Choosing the Right Model

Situation	Recommended model
Interpretability is important	CSTreeClassifier — a single, visualisable tree
Best predictive performance	CSForestClassifier — the most accurate option in most cases
Non-standard base estimator needed	CSBaggingClassifier — fully customisable ensemble
Objective is non-smooth or non-convex	ProfTreeClassifier — gradient-free evolutionary search
Small dataset	ProfTreeClassifier with a small population_size or CSTreeClassifier with ccp_alpha pruning

3.3.9. References

[1] (1,2,3)

Correa Bahnsen, A., Aouada, D., & Ottersten, B. “Example-Dependent Cost-Sensitive Decision Trees.” Expert Systems with Applications, 42(19), 6609–6619, 2015. https://doi.org/10.1016/j.eswa.2015.04.042

[2]

Höppner, S., Stripling, E., Baesens, B., vanden Broucke, S., & Verdonck, T. (2020). Profit driven decision trees for churn prediction. European journal of operational research, 284(3), 920-933.