Tutorial

This tutorial works through one problem from beginning to end: a telecom company wants to run a retention campaign, and needs to decide which customers to contact.

We use the Iranian churn dataset, which is well suited to value-driven modelling because every customer carries their own lifetime value. That makes the interesting question not “who is most likely to leave?” but “who is most profitable to keep?” — and those two questions have different answers.

Along the way you will see why a model with a good ROC AUC can still lose money, how to write a cost matrix that captures the campaign’s economics, how to train a model on it, and how to choose the threshold and validate the whole thing.

Note

Needs pip install empulse[optional] pandas. The dataset downloads once and is cached under ~/empulse_data.

Each page is self-contained: the first code block re-loads the data, so you can jump in anywhere.

If you only have five minutes, read the Quickstart instead.

• 1. The Problem with AUC
  • 1.1. The business problem
  • 1.2. The data
  • 1.3. A conventional baseline
  • 1.4. What it is actually worth
  • 1.5. Why it goes wrong
  • 1.6. Next
• 2. Writing the Cost Matrix
  • 2.1. The four outcomes
  • 2.2. Building it in Empulse
  • 2.3. Which parameters vary per customer?
  • 2.4. Sanity-check your matrix
  • 2.5. Next
• 3. Measuring in Money
  • 3.1. Setup
  • 3.2. Expected cost: what does this model cost me?
  • 3.3. Savings: how good is that, really?
  • 3.4. Maximum profit: what if the threshold is not fixed?
  • 3.5. Handling uncertainty in the business parameters
  • 3.6. Which should you use?
  • 3.7. Using them as scikit-learn scorers
  • 3.8. Next
• 4. Training on the Cost Matrix
  • 4.1. Setup
  • 4.2. Cost-sensitive logistic regression
  • 4.3. Read the results carefully
  • 4.4. Cost-sensitive gradient boosting
  • 4.5. Choosing a backend
  • 4.6. Other cost-sensitive models
  • 4.7. Plain costs, without a Metric
  • 4.8. Next
• 5. Deciding Who to Actually Contact
  • 5.1. Setup
  • 5.2. How many customers should we contact?
  • 5.3. Converting a rate into a threshold
  • 5.4. Letting a meta-estimator do it
  • 5.5. A caveat on this dataset
  • 5.6. Thresholds move with the business, not the model
  • 5.7. Next
• 6. Validating the Whole Thing
  • 6.1. The complication: costs must follow the folds
  • 6.2. Cross-validating on savings
  • 6.3. Tuning hyperparameters on business value
  • 6.4. Tuning the business parameters too
  • 6.5. Turning routing back off
  • 6.6. What we built
  • 6.7. Where next