Evaluating Models

Accuracy is the easiest metric to understand, and one of the easiest to be misled by. This lesson shows exactly how, using a deliberately imbalanced dataset, and introduces the metrics that actually reveal what's going on.

When accuracy lies

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score

# 95% of samples belong to class 0, only 5% to class 1, a realistic fraud/spam-style imbalance
X, y = make_classification(n_samples=1000, weights=[0.95, 0.05], random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = DecisionTreeClassifier(random_state=42)
model.fit(X_train, y_train)
preds = model.predict(X_test)

print('Accuracy:', accuracy_score(y_test, preds))

make_classification(weights=[0.95, 0.05]) generates a synthetic dataset with a 95/5 class imbalance. A model that lazily predicts "class 0" for everything, without learning anything real, would still score around 95% accuracy here, high accuracy alone doesn't prove the model is actually useful.

Metrics that reveal the truth

from sklearn.metrics import confusion_matrix, precision_score, recall_score

print('Confusion matrix:\n', confusion_matrix(y_test, preds))
print('Precision:', precision_score(y_test, preds))
print('Recall:', recall_score(y_test, preds))

• The confusion matrix breaks predictions into a 2x2 grid: true negatives, false positives, false negatives, and true positives (in that reading order for binary classification). It's the raw material every other classification metric is computed from.
• Precision answers "of everything the model flagged as positive, how much was actually positive?", important when false alarms are costly.
• Recall answers "of everything that was actually positive, how much did the model catch?", important when missing a positive case is costly (like a real fraud case slipping through).

A model can have high accuracy and terrible recall on the minority class simultaneously, that's exactly the trap this lesson demonstrates.