Overfitting and Cross-Validation

A model that performs great on training data but poorly on new data has overfit, it memorized noise and quirks specific to the training set instead of learning patterns that generalize. This lesson shows how to spot it, and how to evaluate and tune models more robustly than a single train/test split allows.

Spotting overfitting

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.2, random_state=42
)

deep_tree = DecisionTreeClassifier(random_state=42)  # no depth limit, free to grow as complex as it wants
deep_tree.fit(X_train, y_train)

print('Train accuracy:', deep_tree.score(X_train, y_train))
print('Test accuracy: ', deep_tree.score(X_test, y_test))

model.score(X, y) is a shortcut that fits/predicts/scores in one call (accuracy for classifiers, R² for regressors). A wide gap between train accuracy (often near 1.0, an unconstrained tree can memorize the training set almost perfectly) and test accuracy is the classic signature of overfitting.

Cross-validation: don't trust a single split

A single train/test split can be lucky or unlucky, by chance, the test set might be unusually easy or hard. k-fold cross-validation splits the data into k chunks, trains and evaluates k times (each time using a different chunk as the "test" fold), and averages the results, a far more reliable estimate.

from sklearn.model_selection import cross_val_score

scores = cross_val_score(DecisionTreeClassifier(random_state=42), iris.data, iris.target, cv=5)
print('Scores per fold:', scores)
print('Mean CV accuracy:', scores.mean())

cv=5 means 5-fold cross-validation, the data is split 5 ways, and the model is trained and scored 5 separate times. scores holds one accuracy per fold, its mean is a much sturdier estimate than any single train_test_split result.

Tuning hyperparameters with GridSearchCV

from sklearn.model_selection import GridSearchCV

param_grid = {'max_depth': [1, 2, 3, 4, 5, None]}
grid = GridSearchCV(DecisionTreeClassifier(random_state=42), param_grid, cv=5)
grid.fit(iris.data, iris.target)

print('Best max_depth:', grid.best_params_)
print('Best CV score:', grid.best_score_)

GridSearchCV automates exactly what you'd otherwise do by hand: try every value in param_grid, cross-validate each one, and report which setting scored best, max_depth limits how many yes/no questions deep a tree can grow, a smaller number fights overfitting at the risk of underfitting if set too low.