Classification with Decision Trees and k-NN

Classification means predicting a category (which species? spam or not?) rather than a number. This lesson compares two classic, intuitive classifiers on the same data, and shows off scikit-learn's consistent API in action.

Two very different ideas, one shared interface

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.2, random_state=42
)

tree = DecisionTreeClassifier(random_state=42)
tree.fit(X_train, y_train)
tree_accuracy = accuracy_score(y_test, tree.predict(X_test))

knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train, y_train)
knn_accuracy = accuracy_score(y_test, knn.predict(X_test))

print('Decision Tree accuracy:', tree_accuracy)
print('k-NN accuracy:         ', knn_accuracy)

• A decision tree learns a sequence of yes/no questions ("is petal length < 2.5cm?") that splits the data into increasingly pure groups, similar in spirit to a flowchart you might design by hand, except the tree figures out which questions to ask and in what order.
• k-NN (k-nearest neighbors) works completely differently: to classify a new point, it finds the k closest points in the training data (by distance) and takes a majority vote among their labels. n_neighbors=5 is a hyperparameter, a setting you choose before training, not something the model learns on its own.
• Despite radically different internal logic, both models are trained and used identically: fit(X_train, y_train) then predict(X_test). This consistency is exactly why scikit-learn is so easy to experiment with, swapping one model for another is often a one-line change.

accuracy_score

accuracy_score(y_true, y_pred) simply computes the fraction of predictions that matched the actual label, 0.0 (nothing right) to 1.0 (everything right). It's the simplest possible evaluation metric, and, as you'll see in a later lesson, sometimes a misleading one.