Feature scaling and preprocessing

Understanding Feature scaling and preprocessing

An estimator sees the feature matrix as plain numbers with no memory of units. If one column is income in dollars (tens of thousands) and another is age in years (tens), then any model that measures distance between rows, takes a dot product with a weight vector, or penalises the size of coefficients will treat income as about a thousand times more important, purely because of the unit someone happened to record it in. Scaling removes that accident: after standardising, one unit means one standard deviation in every column, so the model has to earn its weighting from the data instead of inheriting it from the measuring stick.

A scaler is itself a fitted model. StandardScaler.fit computes a mean and a standard deviation per column and stores them in .mean_ and .scale_; transform then applies (x - mean) / scale using those frozen numbers. That is why you fit on the training split and only call transform on validation, test and production rows: they have to be pushed through the same arithmetic, otherwise your test features live in a different coordinate system than the one the model learnt in. It also means scaled test values are not guaranteed to be centred or bounded, so a test row far from the training mean coming out at 2.68 is the correct answer, not a bug.

Which scaler you use depends on the column, not on habit. StandardScaler suits roughly symmetric data; MinMaxScaler maps the training range onto [0, 1] and is extremely sensitive to a single extreme value, which can squash every ordinary row into a tiny sliver; RobustScaler centres on the median and divides by the interquartile range, so extremes barely move it. Decision trees and forests are indifferent, because a split is just x <= t and any monotone rescaling moves the threshold along with the data. None of these transformers change the shape of a distribution either: standardising a skewed feature gives you a skewed feature with mean 0.

import numpy as np
from sklearn.preprocessing import StandardScaler

# column 0 = age in years, column 1 = income in dollars
X_train = np.array([[20., 30000.],
                    [30., 50000.],
                    [40., 70000.],
                    [50., 90000.]])
X_test = np.array([[35., 60000.],
                   [60., 120000.]])

scaler = StandardScaler().fit(X_train)   # statistics come from train only
print(f"train mean:  age={scaler.mean_[0]:.1f} income={scaler.mean_[1]:.1f}")
print(f"train scale: age={scaler.scale_[0]:.3f} income={scaler.scale_[1]:.3f}")

Z_train = scaler.transform(X_train)
Z_test = scaler.transform(X_test)        # reuses the same two numbers per column
print("scaled train:", np.round(Z_train, 3).tolist())
print("scaled test: ", np.round(Z_test, 3).tolist())
print("train centred:", np.allclose(Z_train.mean(axis=0), 0))
print("train unit std:", np.allclose(Z_train.std(axis=0), 1))

Worked examples

import numpy as np
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

# column 0 carries the signal (near 0 -> class 0, near 1 -> class 1)
# column 1 is noise recorded in a huge unit
X = np.array([[0.0, 1000.],
              [0.1, 9000.],
              [0.9, 1050.],
              [1.0, 8950.]])
y = np.array([0, 0, 1, 1])
new = np.array([[0.2, 5000.]])

raw = KNeighborsClassifier(n_neighbors=1).fit(X, y)
scaled = make_pipeline(StandardScaler(),
                       KNeighborsClassifier(n_neighbors=1)).fit(X, y)

print("raw features    ->", raw.predict(new))
print("scaled features ->", scaled.predict(new))

import numpy as np
from sklearn.preprocessing import MinMaxScaler, RobustScaler

X_train = np.array([[10.], [20.], [30.], [40.], [1000.]])  # last row is an outlier
X_test = np.array([[5.], [25.], [2000.]])

mm = MinMaxScaler().fit(X_train)
rb = RobustScaler().fit(X_train)

print("min-max train:", mm.transform(X_train).ravel().round(3).tolist())
print("min-max test: ", mm.transform(X_test).ravel().round(3).tolist())
print("robust train: ", rb.transform(X_train).ravel().round(3).tolist())
print("median, IQR:  ", rb.center_[0], rb.scale_[0])

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

df = pd.DataFrame({"age": [20., 40., 60.],
                   "city": ["oslo", "rome", "oslo"]})

pre = ColumnTransformer([
    ("num", StandardScaler(), ["age"]),
    ("cat", OneHotEncoder(), ["city"]),
])
out = pre.fit_transform(df)

print(np.round(out, 3).tolist())
print(pre.get_feature_names_out().tolist())

Important notes

Common mistakes

Try it yourself