Loading Curriculum...

Artificial Intelligence (AI) Frameworks

ScikitLearn

Lesson 5/6

Frameworks

scikit-learn

scikit-learn is the go-to library for classical machine learning: preprocessing, feature engineering, models, evaluation, and pipelines.

NumPy Pandas scikit-learn PyTorch TensorFlow Hugging Face

Strengths

Consistent API

Every estimator follows fit / predict / transform. Switch models by changing one class name — the rest of your code stays the same.

Broad algorithm coverage

Linear models, decision trees, ensembles, SVMs, KNN, naive Bayes, clustering, dimensionality reduction — all in one library.

Pipeline system

Chain preprocessing and models into a single object. Prevents data leakage, simplifies cross-validation, and makes deployment clean.

Evaluation tooling

Cross-validation, GridSearchCV, RandomizedSearchCV, and 30+ metrics out of the box. The standard for reproducible ML evaluation.

Composable preprocessing

ColumnTransformer applies different transformers to different columns in a single step — essential for mixed numeric/categorical data.

Interoperability

Outputs are NumPy arrays; inputs accept Pandas DataFrames. Plays well with XGBoost, LightGBM, and any sklearn-compatible estimator.

The pipeline pattern

A Pipeline chains preprocessing and a model into one object. When you call fit, each step is fitted on training data only — the scaler never sees test data, eliminating the most common source of data leakage. ColumnTransformer lets you apply different transforms to different column types in one step.

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.ensemble import RandomForestClassifier

numeric_features = ["age", "income", "tenure"]
categorical_features = ["country", "plan"]

numeric_transformer = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler()),
])

categorical_transformer = Pipeline([
    ("imputer", SimpleImputer(strategy="most_frequent")),
    ("encoder", OneHotEncoder(handle_unknown="ignore")),
])

preprocessor = ColumnTransformer([
    ("num", numeric_transformer, numeric_features),
    ("cat", categorical_transformer, categorical_features),
])

pipe = Pipeline([
    ("preprocessor", preprocessor),
    ("classifier", RandomForestClassifier(n_estimators=200, random_state=0)),
])

pipe.fit(X_train, y_train)
pipe.score(X_test, y_test)

Pipeline

Sequential chain of (name, estimator) tuples. Last step must be an estimator; all others must implement transform.

ColumnTransformer

Apply different transformers to different subsets of columns and concatenate the results.

FunctionTransformer

Wrap any stateless function (e.g. np.log1p) as a pipeline-compatible transformer.

Estimator cheat-sheet

Start with a simple baseline, then move to ensembles. All follow the same fit / predict interface.

Linear

LogisticRegressionClassification

Strong baseline; interpretable coefficients; fast.

Ridge / LassoRegression

Ridge = L2 regularisation; Lasso = L1 + feature selection.

SGDClassifierClassification

Scales to millions of samples with partial_fit.

Tree-based

DecisionTreeClassifierClassification

Interpretable; prone to overfitting without pruning.

RandomForestClassifierBoth

Robust ensemble; good out-of-the-box; built-in feature importance.

GradientBoostingClassifierBoth

High accuracy; slower than RF; tune n_estimators + learning_rate.

HistGradientBoosting*Both

sklearn's fast GBDT for large datasets; handles NaNs natively.

Other

SVCClassification

Strong on high-dimensional data; doesn't scale past ~100k rows.

KNeighborsClassifierBoth

Simple, no training phase; slow at inference on large data.

MLPClassifierBoth

Shallow neural net; use PyTorch instead for anything serious.

* HistGradientBoostingClassifier / Regressor — sklearn s fastest tree method for >10k rows.

Hyperparameter search

Prefer RandomizedSearchCV over GridSearchCV — it samples the parameter space randomly, finding good regions much faster than exhaustive grid search and scaling to large param spaces. Set n_jobs=-1 to use all CPU cores.

from sklearn.model_selection import RandomizedSearchCV
from sklearn.ensemble import GradientBoostingClassifier
from scipy.stats import randint, uniform

param_dist = {
    "classifier__n_estimators": randint(50, 500),
    "classifier__max_depth": randint(2, 8),
    "classifier__learning_rate": uniform(0.01, 0.3),
    "classifier__subsample": uniform(0.6, 0.4),
}

search = RandomizedSearchCV(
    pipe,
    param_distributions=param_dist,
    n_iter=50,
    cv=5,
    scoring="roc_auc",
    n_jobs=-1,
    random_state=0,
    verbose=1,
)
search.fit(X_train, y_train)
print(f"Best AUC: {search.best_score_:.4f}")
print(search.best_params_)

GridSearchCV

Exhaustive — tries every combination. Fine for 2–3 params with a few values; explodes otherwise.

RandomizedSearchCV

Samples n_iter combinations at random. Scales well; often finds better results in fewer evaluations.

HalvingRandomSearchCV

Successive halving — eliminates weak candidates early. Faster than RandomizedSearchCV for large budgets.

Nested CV

Outer CV evaluates generalisation; inner CV tunes hyperparams. Prevents overfitting the validation set.

Evaluation & metrics

Always plot a confusion matrix, ROC curve, and precision-recall curve together. ROC AUC is optimistic on imbalanced data — PR AUC is more informative when the positive class is rare.

from sklearn.metrics import (
    classification_report,
    ConfusionMatrixDisplay,
    RocCurveDisplay,
    PrecisionRecallDisplay,
)
import matplotlib.pyplot as plt

y_pred = pipe.predict(X_test)
y_proba = pipe.predict_proba(X_test)[:, 1]

print(classification_report(y_test, y_pred))

fig, axes = plt.subplots(1, 3, figsize=(15, 4))
ConfusionMatrixDisplay.from_predictions(y_test, y_pred, ax=axes[0])
RocCurveDisplay.from_predictions(y_test, y_proba, ax=axes[1])
PrecisionRecallDisplay.from_predictions(y_test, y_proba, ax=axes[2])
plt.tight_layout()

Accuracy

Misleading when classes are imbalanced. Use as a secondary metric only.

ROC AUC

Threshold-free ranking metric. Good general-purpose metric for binary classification.

F1 / PR AUC

Precision-recall trade-off — use when false positives and false negatives have different costs.

R² / MAE / RMSE

For regression. MAE is robust to outliers; RMSE penalises large errors more heavily.

Log loss

Measures calibration of probability estimates — important when you need reliable probabilities.

Cohen's kappa

Agreement corrected for chance — useful for multi-class problems with class imbalance.

Feature importance & inspection

Tree-based models expose feature_importances_ (impurity-based — can be biased toward high-cardinality features). Use permutation_importance as a model-agnostic, unbiased alternative. For linear models, inspect coef_ after scaling.

import pandas as pd
import numpy as np

# For tree-based models inside a Pipeline
rf = pipe.named_steps["classifier"]
feature_names = (
    numeric_features
    + pipe.named_steps["preprocessor"]
      .named_transformers_["cat"]
      .named_steps["encoder"]
      .get_feature_names_out(categorical_features)
      .tolist()
)

importance_df = pd.DataFrame({
    "feature": feature_names,
    "importance": rf.feature_importances_,
}).sort_values("importance", ascending=False)

print(importance_df.head(10))

# Model-agnostic: permutation importance
from sklearn.inspection import permutation_importance
result = permutation_importance(pipe, X_test, y_test, n_repeats=10, random_state=0)

Writing custom transformers

Subclass BaseEstimator and TransformerMixin. Implement fit (return self) and transform. Your custom transformer is now a first-class Pipeline citizen — it participates in cross-validation, cloning, and serialisation.

from sklearn.base import BaseEstimator, TransformerMixin
import numpy as np

class LogTransformer(BaseEstimator, TransformerMixin):
    """Log1p-transform skewed numeric columns."""

    def __init__(self, columns=None):
        self.columns = columns

    def fit(self, X, y=None):
        return self  # stateless

    def transform(self, X):
        X = X.copy()
        cols = self.columns or X.columns.tolist()
        X[cols] = np.log1p(X[cols].clip(lower=0))
        return X

# Drop into any Pipeline
pipe = Pipeline([
    ("log", LogTransformer(columns=["income", "tenure"])),
    ("scaler", StandardScaler()),
    ("clf", LogisticRegression()),
])

Key rule: fit must only learn from X (the training data) and must return self. Never store test-time data in fit. If your transformer is stateless (like a log transform), fit can simply return self immediately.

Preprocessing reference

StandardScalerLinear models, SVMs, KNN, PCA

Zero mean, unit variance. Sensitive to outliers.

MinMaxScalerNeural nets, image pixels

Scales to [0, 1]. Distorted by outliers.

RobustScalerData with outliers

Uses median + IQR — not affected by extreme values.

OneHotEncoderNominal categories

Creates one binary column per category. Use handle_unknown='ignore' for unseen categories.

OrdinalEncoderOrdinal categories

Encodes as integers. Only use when order is meaningful.

SimpleImputerMissing values

Strategies: mean, median, most_frequent, constant.

IterativeImputerComplex missingness

Models each feature as a function of others — more accurate, more expensive.

PolynomialFeaturesLinear model + interactions

Creates x², x³, x·y interaction terms. Degree=2 is usually sufficient.

PCADimensionality reduction

Unsupervised. Useful before KNN or SVMs on high-dimensional data.

Install

Create a virtual environment (recommended) and activate it.
Install: `pip install scikit-learn`.
For the full tabular ML stack: `pip install scikit-learn pandas numpy matplotlib seaborn`.
Verify: `python -c "import sklearn; print(sklearn.__version__)"`.

For tabular ML, pair this with Pandas for ETL and NumPy for array math.
Pin `scikit-learn==X.Y.Z` in requirements.txt — minor version changes can shift model defaults.
For gradient boosting on tabular data, also consider `pip install xgboost lightgbm` — they follow the same fit/predict API.

Hello world

from sklearn.datasets import load_iris from sklearn.model_selection import train_test_split, cross_val_score from sklearn.linear_model import LogisticRegression from sklearn.preprocessing import StandardScaler from sklearn.pipeline import Pipeline from sklearn.metrics import classification_report X, y = load_iris(return_X_y=True) X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=0, stratify=y ) pipe = Pipeline([ ("scaler", StandardScaler()), ("clf", LogisticRegression(max_iter=1000)), ]) # 5-fold cross-validation on training set cv_scores = cross_val_score(pipe, X_train, y_train, cv=5) print(f"CV accuracy: {cv_scores.mean():.3f} ± {cv_scores.std():.3f}") pipe.fit(X_train, y_train) print(classification_report(y_test, pipe.predict(X_test)))

Always use stratify=y in train_test_split for classification to preserve class ratios.
Wrap preprocessing in a Pipeline before cross-validating — prevents data leakage from the scaler.
classification_report shows per-class precision, recall, and F1 — more informative than raw accuracy.

Best for / Not for

Best for

Classical ML baselines: linear models, trees, ensembles, SVMs, KNN.
Evaluation, cross-validation, and model selection (GridSearchCV, RandomizedSearchCV).
Packaging preprocessing and models with Pipeline to prevent data leakage.
Feature engineering: scaling, encoding, imputation, polynomial features.
Interpretable models where coefficients and feature importances matter.
Small-to-medium tabular datasets where neural networks are overkill.

Not for

Large neural nets and GPU-heavy training (use PyTorch / TensorFlow).
End-to-end fine-tuning of LLMs (use Hugging Face + PyTorch).
Unstructured data like raw images, audio, or text at scale.
Online / streaming learning at very large scale (consider river or Vowpal Wabbit).

Common pitfalls

Data leakage: fitting the scaler on the full dataset before splitting — always use Pipeline.
Wrong metric for the business goal — accuracy is misleading on imbalanced data; use F1, AUC-ROC, or PR curve.
Not using cross-validation when data is limited — a single split gives a noisy estimate.
Ignoring class imbalance — use stratified splits, class_weight='balanced', or SMOTE.
Overfitting the validation set during hyperparameter search — use nested CV for unbiased estimates.
Forgetting random_state — results are not reproducible without fixing it.
Using default hyperparameters and calling it done — most estimators benefit from even light tuning.

cheat_sheet

scikit-learn syntax sheet

Fit/predict, pipelines, and train/test split.

from sklearn.model_selection import train_test_split from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.linear_model import LogisticRegression Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.2, random_state=0, stratify=y) pipe = Pipeline([ ("scale", StandardScaler()), ("clf", LogisticRegression(max_iter=1000)), ]) pipe.fit(Xtr, ytr) pred = pipe.predict(Xte) score = pipe.score(Xte, yte)

Use `Pipeline` to avoid leakage: preprocessing must be fit on train only.
Use cross-validation when data is limited (`cross_val_score`).

Quick comparison

Framework	Best at	Typical tasks	Output
NumPy	Fast array math + linear algebra	Numerics, prototyping, preprocessing	ndarray (CPU)
Pandas	Tabular ETL + joins + aggregation	Cleaning, feature engineering, analysis	DataFrame / Series
scikit-learn	Classical ML + evaluation + pipelines	Baselines, CV, model selection	Estimator / Pipeline
PyTorch	Neural nets + custom training loops	Deep learning, research, fine-tuning	Tensors + nn.Module
TensorFlow	Keras training + production deployment	Deep learning, serving, mobile/edge	Tensors + Keras Model
Hugging Face	Model hub + transformer tooling	Inference, fine-tuning, sharing	Checkpoints + pipelines

Related lessons

Pandas

ETL and feature engineering for tabular ML.

NumPy

Array math under the hood of many sklearn workflows.

PyTorch

For neural nets when classical ML is not enough.

XGBoost / LightGBM

Gradient boosting libraries with sklearn-compatible APIs — often the best baseline for tabular data.

Navigation

PandasPrev PyTorch Next Back to topicIndex

Loading Lab...