Techniques for Hyperparameter Tuning

Models can have many hyperparameters and finding the best combination can be treated as a search problem.

• GridSearchCV
• RandomizedSearchCV
• Bayesian Optimization

1. GridSearchCV

GridSearchCV is a brute-force technique for hyperparameter tuning. It trains the model using all possible combinations of specified hyperparameter values to find the best-performing setup.

It is slow and computationally expensive because it evaluates every possible combination of parameters.

Working Process

1. Create a grid of possible hyperparameter values.
2. Train the model on every combination.
3. Evaluate each model using cross-validation.
4. Select the combination with the highest score.

Example Parameter Space

C = [0.1, 0.2, 0.3, 0.4, 0.5]
penalty = [0.01, 0.1, 0.5, 1.0]

Total Models = 5 × 4 = 20

Example: Logistic Regression with GridSearchCV

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV
import numpy as np
from sklearn.datasets import make_classification

X, y = make_classification(
    n_samples=1000,
    n_features=20,
    n_informative=10,
    n_classes=2,
    random_state=42
)

c_space = np.logspace(-5, 8, 15)

param_grid = {
    'C': c_space,
    'penalty': ['l1', 'l2']
}

logreg = LogisticRegression(solver='liblinear')

logreg_cv = GridSearchCV(
    logreg,
    param_grid,
    cv=5
)

logreg_cv.fit(X, y)

print(
    "Tuned Logistic Regression Parameters: {}".format(
        logreg_cv.best_params_
    )
)

print(
    "Best score is {}".format(
        logreg_cv.best_score_
    )
)

Output

Tuned Logistic Regression Parameters:
{'C': 0.006105402296585327}

Best score is 0.853

The best score of 0.853 means the model achieved 85.3% validation accuracy using the optimal hyperparameter combination.

2. RandomizedSearchCV

RandomizedSearchCV randomly selects combinations of hyperparameters instead of checking every possible combination like GridSearchCV.

Working Process

1. Select random hyperparameter combinations.
2. Train and evaluate the model.
3. Track the performance of each configuration.
4. Choose the best-performing setup.

Example: Decision Tree with RandomizedSearchCV

import numpy as np
from sklearn.datasets import make_classification

X, y = make_classification(
    n_samples=1000,
    n_features=20,
    n_informative=10,
    n_classes=2,
    random_state=42
)

from scipy.stats import randint
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import RandomizedSearchCV

param_dist = {
    "max_depth": [3, None],
    "max_features": randint(1, 9),
    "min_samples_leaf": randint(1, 9),
    "criterion": ["gini", "entropy"]
}

tree = DecisionTreeClassifier()

tree_cv = RandomizedSearchCV(
    tree,
    param_dist,
    cv=5
)

tree_cv.fit(X, y)

print(
    "Tuned Decision Tree Parameters: {}".format(
        tree_cv.best_params_
    )
)

print(
    "Best score is {}".format(
        tree_cv.best_score_
    )
)

Output

Tuned Decision Tree Parameters:
{
  'criterion': 'entropy',
  'max_depth': None,
  'max_features': 6,
  'min_samples_leaf': 6
}

Best score is 0.8

A score of 0.8 means the tuned model achieved 80% validation accuracy with the selected hyperparameters.

3. Bayesian Optimization

Grid Search and Random Search can be inefficient because they try many unnecessary combinations.

Bayesian Optimization uses previous results to intelligently decide which hyperparameter combination should be tested next.

Working Process

1. Build a probabilistic surrogate model.
2. Predict performance using previous evaluations.
3. Select the most promising hyperparameters.
4. Update the model after every evaluation.

Surrogate Function

P(score(y) | hyperparameters(x))

Here, x represents hyperparameters and y represents model performance.

Common Surrogate Models

• Gaussian Processes
• Random Forest Regression
• Tree-structured Parzen Estimators (TPE)

Advantages of Hyperparameter Tuning

• Improves model accuracy and robustness.
• Helps prevent overfitting and underfitting.
• Improves generalization on unseen data.
• Optimizes computational resources efficiently.
• Makes models easier to interpret and understand.