Regularization in Machine Learning | Machine Learning Simplified

What is Regularization in Machine Learning?

Regularization is an essential technique in machine learning used to prevent overfitting of the model. Overfitting happens when a model fits too well to the training data, including the noise or random fluctuations in the data, and performs poorly on new, unseen data. Regularization is used to overcome this problem by adding a penalty term to the objective function that the model is trying to minimize. In this blog post, we will discuss regularization in machine learning, its importance, and how it can be implemented using TensorFlow.

Regularization in Machine Learning and its types:

In machine learning, the primary goal is to build a model that can make accurate predictions on new, unseen data. However, sometimes the model tends to memorize the training data, including the noise or random fluctuations, and perform poorly on new data. This problem is known as overfitting. Overfitting can happen in any machine learning algorithm, including linear regression, logistic regression, decision trees, and neural networks.

To overcome overfitting, regularization is used. Regularization is a technique that adds a penalty term to the objective function that the model is trying to minimize. The penalty term depends on the complexity of the model. In other words, the penalty term is added to the objective function to discourage the model from fitting the training data too well and encourages it to learn a more generalized pattern that is applicable to new data.

There are two types of regularization: L1 regularization (also known as Lasso) and L2 regularization (also known as Ridge). In L1 regularization, the penalty term is the sum of the absolute values of the model parameters. In L2 regularization, the penalty term is the sum of the squares of the model parameters. L1 regularization is used when we want to perform feature selection, i.e., identify the most important features in the data. L2 regularization is used when we want to avoid large weights in the model and simplify the model.

L1 Regularization (Lasso):

L1 regularization also known as Lasso regularization is often used when there are many features in the data, and only a few of them are expected to be important. L1 regularization encourages the model to have sparse weights, meaning that many of the weights will be exactly zero, and only a few of them will be non-zero. As a result, L1 regularization can be used for feature selection, where we want to identify the most important features in the data.

In L1 regularization, the penalty term is the sum of the absolute values of the model parameters. Mathematically, L1 regularization can be expressed as

$J(w) = J_0(w) + \alpha \cross \sum |w_i|$

Where $J_0(w)$ is the original cost function without regularization, $w_i$ is the $i^{th}$ model parameter, and $\alpha$ is the regularization parameter that controls the strength of regularization.

The term $\sum |w_i|$ is the $L1$ penalty term, which is added to the original cost function. This term encourages the model to have sparse weights, i.e., many of the weights will be exactly zero. As a result, $L1$ regularization can be used for feature selection, i.e., identifying the most important features in the data.

TensorFlow Example on L1 Regularization:

Here is an example of L1 regularization in TensorFlow using the Boston Housing dataset:

import tensorflow as tf
from tensorflow.keras.datasets import boston_housing
from tensorflow.keras import models, layers, regularizers
import numpy as np

# Load the Boston Housing dataset
(train_data, train_targets), (test_data, test_targets) = boston_housing.load_data()

# Normalize the data
mean = train_data.mean(axis=0)
std = train_data.std(axis=0)
train_data = (train_data - mean) / std
test_data = (test_data - mean) / std

# Define the model with L1 regularization
model = models.Sequential()
model.add(layers.Dense(64, kernel_regularizer=regularizers.l1(0.001), activation='relu', input_shape=(train_data.shape[1],)))
model.add(layers.Dense(64, kernel_regularizer=regularizers.l1(0.001), activation='relu'))
model.add(layers.Dense(1))

# Compile the model
model.compile(optimizer='rmsprop', loss='mse', metrics=['mae'])

# Train the model
history = model.fit(train_data, train_targets, epochs=100, batch_size=16, validation_split=0.2)

# Evaluate the model on the test data
test_mse_score, test_mae_score = model.evaluate(test_data, test_targets)
print('Test MAE:', test_mae_score)

In this example, we load the Boston Housing dataset and normalize the data. We define a neural network with two hidden layers and L1 regularization with a regularization parameter of 0.001. The model is compiled with the mean squared error loss function and the mean absolute error metric. We train the model on the training data for 100 epochs with a batch size of 16 and a validation split of 0.2. Finally, we evaluate the model on the test data and print the mean absolute error score.

Note that in the layers.Dense function, we pass regularizers.l1(0.001) as the kernel_regularizer parameter to specify L1 regularization with a regularization parameter of 0.001. This adds an L1 penalty term to the cost function, which encourages the model to have sparse weights.

By using L1 regularization, the model will tend to have many weights with a value of 0, resulting in a smaller and more interpretable model. L1 regularization can also help prevent overfitting by reducing the complexity of the model.

L2 Regularization (Ridge):

L2 regularization is also known as Ridge regularization. In L2 regularization, the penalty term is the sum of the squares of the model parameters. Mathematically, L2 regularization can be expressed as follows:

$J(w) = J_0(w) + \alpha \cross \sum |w_{i}^{2}$

Where $J_0(w)$ is the original cost function without regularization, $w_i$ is the $i^{th}$ model parameter, and $\alpha$ is the regularization parameter that controls the strength of regularization.

The term $\sum |w_{i}^{2}$ is the $L2$ penalty term, which is added to the original cost function. This term encourages the model to have sparse weights, i.e., many of the weights will be exactly zero. As a result, $L2$ regularization can be used for feature selection, i.e., identifying the most important features in the data.

TensorFlow Example on L2 Regularization:

TensorFlow is an open-source machine learning library developed by Google. TensorFlow provides a simple and efficient way to implement regularization techniques in machine learning models. Here is an example of how to implement L2 regularization in a TensorFlow model:

import tensorflow as tf

# Define the model architecture
model = tf.keras.Sequential([
  tf.keras.layers.Dense(128, activation='relu', kernel_regularizer=tf.keras.regularizers.l2(0.01)),
  tf.keras.layers.Dense(64, activation='relu', kernel_regularizer=tf.keras.regularizers.l2(0.01)),
  tf.keras.layers.Dense(10, activation='softmax')
])

# Define the loss function
loss_fn = tf.keras.losses.SparseCategoricalCrossentropy()

# Compile the model with an optimizer and metrics
model.compile(optimizer='adam', loss=loss_fn, metrics=['accuracy'])

# Train the model with regularization
model.fit(x_train, y_train, epochs=10, validation_data=(x_test, y_test))

In this example, we define a sequential model with two dense layers that use the ReLU activation function and L2 regularization with a regularization factor of 0.01. We then define the loss function, compile the model with an optimizer and metrics, and train the model on the training data with 10 epochs and a validation set. The L2 regularization term is automatically added to the loss function during training and encourages the model to have smaller weights to reduce overfitting.

When to use L1 and L2 regularization?

L1 regularization encourages the model to have sparse weights and can be used for feature selection, while L2 regularization encourages the model to have small weights and can be used to avoid large weights in the model and simplify the model.

In some cases, a combination of L1 and L2 regularization (Elastic Net) can be used, which combines the strengths of both L1 and L2 regularization. Elastic Net regularization can be used when we want to select a subset of features while avoiding overfitting and reducing the impact of correlated features.

In summary, the choice of regularization technique depends on the specific problem and the characteristics of the data. In general, if we expect that only a few features are important, L1 regularization should be used. If we expect that all the features are somewhat important, L2 regularization should be used. If we are uncertain about the importance of features, Elastic Net regularization can be used. It is also important to tune the regularization parameter to find the optimal balance between model complexity and overfitting.