ElasticNet Regression: Method & Codes

ElasticNet regression is a regularized regression method that linearly combines both L1 and L2 penalties of the Lasso and Ridge methods. This allows it to perform both feature selection (like Lasso) and maintain some of the benefits of Ridge (such as handling multicollinearity). The objective function for ElasticNet is:

where are the actual values, are the predicted values, are the coefficients, is the number of observations, is the number of features, and and are the regularization parameters for the L1 and L2 penalties, respectively.

Implementation:

In the following codes, we will:

1. Import/Load Packages: Necessary packages are imported or installed (glmnet, caret, e1071 for R; numpy, matplotlib, scikit-learn for Python).
2. Load Dataset: Synthetic datasets are created with 100 samples and 20 features.
3. Split Dataset: The datasets are split into training and test sets.
4. Train Elastic Net Model: Elastic Net regression models are trained using cv.glmnet for R and ElasticNet for Python. The alpha parameter in cv.glmnet (R) and l1_ratio in ElasticNet (Python) are used to control the mix ratio between L1 and L2 penalties.
5. Evaluate Model: The model’s performance is evaluated using mean squared error (MSE) and R² score.
6. Visualize Results: Coefficients of the Elastic Net model are plotted to visualize the effect of combined L1 and L2 regularization on the feature coefficients.

Python Code:

# Step 1: Import the necessary libraries
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.linear_model import ElasticNet
from sklearn.metrics import mean_squared_error, r2_score

# Step 2: Load the dataset (using a synthetic dataset for this example)
X, y = make_regression(n_samples=100, n_features=20, noise=0.1, random_state=42)

# Step 3: Split the dataset into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Step 4: Train the Elastic Net regression model
alpha = 1.0  # Regularization strength
l1_ratio = 0.5  # Mix ratio between L1 and L2 (0: Ridge, 1: Lasso)
elastic_net = ElasticNet(alpha=alpha, l1_ratio=l1_ratio, random_state=42)
elastic_net.fit(X_train, y_train)

# Step 5: Evaluate the model
y_train_pred = elastic_net.predict(X_train)
y_test_pred = elastic_net.predict(X_test)

print("Training set evaluation:")
print("Mean Squared Error:", mean_squared_error(y_train, y_train_pred))
print("R^2 Score:", r2_score(y_train, y_train_pred))

print("\nTest set evaluation:")
print("Mean Squared Error:", mean_squared_error(y_test, y_test_pred))
print("R^2 Score:", r2_score(y_test, y_test_pred))

# Step 6: Visualize the results (if applicable, here we'll plot the coefficients)
plt.figure(figsize=(10, 6))
plt.plot(elastic_net.coef_, marker='o', linestyle='none')
plt.title('Elastic Net Regression Coefficients')
plt.xlabel('Feature Index')
plt.ylabel('Coefficient Value')
plt.grid(True)
plt.show()

R Code:

# Step 1: Install and load the necessary packages
install.packages("glmnet")
install.packages("caret")
install.packages("e1071")
library(glmnet)
library(caret)
library(e1071)

# Step 2: Load the dataset (using a synthetic dataset for this example)
set.seed(42)
n <- 100
p <- 20
X <- matrix(rnorm(n * p), n, p)
beta <- rnorm(p)
y <- X %*% beta + rnorm(n) * 0.1

# Step 3: Split the dataset into training and test sets
trainIndex <- createDataPartition(y, p = 0.8, list = FALSE)
X_train <- X[trainIndex, ]
X_test <- X[-trainIndex, ]
y_train <- y[trainIndex]
y_test <- y[-trainIndex]

# Step 4: Train the Elastic Net regression model
alpha <- 0.5  # Mix ratio between L1 and L2 (0: Ridge, 1: Lasso)
elastic_net_model <- cv.glmnet(X_train, y_train, alpha = alpha)

# Step 5: Evaluate the model
y_train_pred <- predict(elastic_net_model, X_train, s = "lambda.min")
y_test_pred <- predict(elastic_net_model, X_test, s = "lambda.min")

train_mse <- mean((y_train - y_train_pred)^2)
test_mse <- mean((y_test - y_test_pred)^2)
train_r2 <- 1 - sum((y_train - y_train_pred)^2) / sum((y_train - mean(y_train))^2)
test_r2 <- 1 - sum((y_test - y_test_pred)^2) / sum((y_test - mean(y_test))^2)

cat("Training set evaluation:\n")
cat("Mean Squared Error:", train_mse, "\n")
cat("R^2 Score:", train_r2, "\n")

cat("\nTest set evaluation:\n")
cat("Mean Squared Error:", test_mse, "\n")
cat("R^2 Score:", test_r2, "\n")

# Step 6: Visualize the results (if applicable, here we'll plot the coefficients)
elastic_net_coefficients <- coef(elastic_net_model, s = "lambda.min")

plot(elastic_net_coefficients, main = "Elastic Net Regression Coefficients", xlab = "Feature Index", ylab = "Coefficient Value", pch = 16, col = "blue")