Question

In the context of the Bias-Variance tradeoff, which specific regularization technique—Lasso (L1) or Ridge (L2)—is mathematically capable of forcing the coefficients of irrelevant features to become exactly zero?

Accepted Answer

Lasso (L1) regularization is the specific technique mathematically capable of forcing the coefficients of irrelevant features to become exactly zero. Regularization is a process used in machine learning to prevent overfitting, which occurs when a model learns noise in the data rather than the underlying pattern, by adding a penalty term to the model&#x27;s loss function based on the size of its coefficients. Lasso stands for Least Absolute Shrinkage and Selection Operator. It uses L1 regularization, which adds the sum of the absolute values of the coefficients to the loss function. Geometrically, the constraint region for L1 regularization is a diamond shape with sharp corners located on the axes. When the optimization process attempts to minimize the loss function, the diamond shape is highly likely to intersect the objective function at these axis-aligned corners, where one or more coefficients are exactly zero. This creates a feature selection effect because the model effectively removes the influence of irrelevant features entirely. In contrast, Ridge (L2) regularization adds the sum of the squared values of the coefficients to the loss function. The constraint region for L2 is a circular shape. Because a circle does not have sharp corners on the axes, the optimization process will shrink coefficients toward zero to reduce the penalty, but they will never reach exactly zero. Therefore, while both techniques reduce model variance, only Lasso performs feature selection by setting coefficients to zero.

Home → All Courses → Science and Mathematics Courses → Data Science and Predictive Analytics → Flashcard

In the context of the Bias-Variance tradeoff, which specific regularization technique—Lasso (L1) or Ridge (L2)—is mathematically capable of forcing the coefficients of irrelevant features to become exactly zero?