A model has many weights, and some are very large, making it too complex. What technique adds a penalty to make these weights smaller during training?

The technique that adds a penalty to make weights smaller during training to reduce model complexity is called Regularization. When a model has many very large weights, it can become overly sensitive to small fluctuations in the training data, leading to overfitting. Overfitting means the model performs exceptionally well on the training data but poorly on new, unseen data because it has learned the noise or specific patterns of the training set too closely rather than generalizing. Regularization addresses this by adding a penalty term to the model's loss function. The loss function quantifies how well the model is performing, with lower values indicating better performance. During training, the model tries to minimize this loss. By adding a penalty for large weights to the loss function, the optimization process is incentivized to find a balance between fitting the training data accurately and keeping the weights small. There are two primary types of regularization for this purpose: L1 Regularization (also known as Lasso Regularization) and L2 Regularization (also known as Ridge Regularization). L2 Regularization adds the sum of the squared values of all the weights to the loss function. The penalty term is proportional to the sum of (weight_i)^2 for all weights 'i'. This encourages the weights to be small but rarely drives them exactly to zero. It spreads the impact across all features, making the model less sensitive to any single feature and improving its generalization ability. L1 Regularization adds the sum of the absolute values of all the weights to the loss function. The penalty term is proportional to the sum of |weight_i| for all weights 'i'. A unique characteristic of L1 regularization is that it can drive some weights to become exactly zero. This effectively performs feature selection by eliminating less important features from the model, leading to a sparser model with fewer active components. In both L1 and L2 regularization, the strength of the penalty is controlled by a hyperparameter, often denoted as lambda (λ) or alpha (α). A larger lambda value imposes a stronger penalty, forcing weights to be even smaller, potentially leading to underfitting (where the model is too simple and cannot capture the underlying patterns). A smaller lambda value reduces the penalty, allowing weights to be larger and potentially leading to overfitting. The optimal lambda is typically found through techniques like cross-validation. By penalizing large weights, regularization effectively reduces the model's complexity, making it less prone to overfitting and improving its ability to generalize to new data.