Describe the limitations of ReLU activation functions and explain how alternative activation functions, such as leaky ReLU and ELU, address these limitations.

ReLU (Rectified Linear Unit) is a popular activation function in deep learning due to its simplicity and efficiency in computation. It outputs the input directly if it is positive; otherwise, it outputs zero. Mathematically, ReLU(x) = max(0, x). Despite its advantages, ReLU suffers from several limitations, most notably the "dying ReLU" problem. Limitations of ReLU: 1. Dying ReLU: The most significant limitation of ReLU is the "dying ReLU" problem. This occurs when a ReLU neuron gets stuck in the inactive state, always outputting zero, for all inputs. This can happen if the neuron receives a large negative gradient during training, which pushes the weights such that the neuron never activates for any input in the training set. Once a neuron enters this state, it becomes permanently inactive and stops contributing to the learning process. This can effectively reduce the capacity of the network and hinder its ability to learn complex patterns. Example: Consider a neuron with a large negative bias. Even if the input is positive, the sum of the weighted inputs plus the bias might still be negative, causing the ReLU to output zero. If this happens consistently during training, the neuron will never update its weights and effectively becomes useless. 2. Non-Zero Centered Output: ReLU outputs values that are always positive or zero, meaning its output is not zero-centered. This can lead to slower convergence during training because the gradients can be biased in one direction. This is especially true for deeper layers. 3. Unbounded Activation: ReLU has an unbounded positive activation, which can lead to exploding gradients, especially in deep networks. While ReLU can help mitigate the vanishing gradient problem compared to sigmoid or tanh, it does not inherently prevent exploding gradients. Alternative Activation Functions: To address the limitations of ReLU, several alternative activation functions have been proposed, including Leaky ReLU and ELU (Exponential Linear Unit). 1. Leaky ReLU: Leaky ReLU addresses the dying ReLU problem by allowing a small, non-zero gradient when the input is negative. Mathematically, Leaky ReLU(x) = x if x > 0, and Leaky ReLU(x) = alpha x if x <= 0, where alpha is a small constant, typically between 0.01 and 0.1. This small slope for negative inputs ensures that the neuron remains active, even when the input is negative, preventing it from getting stuck in the inactive state. Advantages: - Prevents Dying ReLU: The small non-zero gradient for negative inputs ensures that the neuron continues to learn, even when the input is negative. - Simple to Implement: Leaky ReLU is easy to implement and adds minimal computational overhead compared to ReLU. Disadvantages: - Choice of Alpha: The performance of Leaky ReLU can be sensitive to the choice of the alpha parameter. - Not Zero Centered: Leaky ReLU, although improved, is still not truly zero-centered. Example: If alpha = 0.01, then for x = -10, Leaky ReLU(x) = -0.1. This ensures that the neuron is still responsive to negative inputs and can update its weights accordingly. If it was a normal ReLU, the neuron would've outputted 0, causing the Dying ReLU problem. 2. ELU (Exponential Linear Unit): ELU is another alternative to ReLU that addresses both the dying ReLU problem and the non-zero-centered output issue. ELU is defined as ELU(x) = x if x > 0, and ELU(x) = alpha (exp(x) - 1) if x <= 0, where alpha is a positive constant. The exponential term for negative inputs allows ELU to have negative values, which helps to center the output around zero. It also allows for the saturation to a negative value, helping the neurons to learn more robust features. Advantages: - Prevents Dying ReLU: Like Leaky ReLU, ELU prevents the dying ReLU problem by having non-zero output for negative inputs. - Near Zero Centered: ELU has a near-zero mean output, which can accelerate learning by reducing the bias in the gradients. - Saturation for Negative Values: The saturation of ELU for negative values can make the network more robust to noise. Disadvantages: - Computational Complexity: ELU is slightly more computationally expensive than ReLU and Leaky ReLU due to the exponential term. - Choice of Alpha: The performance of ELU can be sensitive to the choice of the alpha parameter. Example: If alpha = 1.0, then for x = -1, ELU(x) = 1.0 (exp(-1) - 1) = -0.632. This allows the neuron to output negative values, which helps to center the output around zero. If it was a normal ReLU, the neuron would've outputted 0, causing the Dying ReLU problem. Comparison: Dying ReLU: ReLU suffers from it; Leaky ReLU and ELU mitigate it. Zero-Centered Output: ReLU does not have zero-centered output; Leaky ReLU improves it, and ELU approximates it better. Computational Cost: ReLU is the cheapest; Leaky ReLU adds minimal overhead, and ELU has slightly higher computational cost. Parameter Sensitivity: Leaky ReLU and ELU depend on ....