Explain the role of activation functions in neural networks and provide examples of commonly used activation functions.

Activation functions play a crucial role in neural networks by introducing non-linearity to the network's computations. They determine the output of a neuron based on the weighted sum of its inputs. Activation functions help neural networks learn complex patterns and make predictions by introducing non-linear relationships between inputs and outputs.

The main purpose of an activation function is to introduce non-linearity, allowing the neural network to model and approximate non-linear functions effectively. Without activation functions, neural networks would be limited to representing only linear functions, severely restricting their ability to learn and solve complex problems.

There are several commonly used activation functions in neural networks, each with its own characteristics and applicability in different scenarios. Here are some examples:

1. Sigmoid Activation Function:
The sigmoid function is a popular choice for binary classification problems. It maps the input to a range between 0 and 1, making it suitable for representing probabilities. The sigmoid function is defined as f(x) = 1 / (1 + e^-x).
2. Hyperbolic Tangent (Tanh) Activation Function:
The hyperbolic tangent function is similar to the sigmoid function but maps the input to a range between -1 and 1. It is often used in hidden layers of neural networks and can capture both positive and negative activations. The tanh function is defined as f(x) = (e^x - e^-x) / (e^x + e^-x).
3. Rectified Linear Unit (ReLU) Activation Function:
The ReLU function is widely used in deep learning models. It introduces sparsity and non-linearity to the network by returning the input directly if it is positive, and 0 otherwise. The ReLU function is defined as f(x) = max(0, x).
4. Leaky ReLU Activation Function:
The Leaky ReLU function is a variation of the ReLU function that allows small negative values instead of setting them to 0. It helps mitigate the "dying ReLU" problem where neurons can become inactive during training. The Leaky ReLU function is defined as f(x) = max(ax, x), where a is a small constant.
5. Softmax Activation Function:
The softmax function is commonly used in the output layer of neural networks for multi-class classification problems. It transforms the output into a probability distribution, ensuring that the sum of the outputs is equal to 1. The softmax function is defined as f(x\_i) = e^x\_i / (sum(e^x\_j) for j in all outputs).

These are just a few examples of activation functions used in neural networks. Each activation function has its own characteristics and affects how information flows through the network. The choice of activation function depends on the specific problem at hand, the type of data, and the desired properties of the network.

Activation functions are essential components of neural networks as they introduce non-linearity, enabling the network to model complex relationships and make accurate predictions. Their selection and proper tuning contribute significantly to the network's performance and its ability to learn and generalize from data.