Describe the mathematical process of calculating attention weights in self-attention.

The mathematical process of calculating attention weights in self-attention involves three key steps: calculating the similarity between each word's query and all other words' keys, scaling the resulting similarity scores, and applying a softmax function to obtain probabilities representing the attention weights. First, each word in the input sequence is transformed into three vectors: a query (q), a key (k), and a value (v). These vectors are typically obtained by multiplying the word's embedding by learned weight matrices. The query represents what the word is "looking for," the key represents what the wo....