Language & LLMs

What Is Self-Attention?

Self-attention is a mechanism that allows each token in a sequence to attend to, or focus on, all other tokens when building its representation. It computes relevance scores between tokens and uses them to combine information across the sequence. Self-attention is the core operation inside transformer models.