Language & LLMs

What Is Cross-Attention?

Cross-attention is an attention mechanism where queries come from one sequence and keys and values come from a different sequence. It is commonly used in encoder-decoder models so the decoder can focus on relevant parts of the encoder's output. This is essential for tasks like machine translation.