Language & LLMs

What Is Masked Language Modeling?

Masked language modeling is a pretraining objective where some tokens in the input are hidden and the model learns to predict them from the surrounding context. Because it can use both left and right context, it produces bidirectional representations. BERT popularized this approach.