Language & LLMs

What Is RoBERTa?

RoBERTa, short for Robustly Optimized BERT Approach, is a language model that refines BERT's pretraining procedure. It trains on more data for longer, uses larger batches, and removes the next-sentence prediction objective. These changes improve its performance on many language understanding tasks.