Language & LLMs

What Is Byte-Pair Encoding?

Byte-pair encoding (BPE) is a data compression and tokenization technique that repeatedly merges the most frequent pairs of characters or byte sequences into single tokens. In language models, it creates a subword vocabulary that balances vocabulary size with the ability to represent rare and unknown words. BPE is widely used by models like GPT to break text into manageable tokens.