Core Concepts

What Is a Mixture of Experts (MoE)?

A mixture of experts (MoE) is a model design in which many specialized sub-networks (“experts”) exist, but only a few are activated for any given input by a routing mechanism. This lets a model have very large total capacity while using less computation per query, and it features in several recent high-performing models.