What is Mixture of Experts?

Definition

Mixture of Experts (MoE) is a model architecture where the network is divided into multiple specialized sub-networks, called experts, along with a gating mechanism (router) that selects which experts process each input token. Instead of every parameter being active for every input, MoE models activate only a fraction of their total parameters per forward pass — typically 2 out of 8 or 16 experts. This sparse activation pattern allows MoE models to have much larger total parameter counts (and thus greater knowledge capacity) while keeping per-token compute cost comparable to a much smaller dense model.

The most prominent MoE language model is Mixtral 8x7B from Mistral AI, which contains 8 expert feed-forward networks in each transformer layer. For each token, the router selects the top 2 experts, meaning only about 13B of the model's 47B total parameters are active per token. This gives Mixtral the knowledge capacity of a 47B model with the inference speed of a 13B model — an attractive trade-off.

MoE architectures have been explored since the 1990s but gained practical significance with the scale of modern LLMs. Google's Switch Transformer and GLaM models demonstrated that MoE could scale to trillions of parameters, and open-source implementations like Mixtral proved that MoE models could match or exceed dense models of similar compute cost. The architecture is now widely adopted across frontier labs, with GPT-4 rumored to use a MoE design.

Why It Matters

As language models scale, the compute cost of dense architectures becomes prohibitive. Doubling the parameters of a dense model roughly doubles both training and inference cost. MoE breaks this relationship by allowing parameter count to scale independently of compute cost. This makes it possible to build models with enormous knowledge capacity — important for multilingual, multi-domain applications — without requiring proportionally enormous GPU clusters for inference.

For practitioners, MoE models offer better quality-per-dollar at inference time. A Mixtral 8x7B model outperforms Llama 2 70B on many benchmarks while being significantly cheaper to run. This cost-performance advantage makes MoE models particularly attractive for production deployments where inference cost directly impacts profitability.

How It Works

In each MoE transformer layer, the standard feed-forward network (FFN) is replaced by N parallel expert FFNs and a gating network. The gating network takes the hidden state of each token as input and outputs a probability distribution over the N experts. The top-k experts (usually k=2) with the highest gating scores are selected, and their outputs are combined as a weighted sum according to the gating scores.

Training MoE models requires careful load balancing to prevent expert collapse — a failure mode where the router learns to send all tokens to a small number of experts while the rest remain untrained. Auxiliary load-balancing losses encourage the router to distribute tokens evenly across experts. During inference, efficient MoE implementations use specialized kernels that route tokens to the selected experts without wasting compute on inactive experts, achieving near-linear speedups over equivalently-sized dense models.

Example Use Case

A multilingual content platform deploys Mixtral 8x7B to handle customer queries in 12 languages. The MoE architecture naturally develops language-specialized experts during training — some experts activate primarily for Romance languages, others for Germanic or Asian languages. This implicit specialization delivers better multilingual performance than a dense 13B model while maintaining comparable inference costs, and the 47B total parameter count ensures sufficient knowledge capacity across all supported languages.

Key Takeaways

• MoE models use a router to activate only a subset of expert sub-networks per input, reducing compute cost.
• Total parameter count can be 3-8x larger than the active parameter count per token.
• MoE achieves better quality-per-dollar than dense models at equivalent compute budgets.
• Load balancing during training prevents expert collapse where some experts go unused.
• Models like Mixtral 8x7B demonstrate MoE's viability for open-source LLM deployment.