Mixture of Experts (MoE): How Transformers Scale Without Activating Everything.

• Mixture of Experts (MoE) is one of the main techniques used to scale modern language models without making every token pay the full computational cost of the model.
• The basic idea is surprisingly simple: instead of sending every token through one enormous feed-forward network, we split it into many smaller expert networks and only activate a few experts for each token.
• These notes walk through the intuition behind MoE, starting from the role of the feed-forward network (FFN) inside a Transformer, then moving through routing, top-$k$ selection, fine-grained experts, shared experts, expert capacity, dropless MoE, load balancing, and router stability.
• The focus is not just on what MoE does, but why each piece exists and what problem it is trying to solve.

I originally learned this material from Jia-Bin Huang’s visual explanation of MoE ¹. I have rewritten the ideas here as notes for myself, with the equations and implementation details that I found useful when trying to understand how modern sparse MoE models actually work.

Table of Contents

• Feed-Forward Networks in Transformers
  • RMSNorm
  • The FFN as a Knowledge Store
• Why Mixture of Experts?
• Sparse Mixture of Experts
  • The Router
  • Top-$k$ Routing
• Fine-Grained Experts
• Shared Experts
• Expert Capacity & Token Overflow
• Dropless MoE
• The Load Balancing Problem
  • Noisy Top-$k$ Gating
  • Importance vs Load
  • Load Balancing Loss
  • Device-Level Load Balancing
• Auxiliary-Loss-Free Load Balancing
• Router Stability & the Router Z-Loss
• Putting It All Together

Appendix

• References
• Citation

Feed-Forward Networks in Transformers

A Transformer layer typically alternates between an attention mechanism and a feed-forward network (FFN).

Attention allows each token to incorporate information from other tokens in the sequence. The FFN then processes each token independently.

For a single token embedding $x$, a simplified FFN can be written as:

where:

• $W_{\text{up}}$ projects the token into a larger hidden dimension.
• $\sigma$ is the nonlinear activation function.
• $W_{\text{down}}$ projects the representation back to the original model dimension.

Typically,

where $d$ is the input dimension and $d_h$ is the hidden dimension of the FFN.

RMSNorm

Before the FFN, modern Transformers commonly use a normalization layer such as RMSNorm.

For an input vector $x$:

and

where $\gamma$ is a learnable scaling vector.

The important idea is that normalization keeps the magnitude of activations under control while allowing the model to learn a different scale for each dimension.

The FFN as a Knowledge Store

The FFN is more than just a generic nonlinear transformation.

One useful interpretation is that the first projection asks a collection of learned questions about the token representation.

Each row of $W_{\text{up}}$ can be thought of as a learned direction in representation space.

If $z_i$ is large, the input strongly matches the feature represented by that row.

The activation function then suppresses irrelevant features.

For a simple ReLU example:

The second projection maps the activated features back into the model dimension.

This gives us an intuitive picture:

The FFN can be thought of as a large collection of learned feature detectors that activate different pieces of stored information depending on the input.

This interpretation is useful for understanding why increasing the FFN hidden dimension can improve model capacity.

But there is a problem.

Why Mixture of Experts?

If we simply increase $d_h$, the FFN becomes larger.

That gives the model more capacity, but it also increases:

• training computation,
• inference computation,
• parameter memory,
• communication requirements.

And there is another observation:

A token does not need every feature in the FFN.

For example, a token about chemistry probably does not need every feature that might be useful for programming, mathematics, or another language.

So instead of making one enormous FFN that processes every token, we can divide the FFN into multiple smaller networks.

Each one becomes an expert.

The model can then choose which experts should process each token.

This is the central idea behind a sparse Mixture of Experts (MoE).

Sparse Mixture of Experts

Suppose we have $N$ experts:

Each expert is itself an FFN.

Instead of evaluating all $N$ experts for every token, we select only the top $k$ experts.

If $k \ll N$, then the model can contain many more parameters while only activating a small fraction of them for each token.

This creates an important distinction:

Total parameters and active parameters are no longer the same thing.

A model can therefore have a very large parameter count without requiring every token to use the entire model.

The Router

The model needs some mechanism to decide which experts should receive each token.

This is the job of the router.

For an input token representation $x$, the router computes a score for every expert:

where

and therefore

Each value in $h(x)$ is the router’s score, or logit, for one expert.

We can turn these logits into probabilities using softmax:

The resulting vector represents the router’s preference over the experts.

Top-$k$ Routing

We then select only the $k$ experts with the highest routing scores.

Let

Then the MoE output can be written as:

In practice, implementations may renormalize the selected routing weights, but the basic idea remains the same.

The process is therefore:

The routing happens at the token level.

Different tokens in the same sequence can therefore be sent to completely different combinations of experts.

Fine-Grained Experts

A natural question is:

Why not just have a small number of large experts?

One answer is flexibility.

Suppose we have eight large experts and select two of them. The number of possible combinations is limited.

Instead, we can split the expert computation into many smaller experts and activate more of them.

This is the idea behind fine-grained expert segmentation.

DeepSeekMoE, for example, proposed splitting the experts into a much larger number of smaller experts and activating a correspondingly larger number of them. The goal is to allow more flexible combinations of specialized knowledge. ²

This gives the router a finer-grained set of choices:

This is one of the important ideas behind modern DeepSeek-style MoE architectures.

Shared Experts

DeepSeekMoE also introduced the idea of shared experts.

A shared expert is always active rather than being selected by the router.

The intuition is that some information is useful for almost every token.

Instead of forcing the routed experts to repeatedly learn this common information, we can give the model a dedicated expert for broadly useful patterns.

The architecture therefore contains:

The shared component handles more general information, while the routed experts can specialize.

The DeepSeekMoE paper found benefits from combining fine-grained experts with shared experts, although the value of shared experts is architecture-dependent and is not universally guaranteed. ²

Expert Capacity & Token Overflow

There is a practical problem with routing.

Suppose we have $16$ tokens and $8$ experts.

If tokens were distributed perfectly evenly, each expert would receive:

tokens.

So we could give every expert a capacity of two.

But routing decisions depend on the actual input.

We might instead get something like:

Now the first expert has received more tokens than it can process.

This creates token overflow.

One traditional solution is to increase the expert’s capacity.

For example, a capacity factor of $1.5$ would increase the available capacity above the ideal balanced allocation.

But larger capacity means:

• more memory,
• more computation,
• more padding,
• more communication.

So we have a trade-off:

So can we avoid both?

Dropless MoE

Instead of forcing every expert to process the same number of tokens, we can organize the expert computation around the actual routing assignments.

One way to think about this is using block-sparse matrix multiplication.

Rather than padding every expert to the same token count, the computation can use blocks whose sizes correspond to the number of tokens actually assigned to each expert.

Conceptually:

This allows us to avoid dropping tokens simply because an expert received more tokens than expected, while also avoiding unnecessary padding.

This style of implementation is often referred to as dropless MoE.

The exact implementation depends heavily on the GPU kernels and distributed training system, but the underlying goal is simple:

Compute only the expert work that actually exists.

The Load Balancing Problem

There is a deeper problem with sparse routing.

At the beginning of training, all experts are randomly initialized.

Suppose the router happens to send many of the first tokens to experts $E_1$ and $E_2$.

Those experts are updated more often.

They become better.

The router then has even more reason to send tokens to them.

This can create a feedback loop:

Eventually, some experts may receive very few tokens.

These underused experts become effectively dead.

This is the load balancing problem.

The goal is not necessarily to make every expert equally good.

The goal is to prevent the router from collapsing onto a small subset of experts.

Noisy Top-$k$ Gating

One early approach is to add noise to the router logits:

The noise encourages exploration.

Instead of always selecting the same experts, the router occasionally explores other experts, giving them opportunities to receive tokens and learn.

This idea appears in noisy top-$k$ gating approaches to MoE routing. ³

But exploration alone is not enough.

We still need an explicit mechanism to encourage balanced expert utilization.

Importance vs Load

There are two slightly different things we can measure.

Importance

We can measure how much routing probability an expert receives.

For expert $i$:

An expert can therefore have high importance even if it is not selected very often.

Load

Instead, we can count how many tokens are actually routed to the expert:

This measures actual expert usage.

These two quantities are not necessarily the same.

For example, the router might give eight experts reasonably balanced probabilities while repeatedly selecting only four of them through top-$k$ routing.

So balancing probabilities alone does not guarantee balanced computation.

Load Balancing Loss

A common approach is to add an auxiliary load balancing loss to the language modeling objective.

Conceptually:

where:

• $L_{\text{LM}}$ is the normal next-token prediction loss.
• $L_{\text{balance}}$ encourages more uniform expert usage.
• $\alpha$ controls how strongly the balancing objective affects training.

A commonly used formulation combines the fraction of tokens routed to each expert with the average routing probability assigned to that expert:

where:

• $f_i$ is the fraction of tokens routed to expert $i$.
• $p_i$ is the average router probability assigned to expert $i$.
• $N$ is the number of experts.

The factor $N$ keeps the scale of the loss comparable as the number of experts changes.

If routing is perfectly uniform:

and therefore:

The important idea is that the balancing loss gives the router a reason to spread tokens across experts instead of collapsing onto a few.

However, there is an obvious downside.

If $\alpha$ is too large, the model may optimize for balanced routing at the expense of the actual language modeling objective.

If $\alpha$ is too small, the balancing loss may have almost no effect.

So:

Device-Level Load Balancing

In a real distributed MoE model, experts are often spread across different GPUs.

This introduces another problem.

Even if the experts are balanced globally, the devices might not be.

For example:

The model is still bottlenecked by the overloaded GPU.

Therefore, large-scale MoE systems may also consider load at the device level.

The same general principle applies:

This matters because MoE introduces communication between devices whenever tokens need to be dispatched to experts located on different GPUs.

Auxiliary-Loss-Free Load Balancing

DeepSeek-V3 introduced an interesting alternative to the traditional auxiliary balancing loss.

Instead of adding another loss term that directly competes with the language modeling objective, DeepSeek-V3 uses a dynamic expert-level bias to influence which experts are selected. ⁴

The idea is roughly:

1. Measure how many tokens each expert is receiving.
2. Compare each expert’s load with the average.
3. Increase the routing bias for under-utilized experts.
4. Decrease the routing bias for overloaded experts.
5. Use the adjusted scores for top-$k$ selection.
6. Keep the original router scores for weighting the selected expert outputs.

Conceptually:

where $b_i$ is a dynamically updated expert-specific bias.

If an expert is overloaded:

If an expert is underloaded:

The key detail is that this bias is used for routing selection, rather than changing the probabilities used to combine the final expert outputs.

This allows DeepSeek-V3 to perform load balancing without introducing the same auxiliary-loss trade-off. The DeepSeek-V3 technical report describes this as an auxiliary-loss-free load balancing strategy. ⁴

Router Stability & the Router Z-Loss

Load balancing is not the only issue with the router.

Recall that the router begins with logits:

and converts them to probabilities using softmax:

An interesting property of softmax is that it is shift invariant.

For any constant $c$:

because:

This creates a subtle problem.

The router can increase all of its logits by the same amount without changing its output probabilities.

So the probabilities remain stable while the underlying logits can grow arbitrarily large.

This becomes dangerous when using reduced-precision arithmetic such as FP16.

Safe Softmax

A standard numerical trick is to subtract the maximum logit before exponentiation:

This prevents excessively large exponentials from overflowing.

But it only treats the numerical symptom.

The underlying logits can still drift.

Router Z-Loss

A more direct approach is to regularize the normalization term.

Define:

Then the router Z-loss can be written as:

where $B$ is the batch size.

The logarithm prevents the regularization term from growing exponentially, while the square penalizes large positive or negative shifts.

The intuition is:

Keep the router logits numerically well-behaved without unnecessarily changing which experts the router prefers.

This is particularly important when scaling MoE training to large models.

Putting It All Together

The entire sparse MoE pipeline can now be summarized as:

But making this work at scale requires solving several different problems:

The thing I find most interesting about MoE is that the main idea is actually very simple:

Build a much larger network, but only use a small part of it for each token.

The difficult part is everything that comes afterward.

The router has to learn where each token should go. The experts have to specialize without some of them becoming useless. The tokens have to be distributed across GPUs without creating communication bottlenecks. And the whole routing system has to remain numerically stable while the model is being trained.

That is what makes modern MoE architectures more than simply “a bunch of FFNs with a router.”

Appendix

Citation

If you found this blog post helpful, please consider citing it:

@article{obasi2026mixtureOfExperts,
  title   = "Mixture of Experts (MoE): How Transformers Scale Without Activating Everything",
  author  = "Obasi, Chizoba",
  journal = "chizkidd.github.io",
  year    = "2026",
  month   = "Aug",
  url     = "https://chizkidd.github.io/2026/08/10/mixture-of-experts/"
}

References