The Council of Specialists: Mixture of Experts and How AI Models Grow Without Getting Costlier

The Hook: A Model With 671 Billion Parameters That Computes Like One With 37 Billion

In late 2024, a Chinese lab released a language model named DeepSeek-V3. Its spec sheet carried a number that at first looked like a typo: 671 billion parameters. That is a scale one associates with the absolute frontier of the industry, with systems whose training swallows tens of millions of dollars and whose operation keeps entire data centers busy. Yet right beside it stood a second number that seemed not to belong: 37 billion activated parameters per token.

How can a model possess 671 billion parameters but use only 37 billion of them to predict the next word? And why would anyone want to build such a thing in the first place?

The answer to both questions is one of the most consequential architectural ideas in recent AI history. It goes by the name Mixture of Experts, or MoE. It breaks a coupling that stood for decades as an iron law: that the more a model knows, the more expensive it is to run. MoE separates the knowledge a model holds from the computation it spends on each word. An MoE model is like an enormous hospital full of highly specialized physicians – but every patient is deliberately routed, at the front desk, to the two or three specialists they actually need, rather than visiting every single doctor in turn. The hospital can employ arbitrarily many physicians; treating the individual patient nonetheless stays fast and cheap.

This article takes you the whole distance: from the fundamental dilemma that MoE resolves, through the surprisingly old core idea of conditional computation, its technical realization at the heart of the Transformer, the treacherous problem of load balancing, all the way to the concrete models of the present – Mixtral, Switch Transformer, DeepSeek-V3 – and the open questions this architecture still drags along with it. By the end you should not only know what a Mixture-of-Experts model is, but why the entire industry is currently marching in this direction.

Part 1: The Dilemma of Growth

Why Bigger Is Almost Always Better – and Why That Is a Problem

To understand MoE, one has to understand the problem it solves. And that problem begins with one of the most robust empirical observations of modern AI: larger models are better. Increase the number of parameters, enlarge the training data, and pour in more compute, and with dependable regularity you get a more capable model. The relationship is so reliable that it can be cast as power laws – the so-called scaling laws, to which this vault has devoted its own chapter (see How Big Is Big Enough? Scaling Laws, Chinchilla, and the Measurement of AI).

But this blessing carries a price, and the price is relentless. In a classic, so-called dense neural network, every parameter is used in every single computation. Double the parameter count and you double the compute for every single word – during training and, more expensively still, during every later use. A dense model pays its entire size on every token it ever processes. Capacity and cost are welded together.

This leads to a frustrating cul-de-sac. You want to endow a model with more knowledge, more facts, more languages, more skills – but each additional billion parameters also makes it slower and costlier to operate. For a model that will later answer requests billions of times over, this coupling is the true economic ceiling. In the long run the most expensive line item is not training but the ongoing operation, the inference.

The Decisive Question: Does Everything Really Have to Compute?

Here an almost naive question presents itself, one that is at the same time the key to everything: Does the entire model really have to take part in every single prediction?

Think of your own brain. When you solve a mathematical expression, not every neuron in your head fires at once. When you read a poem, different areas are active than during mental arithmetic. Your brain is vast, yet for any concrete task it activates only a fraction of its capacity. It is – in the language of computer science – sparsely activated.

A dense language model does exactly the opposite. Whether the next word is a legal term of art, a line of Python code, or a rhyme in a nursery song, the same enormous network computes at full power. It is as if, for every question no matter how trivial, one convened the entire expert staff of a corporation for a plenary session. The intuition behind Mixture of Experts is precisely this: build a model of enormous total capacity, but for each individual token let only the small, fitting part of it work. This is called conditional computation: which part of the network computes depends on the input.

Part 2: The Core Idea – Conditional Computation and Where It Happens

What an "Expert" Really Is

The term "expert" is both seductive and misleading, so let us clear up a misconception at the very start. One involuntarily pictures that an MoE model contains a "lawyer expert," a "Python expert," and a "poetry expert" – clearly delineated professionals with humanly nameable responsibilities. Reality is not that tidy. An expert in an MoE is, to begin with, nothing more than a small neural sub-network – technically, usually an ordinary forward-directed feed-forward network. What exactly an expert ends up responsible for over the course of training is decided by the model itself, and the resulting specializations are often not cleanly interpretable by humans. I am of the opinion that this very point is too often smoothed over in popular accounts: the "experts" are emergent, statistical responsibilities, not deliberately designed departments.

To understand where these experts sit within the model, we must briefly recall the blueprint of the Transformer – the architecture on which practically every modern language model rests (covered in detail in Attention Is All You Need: The Transformer, Self-Attention, and the Architecture of Modern AI). A Transformer consists of a stack of identically built layers. Each layer has two main components: an attention mechanism, which relates the words of a sentence to one another, and a feed-forward network (FFN), which sends each word individually through a small processing stage. In the large models the lion's share of parameters sits precisely in these feed-forward networks – often two-thirds and more.

And this is exactly where MoE intervenes. The central idea is astonishingly surgical: replace the one large feed-forward network in a Transformer layer with many smaller feed-forward networks – the experts – and place a doorkeeper in front of them who decides, for each word, which experts it may visit. The attention part is left untouched. Only the most expensive, most parameter-rich component is fanned out.

The Doorkeeper: The Gating Network

The doorkeeper is called, in technical parlance, the gating network or router. It is itself a tiny neural network – often just a single weight matrix. For each word that arrives, the router computes a score for each available expert: "How well does this word fit expert 1, expert 2, expert 3 …?" These scores are turned into probabilities by a softmax function, and then the router selects the best k experts – typically the two highest-scoring (top-2), in some architectures only one (top-1), or eight out of hundreds.

Only these selected experts process the word. All the others stay idle; their parameters are never even touched for this token. The outputs of the active experts are finally blended – weighted by their router scores – into a single result that is then carried onward through the layer. Hence the name "mixture": the answer is a mixture of the contributions of a few selected specialists.

The effect is the miracle described at the outset. A model can contain hundreds of experts and thereby exhibit a gigantic total parameter count. But because only a handful of them are active per token, the actual compute corresponds to that of a much smaller, dense model. Capacity scales with the total number of parameters; cost scales only with the active parameters. That is exactly the decoupling that makes MoE so powerful.

Part 3: A Surprisingly Long History

1991: The Idea Before the Words "Deep Learning" Existed

One might think such a modern idea is a child of the 2020s. In fact its root reaches back more than three decades. In 1991, Robert Jacobs, Michael Jordan, Steven Nowlan and – no less a figure than – Geoffrey Hinton published a paper in the journal Neural Computation titled "Adaptive Mixtures of Local Experts." Its basic thought was already fully present: rather than let a single network despair over a complex task, divide the problem among several specialized sub-networks and let a gating network learn which expert is responsible for which input.

The authors demonstrated this on a vowel-discrimination task: the system divided the task on its own into meaningful sub-problems, each of which a simple expert could solve. The central trick was that experts and doorkeeper were trained together – the experts specialized while the router learned to assign them correctly, in an interplay that married the classic idea of "divide and conquer" with learning from data. All the building blocks of today's MoE were here in principle already. What was missing was simply scale – and the hardware to exploit it.

2017: The Leap Into the Gigantic

The breakthrough came in 2017, the same year as the Transformer, from the same corner at Google. Noam Shazeer and colleagues published a paper with the gloriously immodest title "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer." They inserted an MoE layer with up to thousands of experts between the layers of a recurrent language model and proved that in this way a model's capacity could be raised by more than a thousandfold without the compute per example exploding.

The decisive technical contribution was sparse gating: instead of stirring all experts into a soft mixture, the router hard-selected only the best few. Shazeer additionally introduced a light noise into the gating and – more importantly still – named for the first time, with full clarity, the problem that has accompanied MoE to this day: load balancing. Without countermeasures, such systems tend to let a few experts receive almost all the traffic while the rest atrophies. More on that shortly.

2020 to 2021: GShard and Switch – MoE Meets the Transformer at Scale

The next milestones definitively wedded MoE to the Transformer and to the scale of industrial data centers. GShard (Google, 2020) scaled sparsely activated MoE Transformers to roughly 600 billion parameters, distributed across some two thousand accelerator chips. GShard established the pattern of expert parallelism: different experts physically reside on different chips, and tokens are shipped across the network to their assigned experts. GShard typically used top-2 routing – each token visits the two highest-scoring experts.

In 2021 came the Switch Transformer by William Fedus, Barret Zoph and, once again, Noam Shazeer. Its contribution was a radical simplification: why two experts per token at all? The Switch Transformer consistently selected only one (top-1). This reduced communication and compute, simplified load balancing, and proved remarkably stable. With it the team scaled its models into the trillion-parameter range and showed at the same time that sparse models learn faster than dense ones for the same compute budget. The Switch Transformer made the formula "more parameters at constant compute" respectable.

Part 4: How Routing Works in Detail

A Token's Journey Through the Layer

Let us follow a single word on its journey through an MoE layer to make the interplay tangible. Take the word "integral" in a mathematical text.

First, the attention mechanism has already enriched this word with its context; it now exists as a vector, a list of numbers encoding its context-dependent meaning. This vector reaches the MoE layer and first meets the router. The router multiplies the vector by its weight matrix and obtains, for each of – say – 64 experts, a score. Through a softmax these scores are normalized into probabilities. Suppose expert 12 gets 0.7, expert 39 gets 0.2, and the rest share the remaining 0.1.

With top-2 routing, the router now selects expert 12 and expert 39. The word "integral" is sent through both feed-forward networks and yields two output vectors. These are mixed according to their weights – the contribution of expert 12 counts, roughly speaking, more than that of expert 39, because its router score was higher. The mixed result leaves the layer. The other 62 experts have not performed a single computation for this token.

Wait – Can This Even Be Trained?

Here a subtle problem lurks. Selecting the best k experts is a hard, discontinuous decision: an expert is either selected or not. Such decisions are not differentiable, and the usual training procedure for neural networks – the backpropagation of gradients – needs differentiability like air to breathe. So how does the router learn to make good decisions at all?

The trick lies in the fact that the weights of the mixture – the softmax probabilities of the selected experts – do enter the computation and are therefore differentiable. The gradient flows back through these weights into the router. Put simply: if a selected expert delivered a good result, the router learns to give it an even higher score for similar inputs in the future; if the result was poor, its score drops. In this way experts and router learn together – the experts specialize, the router learns to steer the traffic aptly. It is the same coupled dance that Jacobs and Jordan already described in 1991, only at colossal scale.

Part 5: The Treacherous Problem of Load Balancing

The Vicious Circle of Expert Concentration

Now to the problem that Shazeer already named in 2017 and that to this day dominates half the research literature on MoE. It is a feedback loop with a fatal sign.

Imagine an expert, early in training, happens by chance to fall a little ahead and gets selected somewhat more often than its peers. Because it receives more tokens, it gets more training signal and thus improves. Because it improves, the router gives it even higher scores. Because it receives higher scores, it is selected even more often. A self-reinforcing pull arises. At the end of this vicious circle stands a condition called expert collapse: a handful of experts swallows almost all the traffic, while the great majority lies fallow as dead weight and is never properly trained. The model then formally has hundreds of experts but uses only a handful – the expensive capacity evaporates.

The Classic Remedy: The Auxiliary Loss

The traditional countermeasure is an auxiliary loss for load balancing. In addition to the actual objective – making good predictions – the model receives a small penalty term that triggers whenever the tokens are distributed unevenly across the experts. Typically this term multiplies the fraction of tokens going to an expert by that expert's mean router probability; the product penalizes placing much probability mass on already-overloaded experts. The effect is a gentle pressure toward even utilization.

But this addition carries an uncomfortable price, and it reveals something fundamental. The auxiliary loss fights against the actual learning objective. It sometimes forces the model to send a token to a less fitting expert, merely so the ledger balances. That is why in practice its weight is kept small – a factor of about 0.01 is common. One buys balance by giving up a little model quality for it. It is a regularizer singing against the main melody.

Capacity and Dropped Tokens

A further practical nuisance is the capacity factor. So that computation stays efficient on parallel hardware, one fixes in advance how many tokens each expert may process at most. If an expert overflows – because the router assigns it more tokens than its capacity allows – the surplus tokens are simply dropped (token dropping): they do not pass through the expert layer at all but are handed onward unchanged. Set the capacity factor too small and too many tokens fall through and quality suffers. Set it too large and you waste compute and memory on rarely used reserve capacity. It is a tightrope walk every MoE system must calibrate.

Two Elegant Exits: Expert Choice and the Auxiliary-Loss-Free Approach

Research has found two especially elegant answers to this dilemma. The first reverses the direction of gaze. Instead of letting each token choose its preferred experts (token choice), one lets each expert choose its preferred tokens (expert choice). Because each expert then, by construction, takes in exactly its capacity of tokens, the load is automatically perfectly balanced – no auxiliary loss, no collapse, no dropped tokens. The price: some tokens may be courted by several experts, others by none, which brings its own subtleties.

The second answer comes from the DeepSeek-V3 mentioned at the outset and bears the telling name auxiliary-loss-free load balancing – balancing entirely without an auxiliary loss. The idea: forgo the quality-diminishing penalty term and instead steer the balance via a learnable bias term per expert, added directly onto the router scores. If an expert gets too much traffic, its bias is quietly lowered so it falls back a little in the ranking; if it gets too little, its bias is raised. This bias influences only the selection, not the weighting of the mixture – and therefore does not disturb the actual learning objective. In this way the model achieves even utilization without sacrificing model quality to the demand for balance. I am of the opinion that precisely this maneuver – balancing via the selection rather than via a penalty term in the loss – is one of the underappreciated reasons for the efficiency of the latest MoE generation.

Part 6: Three Models From Practice

Abstraction is good; concrete numbers are better. Let us examine three influential MoE models and read their architecture like a dossier.

Mixtral 8x7B – the Open Reference

In late 2023 the French lab Mistral AI released, with Mixtral 8x7B, an open model that made MoE popular in the broad developer community. The name is at once a set of build instructions: in each of the 32 Transformer layers sit eight experts, and a router selects the best two for each token (top-2).

The numbers illustrate the decoupling in exemplary fashion. In total, each token has access to roughly 47 billion parameters – that much knowledge sits in the model. Yet per token only about 13 billion parameters are active. That means: the compute cost corresponds to a dense 13-billion model, but the stored capacity to that of a much larger one. (That 8×7 does not equal 47 is because the experts share only the feed-forward networks; attention and embeddings are shared and counted only once.) Mixtral thereby matched or surpassed the dense Llama-2-70B and the then-dominant GPT-3.5 on many benchmarks – at a fraction of the compute cost per token, especially markedly on mathematics, programming, and multilingual tasks.

Switch Transformer – the Radical Simplification

The already-mentioned Switch Transformer stands for the opposite design philosophy: maximal simplicity through top-1 routing. Each token visits exactly one expert. This minimizes communication and compute and makes load balancing more manageable. Its historical merit is the proof that one can scale models to trillions of parameters while the compute per token stays constant – and that such models learn faster than their dense counterparts per unit of compute budget invested.

DeepSeek-V3 – the State of the Art

DeepSeek-V3 from late 2024 shows how mature the architecture has become and bundles several advanced ideas at once. It has 671 billion total parameters, of which only 37 billion are active per token. Three features set it apart.

First, it relies on fine-grained experts: instead of a few large experts there are very many small ones – 256 routed experts, of which each token visits eight. The idea behind it: many small specialists can be combined more precisely than a few large ones and allow a finer division of knowledge. Second, there is a shared expert that is always active – it catches the general knowledge useful for all tokens, so that the routed experts can concentrate on the specialized. Third, DeepSeek-V3 uses the auxiliary-loss-free load balancing described above. Together these ingredients yield a model at the frontier of open systems whose operating cost per token nonetheless barely exceeds that of a mid-sized dense model.

A Dossier Comparison

The table makes the common pattern visible: in every row a large gap yawns between the total and the active parameter count. That gap is the gain of Mixture of Experts.

Part 7: The Honest Price – Where MoE Is Not Free

It would be dishonest to present MoE as a miracle cure without cost. The architecture displaces the problem rather than abolishing it, and the displacement has consequences.

The most obvious price is memory. True, an MoE model computes with only a fraction of its parameters – but it must keep all parameters in memory, because which expert is needed is decided only at runtime, token by token. DeepSeek-V3 demands enough graphics memory for 671 billion parameters even though it computes with only 37 billion. MoE thus trades memory for compute time: it saves arithmetic operations by generously occupying memory. For operators with many accelerators this is a good trade; for someone who wants to run a model on a single device it can be a disqualifier.

The second price is complexity in training and operation. The load balancing must be carefully calibrated, or the experts collapse. Distribute the experts across many chips and lively network traffic arises, because tokens migrate to their experts and back again – the so-called all-to-all communication pattern, which presupposes fast connections between the accelerators. MoE models are also considered somewhat more delicate to train; the hard routing decision can make learning more unstable than in dense models.

The third, subtler price concerns interpretability and batch processing. Because different tokens of a request migrate to different experts, hardware utilization is less even than in a dense model, which complicates the efficient bundling of many requests. And the tempting hope that one could, by observing the experts, understand how the model thinks is only partly fulfilled: the responsibilities of the experts are, as stressed at the outset, seldom cleanly humanly nameable. Whoever wants to understand more deeply what goes on in a network needs other tools – such as those of mechanistic interpretability (see The Ghost in the Machine: How to Read a Neural Network From the Inside).

Part 8: Why the Whole Industry Is Marching in This Direction

Despite these costs, MoE has become the dominant architecture for the largest models, and the reason lies in the economics of inference. A model is trained once but thereafter used perhaps billions of times. Over a model's lifetime the inference costs sum to the by far largest line item. Any architecture that lowers these costs without curtailing capability holds an overwhelming economic advantage.

That is exactly what MoE delivers: it allows the capacity – and hence, tendentially, the capability – of a model to be raised while the cost per word stays in check. In the language of the scaling laws, MoE shifts the curve: for a given compute budget one can build a more capable model, because one gets parameters "more cheaply." For providers operating models at large scale, this is not a niche advantage but the difference between a viable and a loss-making business.

I am of the opinion that MoE is therefore less a passing fashion than a deep structural adjustment: the industry has recognized that dense activation – every parameter computing on every word – is a waste that justifies itself less and less as model size grows. Conditional computation, old as it is, was the obvious answer once the scale grew large enough to justify its implementation problems. Whether top-1, top-2, or fine-grained top-8 architectures prevail, whether the auxiliary loss or the bias-based balancing wins – these are questions of detail. The basic principle, decoupling knowledge from compute, will remain.

The Central Takeaway

The central lesson of Mixture of Experts can be captured in one sentence: not everything has to compute all the time. The decades-long tacit assumption that a model must muster its entire size on every prediction was never a law of nature, only the most convenient path. As soon as you drop that assumption and place a clever doorkeeper in front, two things decouple that were held to be inseparable: the knowledge of a system and the effort of using it.

This insight reaches far beyond AI. It is a principle of good system design as such: build capacity generously, but activate it sparingly and on demand. A hospital with many specialists and a good triage is more capable and cheaper than one in which every patient sees every doctor in turn. A software system that loads expensive modules only when needed scales better than one that always keeps everything on hand. The real feat rarely lies in sheer size – it lies in the router, in the shrewdness with which one decides which part of the capacity is truly needed for the task at hand.

For practice this means: whoever today judges the size of a language model should ask not only about the total parameter count, but about the active parameters per token – for only this number reveals the true operating cost, and often more about the model's behavior than the impressive grand total on the spec sheet.

A Question to Reflect On

If the future of AI consists in combining enormous capacity with sparing, conditional activation – how much resemblance does this path bear to your own thinking, which likewise recruits only a tiny part of its possibilities for the task at hand? And if the "knowledge" of a model and its "effort" can be cleanly separated: which systems in your professional day-to-day – software, processes, teams – still keep their entire capacity under full load on every task, when a clever doorkeeper would suffice?

Cross-References in the Vault

• Attention Is All You Need: The Transformer, Self-Attention, and the Architecture of Modern AI – the Transformer, whose feed-forward networks MoE replaces with experts.
• How Big Is Big Enough? Scaling Laws, Chinchilla, and the Measurement of AI – the scaling laws whose cost-coupling MoE breaks open.
• The Ghost in the Machine: How to Read a Neural Network From the Inside – why the "experts" cannot simply be read as human fields of expertise.

Sources

• Jacobs, Jordan, Nowlan & Hinton (1991): Adaptive Mixtures of Local Experts, Neural Computation 3(1):79–87. https://direct.mit.edu/neco/article/3/1/79/5560/Adaptive-Mixtures-of-Local-Experts
• Shazeer et al. (2017): Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. https://arxiv.org/abs/1701.06538
• Fedus, Zoph & Shazeer (2022): Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity, JMLR 23. https://dl.acm.org/doi/abs/10.5555/3586589.3586709
• Jiang et al. / Mistral AI (2024): Mixtral of Experts. https://arxiv.org/pdf/2401.04088
• DeepSeek-AI (2024): DeepSeek-V3 Technical Report. https://arxiv.org/html/2412.19437v1
• Zhou et al. (2022): Mixture-of-Experts with Expert Choice Routing. https://arxiv.org/pdf/2202.09368
• Background on load balancing: A Review on the Evolvement of Load Balancing Strategy in MoE LLMs. https://huggingface.co/blog/NormalUhr/moe-balance

← All articles