Why LLMs Keep Getting Bigger While Using Fewer Parameters Per Token

Imagine your team runs an internal coding assistant called PatchPilot. A developer pastes in a failing TypeScript build, asks why a database migration broke CI, and expects a useful answer before the coffee gets cold. PatchPilot needs broad knowledge: TypeScript syntax, SQL dialects, package managers, cloud APIs, test failures, and ordinary natural language. The obvious way to make it better is to make its neural network larger. But if every new parameter must participate in producing every next token, each answer becomes slower and more expensive. Modern large language models increasingly work around that trade-off by becoming bigger overall while using fewer of their parameters on each token.

The two parameter counts that now matter

A parameter is a learned number in the model: during training, optimization adjusts billions of these numbers so the model becomes better at predicting the next token. A token is a chunk of text processed by the model, often a word fragment, punctuation mark, or short common word. When PatchPilot produces the next token in npm run test, a model may report both its total parameter count and its active parameter count. Those are not interchangeable.

Total parameters are all learned weights stored in the model. Active parameters are the subset that actually performs substantial computation for one token. A dense model activates essentially all of its parameters for every token. If it has 70 billion parameters, each generated token runs through roughly that whole learned network. A sparse model, usually a mixture-of-experts model, can store far more weights but choose only some of them for a particular token.

That is why a model description such as “236B total parameters, 21B activated per token” is not a typo or marketing contradiction. It means the model has a large pool of learned capacity, but its per-token path through that pool is much narrower. DeepSeek-V2, for example, described this kind of design with 236 billion total parameters and 21 billion activated for each token. The numbers do not mean it compresses 236 billion computations into 21 billion computations; they mean most of its expert weights are deliberately skipped on a given forward pass.

Where the experts fit in a Transformer

PatchPilot is still usually built around a Transformer, the neural-network architecture that repeatedly mixes information across the prompt and then transforms each token representation. In each layer, attention lets a token gather relevant information from other tokens. For the migration failure, the token representing constraint might attend to nearby SQL identifiers, an earlier error message, and the user’s question. Attention is not the part that mixture-of-experts primarily replaces.

The usual target is the feed-forward network, sometimes called an MLP, inside each Transformer layer. In a dense Transformer, every token goes through one shared feed-forward network. In a mixture-of-experts, or MoE, that one network is replaced with many feed-forward networks called experts. Each expert has its own parameters. A small router network examines the current token representation and selects a few experts to process it.

Suppose one PatchPilot layer has 64 experts and the router selects the best two for each token. When it reads ALTER TABLE, it might route strongly toward experts that training has made useful for structured code, schema changes, and database terminology. When it reads a human-written explanation of a flaky test, it may choose a different pair. The model is not assigning human-readable job titles to experts, and an expert is not guaranteed to be “the SQL expert.” Specialization emerges imperfectly from training. But the routing mechanism gives the network a way to reuse different pieces of capacity for different contexts.

Why this changes the scaling equation

Adding experts raises total parameter count quickly. If PatchPilot changes one dense feed-forward network into 64 expert copies but routes each token to only two, it has added a great deal of stored capacity without making every token execute all 64 copies. The router itself is comparatively small. In the idealized case, that layer’s expert computation is closer to two experts than 64 experts per token.

This is attractive because language is heterogeneous. The model needs recurring patterns for code, prose, math, configuration files, error logs, foreign languages, and domain-specific vocabulary. A single dense network must use the same parameters for all of them on every step. MoE creates room for more conditional capacity: PatchPilot can call on a larger collection of learned transformations while keeping the route for one next-token prediction relatively cheap.

There is an important wording correction here. These models are not necessarily using fewer parameters per token than an older small model. They are using fewer active parameters per token than their own total parameter count, and often fewer than a dense model with similar total capacity. A 21B-active MoE can still be much more expensive than a 7B dense model. “Sparse” is a comparison to activating the entire large model, not a promise of cheap inference.

Why fewer active parameters does not mean free inference

For PatchPilot, response latency depends on more than arithmetic. The system must keep the model weights available in GPU memory or fetch them efficiently across devices. It must move token activations to whichever machines hold the chosen experts, then collect the results. This communication step is often called an all-to-all exchange because tokens may be sent among many expert-hosting devices. At small scale, that movement can erase the theoretical savings; at large scale, the serving stack must carefully batch requests and place experts to avoid network bottlenecks.

The non-expert parts of the model also still run. Attention, embeddings, normalization layers, output layers, and the key-value cache all cost memory or compute. The key-value cache is stored attention state from previously processed tokens; it grows with the context length and is especially relevant when PatchPilot is reading a large repository diff. MoE lowers one major component of per-token compute, but it does not make long contexts or token-by-token generation disappear.

Capacity management adds another constraint. If the router sends too many tokens to one expert, that expert becomes a queue while other GPUs idle. Training therefore uses load-balancing techniques to encourage traffic across experts. Systems also set an expert capacity: only a limited number of tokens in a batch may be accepted by an expert. Poor routing can lead to dropped, rerouted, or less efficiently processed tokens, and unstable routing has been a practical challenge since early large MoE systems.

What “bigger” buys the model

The hope is not merely that PatchPilot stores more facts. More total expert capacity can let it represent more distinct patterns and combinations of patterns. Across a training batch, different tokens visit different experts, so all experts receive training updates over time. For one token, however, the router chooses a limited computation path. This makes MoE a form of conditional computation: the input determines which part of the model is used.

That conditionality is why model cards increasingly need both numbers. Total parameters say something about the model’s stored capacity, memory footprint, and distribution complexity. Active parameters say more about the dominant neural computation done for one token. Neither number alone tells you whether PatchPilot will be fast, fit on your hardware, write better migrations, or be inexpensive through an API. Context length, quantization, batching, GPU memory bandwidth, expert placement, and the quality of training all matter too.

How to read the numbers as an engineer

When evaluating an MoE coding model, treat “total” as an architectural capacity number and “active” as a useful but incomplete cost hint. Ask how many experts are selected per token, whether the provider reports prompt-processing and generation throughput, what hardware or API pricing is involved, and whether the model fits your deployment constraints. A model with a striking total count may be an efficient choice for a shared, heavily batched service and a poor choice for a single local GPU.

For PatchPilot, the result is straightforward. Instead of building one giant general-purpose circuit that runs in full for every comma and identifier, the team builds a large library of circuits and a fast dispatcher. Each token still receives a coherent answer from the whole Transformer, but only a selected fraction of the expert library does work. LLMs keep getting bigger because more learned capacity remains useful. They use fewer active parameters per token because activating all of that capacity, every time, is usually the wrong engineering bargain.