Mixture Expert Paper: Active Parameter Total, Inference Cost, Latency

Mixture-of-experts active parameters estimate the computation applied to each generated token, while total parameters estimate the complete weight footprint that must be stored and made available at inference; neither number alone determines inference cost or latency. When reading a mixture of experts paper, use active parameters as a first-pass FLOP proxy and total parameters as a memory-and-capacity proxy, then verify the actual deployment claims with workload-specific throughput, time-to-first-token, memory, and cost measurements.

What are active parameters in a mixture-of-experts model?

A dense Transformer runs the same feed-forward-network weights for every token. A mixture-of-experts (MoE) Transformer replaces some of those dense feed-forward layers with many expert networks plus a small router. For each token, the router scores experts and selects the top k experts to execute; their outputs are combined and passed back into the normal Transformer path.

Active parameters are the parameters touched by that routing decision for one token. They include the always-on parts of the model—embeddings, attention blocks, output layers, routers, and any shared experts—plus the selected routed experts. A model with 200 billion total parameters may therefore execute an amount of expert computation closer to a much smaller dense model if it selects only a few experts per MoE layer.

The important qualification is that active parameters are not necessarily “the size of one expert times k.” Attention and other non-expert components are active for every token, and architectures differ in which layers are sparse, how large their experts are, whether they include shared experts, and whether routing is top-1, top-2, or higher. Read the architecture table or formula rather than trying to reconstruct the figure from the model name.

Switch Transformers made the core idea especially clear: route each token to one expert instead of two, reducing router work, expert capacity requirements, and distributed communication compared with top-2 routing. The paper describes sparse expert models as having very large parameter counts with approximately constant computation per token as the number of experts grows—under its architecture and routing assumptions.

What do total parameters mean for MoE inference?

Total parameters are every learned weight in the checkpoint: all experts, all shared layers, routers, embeddings, and output weights. Unlike active parameters, the total is fixed regardless of the prompt. It is the number that largely determines checkpoint download size, minimum weight-storage requirements, cold-start pressure, and the baseline amount of model state a serving system must place across GPUs, CPU RAM, or storage.

That is why “37B active, 671B total” should never be read as “it runs like a 37B dense model in every respect.” DeepSeek-V3, for example, reports 671B total parameters and 37B activated for each token. The active figure conveys why sparse computation can be economical; the total figure conveys that the system still has to host a 671B-weight model and schedule access to a very large pool of experts.

For a rough weight-memory estimate, multiply total parameters by bytes per stored weight, then add overhead. At FP16 or BF16, raw weights are about 2 bytes per parameter; at 8-bit, about 1 byte; and at 4-bit, about 0.5 byte before quantization metadata and runtime buffers. A 671B-parameter model at an idealized 4-bit packing is roughly 335.5 GB of raw weight values—not a 37B-model footprint. This is why active count does not tell you whether a model fits on your hardware.

Do active parameters determine inference cost?

They determine a large part of arithmetic cost, especially during autoregressive decode. A useful approximation is that the FLOP cost of the MoE feed-forward portion grows with the selected experts, not with every installed expert. If a dense baseline and an MoE have comparable always-on layers and comparable active feed-forward width, the MoE can deliver more learned capacity without multiplying per-token matrix-multiply work by its total number of experts.

But cloud inference cost is not a direct conversion from FLOPs. The provider pays for accelerator memory, memory bandwidth, interconnect, idle capacity, batching efficiency, power, networking, and the software stack. If requests route toward a small subset of experts, those experts can become hot while others sit idle. The scheduler may also need to reserve or shard the full model across more devices than active-FLOP math suggests.

The correct engineering statement is: active parameters are a useful lower-level compute metric, not a price quote. For API usage, compare published input and output token prices. For self-hosting, measure GPU-hours per useful request at your target concurrency, context size, quantization, and service-level objective. A lower active count can reduce cost per generated token, but a large total count can still make the deployment expensive to keep resident.

Why can an MoE model have high latency with few active parameters?

Latency is wall-clock time for a request, not arithmetic alone. During decode, the accelerator may be memory-bandwidth-bound: each new token requires reading weights and KV-cache data, and an MoE adds gather, dispatch, expert execution, and combine steps. If chosen experts live on different GPUs, routing can introduce all-to-all communication and synchronization. Small batches leave less opportunity to hide that overhead.

Routing itself is rarely the whole story. In a 2026 empirical preprint, researchers benchmarked a 6.9B-total, 1.3B-active MoE on an Apple M2 Pro and Jetson Orin Nano. The model lagged a similarly active dense baseline, and their node-level measurements attributed the gap primarily to total-parameter memory footprint, expert dispatch, and KV-cache pressure rather than router computation. It is one model, runtime, and hardware set—not a universal result—but it is a useful corrective to the slogan that sparse activation automatically means low latency.

Prompt processing and generation also behave differently. Prefill processes many prompt tokens in parallel and can benefit substantially from batching and high compute utilization. Decode generates one or a few tokens at a time, making bandwidth, dispatch, and synchronization more visible. A model can look excellent in aggregate tokens per second at high batch size while missing an interactive time-to-first-token or per-token latency target.

How should you read an MoE paper’s parameter numbers?

Start by separating three questions that are often compressed into one chart: how much knowledge or representational capacity does the model have; how much computation happens per token; and what infrastructure is required to serve it. Total parameters mostly address the first and third. Active parameters mostly address the second. Neither answers quality, price, or latency without the rest of the experimental setup.

1. Find the routing rule. Is it top-1, top-2, or a variable number of experts? A top-2 model performs more expert work than a comparable top-1 model.
2. Check which layers are MoE. Attention layers and other dense components remain active, so active count is not simply the routed-expert count.
3. Separate training throughput from inference performance. Faster training to a quality target does not establish lower interactive serving latency.
4. Check the hardware topology. Expert parallelism can require cross-device communication; topology and placement matter as much as nominal GPU FLOPs.
5. Look for batch size, prompt length, generated length, precision, quantization, and concurrency. Without them, tokens-per-second and cost claims are not portable.
6. Treat model-card figures as architecture metadata. Benchmark your own target runtime before selecting a model for a production latency or budget requirement.

What is the simplest way to compare an MoE with a dense model?

Use a two-axis comparison. First, compare active parameters—or preferably measured decode FLOPs and output tokens per second—to estimate per-token compute. Second, compare total parameters, quantized weight size, and peak memory to establish deployment feasibility. Then hold task quality and context length constant. A 20B-active/200B-total MoE may compete computationally with a dense model in the tens-of-billions range while requiring infrastructure closer to its 200B total footprint.

For an interactive coding or chat workload, record at least four measurements: time to first token, steady-state output tokens per second, peak GPU memory, and cost per successful task at realistic concurrency. Add p50 and p95 latency rather than reporting only a mean. Routing distributions can vary by prompt, so tail behavior and expert imbalance may matter more than a clean average.

When should you choose an MoE model?

Choose an MoE when you want high model capacity and quality per unit of active computation, and your serving environment can comfortably hold and efficiently distribute the full checkpoint. It is particularly compelling when you can batch enough traffic to keep experts busy, have a runtime that handles expert parallelism well, and value throughput more than the smallest possible single-request latency.

Prefer a smaller dense model when weight memory is the hard limit, deployment must be simple, traffic is low-concurrency and latency-sensitive, or you are running locally on bandwidth-constrained hardware. This is not an architectural verdict: it is a reminder that an MoE’s advantage depends on the bottleneck. The paper’s active-parameter figure tells you what sparsity saves; the total-parameter figure tells you what sparsity does not make disappear.

Try MoE coding models without guessing at the infrastructure

If this distinction matters because you are choosing a coding model rather than designing one, Cline is an open-source coding agent runtime available in an IDE, terminal, and SDK. Its site says it can work with provider APIs, local Ollama or LM Studio models, and OpenAI-compatible endpoints, so you can use the same agent workflow while comparing a local or hosted MoE against a dense alternative.

Cline itself is free to use with model access paid separately. For developers who want included open-weight coding models without separate provider keys, ClinePass is listed at a limited-time $4.99 first month and $9.99 per month afterward, with additional processing fees potentially applying. That makes it practical to test whether an MoE’s active-parameter efficiency translates into the actual coding-agent responsiveness and usage pattern you need.