Tool Suggestions
Cerebras Runs a 3-Trillion-Parameter Kimi Across 59 Wafers
How do you run a model with 3 trillion parameters when a single chip cannot hold it? In a technical post dated October 22, Cerebras published its answer, using Moonshot AI's open Kimi K3 as the worked example: split the model across 59 wafer-scale chips, keep the weights pinned on the chips, and pass only activations between stages. The company advertises output speeds of 1,000 to 2,000-plus tokens per second for multi-trillion-parameter models. The post is a deployment recipe, not a benchmark result — and understanding the difference is the whole point of reading it.
The math: 93 layers, two kinds of wafer
The calculation starts with size. Kimi K3 has roughly 3 trillion parameters. At 4-bit weight quantization, the raw parameter payload divides across the model's 93 layers to about 16 GB per layer. A single Cerebras WSE-3T wafer holds 44 GB of on-chip SRAM, so each layer fits comfortably on one wafer — but how many layers fit per wafer depends on the layer type. Dense layers need additional room for a key-value cache, because each dense, full-attention layer stores the keys and values for every token in every active request. So one dense layer occupies a full wafer, leaving the remaining memory for its KV cache. Linear-attention layers do not need room for a KV cache: instead of storing keys and values for every token, a linear-attention layer keeps a small, fixed-size state per request, regardless of context length. That leaves room for two linear layers per wafer, at 2 times 16 GB, or 32 GB. Kimi K3 has 69 linear layers and 24 dense layers, which works out to 35 wafers for linear layers and 24 for dense layers: 59 total.
The pipeline: weights stay put, activations move
The wafers connect in a pipeline. After the Cerebras compiler maps the model layers across regions of wafers, the runtime maps the path for tokens to flow between them. Activations move between wafers via RDMA over Ethernet, at about 2 microseconds per hop. The key insight is that because each wafer has 44 GB of on-chip SRAM and each layer is only 16 GB, entire layers fit directly on the chip. That means in most cases the model weights never have to move, and layers do not need to be split across wafers — only activations cross from one wafer to the next. This is the opposite of the usual GPU setup, where weights are repeatedly shuttled between memory and compute, and it is why Cerebras can claim its speeds: the expensive part of inference, moving weights, mostly disappears. Each wafer acts like a stage in an assembly line, processing multiple requests concurrently without idling, and the pipeline supports a high number of simultaneous requests based on the on-chip memory reserved for key-value caches. For reference, Kimi K3 itself is no toy: it scores 88.3% on Terminal-Bench 2.1 and 91.2% on BrowseComp, outperforming GPT-5.6 Sol on the latter benchmark.
What the post does not prove
Here is the honest caveat, and it matters. Cerebras has not established whether the Kimi K3 layout was actually run, or supplied an independently verified result tying that example to the advertised speed. Its post does not provide the full benchmark conditions, measured power consumption, or a clear final count of simultaneous requests the layout supports. One Hacker News commenter did the math on the whole rig and estimated roughly 1.18 MW of total draw at about 4.1 joules per token — a commenter's estimate, not a company figure, but a useful reminder that 59 wafers are not free to run. Others pointed out the gap between the marketing and reality: publicly accessible models on Cerebras are still limited to things like GPT OSS 120B and Qwen 3.8 27B, which users describe as fast but often wrong. There is also a structural tension in the pipeline itself: with 59 stages, keeping wafers busy needs many requests in flight, roughly stages times micro-batch, yet each one needs its own KV cache on every dense-layer wafer, in the roughly 28 GB left beside the weights. Agents push the wrong way on both counts — context grows for hours, so fewer requests fit, so stages sit idle — and a fully dense model would need KV cache on every stage, which changes the math entirely. The useful advance here is a concrete deployment recipe for trillion-parameter models. The performance breakthrough still needs a measured result.