Kimi K3 on MI355X: The Memory Capacity Argument Is Real, the Cost Benchmark Is Overstated, and the Real Lesson Is About Procurement

The MI355X Argument Is Half True, and the Half That Is True Matters More Than the Half That Is Not

Wafer published benchmark numbers last week showing their AMD MI355X cluster running Kimi K3 at 952 tok/s per node and 48 tok/s per GPU-hour in cost efficiency, comparing favorably against both B200 and B300 deployments. The HN thread immediately caught fire, with a significant faction calling the post “AI slop” and “lazily written,” and a smaller faction pointing out that the underlying technical content is actually solid regardless of prose quality. Both factions are right about different things, and the argument between them is obscuring the real question: when you are choosing infrastructure for a 2.8 trillion parameter model that requires more than 1.5 TB of VRAM before touching a KV cache, does AMD’s MI355X now make more engineering sense than Blackwell?

My short answer is: yes, under specific conditions that most inference providers are not actually in, and no, under the conditions that the people dismissing AMD are usually in. Let me explain both sides with specifics, because the usual GPU discourse skips the math that actually determines the answer.

First, What Kimi K3 Actually Is — And Why Size Changes the Equation

Kimi K3 is Moonshot AI’s flagship model at 2.8 trillion parameters. That number is not a typo. For comparison, DeepSeek V4-Pro runs 1.6 trillion parameters and GLM5.2 (Z.ai’s model, developed together with Tsinghua University) is 753 billion. K3 more than triples the parameter count of what was already considered a frontier-scale open-weights model six months ago.

The raw VRAM requirement at FP8 precision is approximately 1.4 TB just for weights. Add a KV cache for 1 million tokens of context at K3’s architectural dimensions, and you are looking at over 1.5 TB of VRAM before you have processed a single prefill token. This creates a hard mechanical constraint:

• NVIDIA B200 has 192 GB HBM3e per GPU. A single 8-GPU node gives you 1.536 TB — enough for weights but almost no room for a 1M-token KV cache. You cannot serve K3 at its native context on a single B200 node.
• NVIDIA B300 (shipping to select customers through 2026) has 288 GB HBM3e per GPU. A single 8-GPU node gives you 2.304 TB. You can serve K3 plus a 1M-token KV cache on a single B300 node.
• AMD MI355X has 288 GB HBM3e per GPU, same memory per device as the B300. A single 8-GPU node gives you 2.304 TB. Identical memory headroom to B300.

This is the specific hardware property that makes the MI355X relevant to K3 in a way it was not relevant to smaller models. It is not that AMD suddenly caught up to NVIDIA on compute throughput — the B300 wins roughly 1.65× on aggregate throughput over MI355X. It is that the B200, which is the dominant GPU in every major cloud right now, literally cannot fit K3 on a single node at full context.

If you are building inference infrastructure for K3 today, your option set is narrower than people realize: (1) two B200 nodes running TP16, which means cross-node all-reduce on the decode critical path and roughly 249 tok/s per node due to the networking overhead; (2) a B300 node if you can actually get one, which at current spot market is running $6/GPU-hour or higher; or (3) an MI355X node, which Wafer priced at $2.50/GPU-hour from their arrangement and which gpus.io lists at $2.95/GPU-hour in the broader spot market.

That constraint — not abstract performance-per-FLOP arguments — is what makes this comparison interesting.

The Benchmark Numbers Wafer Published, and What They Actually Show

Here is the table from Wafer’s blog, reproduced verbatim because I will be arguing about specific cells:

Pricing assumed: $2.50/GPU-hr for MI355X, $4.25/GPU-hr for B200, $6.00/GPU-hr for B300.

The HN criticism that this comparison is unfair has merit in two specific places and is wrong in a third.

The B200 comparison is the fairest. At 498 tok/s total across 16 GPUs for a TP16 deployment, the B200 achieves roughly 249 tok/s per node. The MI355X at TP8 gets 952 tok/s on 8 GPUs. That is a 3.8× throughput advantage per node for the MI355X — and it is largely explained by the architectural mismatch. The B200 is paying RoCE v2 cross-node all-reduce latency (~195 Gb/s) on every decode step. The MI355X is doing the same tensor parallelism entirely within single-node interconnect. You are comparing a configuration that fits to one that does not, and drawing throughput conclusions accordingly. Wafer acknowledges this in the article. It is not a deceptive comparison, but it should be labeled “MI355X one-node vs B200 two-node due to memory constraint” rather than implying these are equivalent deployment tiers.

The B300 comparison is the least fair. This is where the HN critics have the most ground. The B300 wins 1,568 tok/s vs 952 tok/s — a 1.65× advantage in aggregate throughput, 196 vs 119 tok/s per GPU. The MI355X only wins on cost-per-throughput because the assumed B300 price ($6.00/GPU-hr) is 2.4× higher than the assumed MI355X price ($2.50/GPU-hr). If the price ratio were 1.65× instead of 2.4×, the B300 would win cost-efficiency too. The cost table reflects a real market moment but is not necessarily stable, and the B300 spot pricing Wafer uses is on the high end of what I have seen quoted for enterprise arrangements.

The prefill performance asymmetry is the most important thing Wafer buries in the piece: a 172k-token cold prefill takes approximately 51 seconds on the MI355X versus approximately 23 seconds on a B300. On a 1M-context model — K3’s advertised capability — workloads with multi-hundred-thousand-token contexts will have huge time-to-first-token (TTFT) bills. Wafer describes the fix (zero-padding attention heads from 12 to 16 to access the AITER MLA prefill kernel, yielding a 2-3× prefill speedup) but does not update the 51-second headline with the patched number. I noticed that omission.

The ROCm Software Stack Problem Is Real, and K3 Is a Best-Case Scenario

I have deployed AMD GPUs in production inference environments, and the ROCm software friction is not NVIDIA marketing talking points — it is a real tax you pay per model deployment. The Wafer team encountered two concrete ROCm bugs getting K3 to production performance.

First: the speculative decoding path in sglang had a missing top_k_renorm_prob definition on the ROCm build. The CUDA build imports this from sgl_kernel; the ROCm build aliases only a Triton top-p kernel and leaves top-k renorm undefined. Any request landing on the dense sampling path crashes the scheduler with a NameError. This is the kind of bug that does not surface in toy benchmarks but destroys you in production when a specific sampling configuration hits. The fix is a single PyTorch function — trivial in retrospect — but you need to know to look for it and have the engineering bandwidth to diagnose it quickly.

Second: the AITER MLA prefill kernel would not load for K3 because K3 at TP8 has 12 attention heads per rank, and AITER’s MLA path is hardcoded for 4, 8, or multiples of 16. Fix: zero-pad to 16, run the fast kernel, extract the real 12 heads. Again trivial once identified — non-obvious when you are looking at a 51-second cold prefill and wondering whether it is a fundamental hardware limitation or a fixable configuration mismatch.

Wafer describes both fixes as straightforward. They are, if you have ML systems engineering context to identify them as kernel configuration problems. What they do not mention is the counterfactual: on NVIDIA hardware with CUDA, both would have worked out of the box. NVIDIA’s kernel coverage for popular inference frameworks is more comprehensively tested across model architectures, especially for architectural edge cases like non-standard attention head counts.

The argument Wafer is making — and which I find genuinely interesting — is that AMD is shipping day-0 ROCm support for K3 specifically, and that the class of software fixes required is shrinking per model. Their GLM5.2 deployment required more engineering work than K3. If that trend continues, the software tax for AMD may approach zero within the next two to three model generations for models AMD actively targets. That is a falsifiable claim and I will say below what I think happens.

The more important observation about speculative decode: Wafer’s spec-dec gains were 2.2× single-stream performance improvement and 1.7× per-stream at moderate load. If you are running K3 without speculative decoding on any hardware — CUDA or ROCm — you are leaving roughly half your achievable throughput on the table. The MI355X spec-dec fix is AMD-specific, but the underlying throughput multiplier is universal and often underweighted in headline benchmark comparisons.

The Open-Weights vs Open-Source Argument the HN Thread Got Distracted By

About half the HN comment thread under the Wafer post became an argument about whether Kimi K3 should be called “open source” or “open weights.” This debate is mildly interesting as philosophy and entirely useless for infrastructure decision-making. I will state my position briefly.

Open weights is the correct term. The key question for “open source” is whether you can reproduce the artifact from the published source. For compiled software, that means source code plus a compiler. For model weights, the analog to source would be training data plus training code. Moonshot released neither. They released the trained weights under a permissive license.

The counterargument I find most interesting comes from commenter “andy99” in the HN thread, citing the FSF’s four freedoms: the freedom to study, use, modify, and redistribute. Weights plus a reference implementation satisfy three of those four in practice — you can run K3, probe its representations, fine-tune it, redistribute modified versions. The freedom you cannot exercise is training from scratch. Whether “training from scratch” is analogous to “compiling from source” is what the debate is actually about, and both sides have defensible positions.

My actual engineering position: the distinction matters less for most practitioners than people think. If you are deciding whether to use K3 in your stack, “open weights” tells you what you need to know — you can run it, you do not depend on Moonshot’s API. If you are a research lab studying training dynamics, “open weights” is insufficient. Both statements are true and refer to different audiences. Call it open weights, move on, and spend the freed attention on the memory capacity question, which is more tractable.

What the MI355X vs B300 Decision Actually Looks Like From an Operator Perspective

Let me make the infrastructure decision concrete. Assume you are building K3 inference today. What does the decision tree look like?

High-concurrency API product: The B300 wins if you can procure it. At 1,568 tok/s peak aggregate vs 952 tok/s, the B300 node handles 1.65× more requests. At current pricing (2.4× more expensive per GPU-hour), the B300 costs approximately 1.45× more per token served at equivalent utilization. That is real money for a product. The MI355X wins on cost-per-token because the 2.4× price ratio exceeds the 1.65× throughput advantage. But the price ratio is the key lever — it is a market condition, not a fundamental constant.

B300 procurement unavailability: B300 availability remains constrained. Many organizations cannot procure B300 nodes at any price right now. If your choice is between waiting three months for B300 allocation or deploying today on MI355X, the cost-per-token calculation becomes secondary to the opportunity cost of not deploying. This is where the MI355X case is strongest — not because AMD hardware is better, but because it is available and B300 is not.

Latency-sensitive applications: The B300’s 172 tok/s single-stream decode versus MI355X’s 118 tok/s is directly visible to users as 45% faster text generation. Combined with the B300’s faster prefill (~2× advantage for large contexts), the user experience difference is measurable for interactive applications. Cost-per-token is the wrong optimization target here. Latency-per-dollar is, and B300 wins that calculation clearly.

Multi-model clusters: If you are serving multiple frontier models on the same infrastructure (K3, GLM5.2, DeepSeek V4-Pro), the ROCm software overhead multiplies. Each new model deployment on AMD risks discovering ROCm-specific bugs in inference framework features that happen to be in that model’s critical path. CUDA gives you more consistent out-of-the-box behavior across diverse model architectures. The fixed cost of ROCm engineering is lower when amortized over a single dedicated K3 cluster; it is higher when shared across diverse model deployments.

AMD’s Real Play Here Is Not MI355X vs B300 — It Is the Memory Tier Positioning

The framing of “MI355X is cheaper” undersells what AMD is actually demonstrating, and the more interesting long-term argument is about HBM capacity as a competitive dimension.

NVIDIA’s GPU lineup is primarily optimized around compute throughput (FLOP/s) and interconnect bandwidth (NVLink). HBM capacity is a secondary consideration — NVIDIA GPUs have historically been engineered to maximize throughput metrics for training and for models that fit on standard node configurations. The B200 at 192 GB HBM3e reflects the model sizes that existed when it was specified, roughly centered on hundreds-of-billions parameter range at 8-GPU node scale.

K3 at 2.8T parameters represents a new class: models too large for a standard 8×B200 node at full context. If this trajectory continues — and the step from GLM5.2 at 753B to DeepSeek V4-Pro at 1.6T to K3 at 2.8T suggests it will — HBM capacity per GPU becomes an increasingly critical axis of competition.

AMD has been shipping 288 GB HBM3e per device since the MI355X launch. NVIDIA’s response is the B300 at the same 288 GB figure. At the high-memory tier, NVIDIA and AMD are now at parity on memory per device. AMD maintains a significant pricing advantage. If the next generation of frontier open-weights models consistently requires 288 GB or more per GPU to run at single-node scale, AMD’s MI-series becomes structurally relevant regardless of the CUDA moat arguments — not because it is better, but because it clears the memory prerequisite that B200 does not.

This is the “is memory the moat?” question in Wafer’s title. Memory is becoming a necessary condition for serving the models that matter, making it a prerequisite rather than a differentiator. The CUDA moat still exists — but it only matters if you can fit the model. The MI355X’s 288 GB per GPU means AMD clears the prerequisite hurdle for K3-class models that B200 cannot clear on a single node. That structural fact matters more than any specific benchmark row.

What the CUDA Kernel Gap Actually Looks Like in Practice

The ROCm kernel coverage gap in 2026 is different in kind from what it was in 2022 or 2023. It is no longer a broad inability to run models — the core inference path for popular architectures is largely functional. What remains is a long tail of optimization features that get tested on CUDA first and reach ROCm later, or not at all for models that AMD has not specifically targeted.

The two bugs Wafer hit illustrate the pattern precisely. The top_k_renorm_prob missing from sglang’s ROCm build is not a core inference failure — it is a sampling path that only executes under specific conditions and which someone would have needed to explicitly test and port. The AITER MLA prefill kernel shape mismatch for 12 attention heads per rank is not a fundamental prefill failure — it is a configuration guard that was not written to handle K3’s specific TP8 head-count math. Both are fixable with small amounts of code. Both require someone with the right knowledge to diagnose them as configuration problems rather than fundamental limitations.

The pattern I expect: AMD will fix both issues upstream in the ROCm/sglang stack. Future models with similar architectural properties will hit them less. Future models with different architectural properties will hit different versions of the same class of problem. The ROCm software tax is not going to zero — it is converging to a smaller but nonzero fixed cost per new model, concentrated in the optimization features (speculative decode, specialized prefill kernels, quantization formats) rather than the core inference path.

The claim that “agents are closing this gap” (from Wafer’s post) is plausible for the subset of problems that are well-specified and diagnosable. The top_k_renorm_prob bug is exactly the kind of thing an agent can identify and fix given the right error message and code context. The prefill kernel shape mismatch is also tractable. What is harder to automate is the discovery of which optimization features are missing before they cause production problems — that still requires a human who knows what to look for.

The Wafer Credibility Question

A few HN commenters raised credibility questions about Wafer specifically — their WaferPass subscription plan launched and was then discontinued as “unsustainable,” and some critics see their benchmark posts as marketing designed to drive inference business rather than honest engineering analysis.

I want to address this because it affects how much weight to put on their numbers. My read: the technical content in the post is detailed enough to be real engineering work, and the bugs they describe and fixes they applied are plausible and consistent with what I know about ROCm’s coverage gaps in sglang. The framing is clearly optimistic about AMD — they chose the comparison points that make AMD look best, and the “performance per dollar” headline is the most favorable possible summary of data that shows AMD losing on every raw throughput metric. The cost comparison relies on market prices that are contestable. A company selling inference services on AMD hardware has an obvious incentive to publish benchmarks showing AMD in the best light.

None of this means the numbers are fabricated. It means you should treat them as directionally correct and use them for understanding the relative positioning of hardware options, not as definitive benchmarks you would stake a procurement decision on without doing your own testing. The absence of published code to reproduce their setup is the most legitimate criticism. Run your own numbers if the procurement decision is large enough to justify it.

What I Expect Over the Next Twelve Months

These are my falsifiable predictions, written in August 2026, to be revisited in August 2027.

Prediction 1: B300 spot prices will decline toward $4.50/GPU-hr by mid-2027 as supply increases. At that price level, the B300’s 1.65× throughput advantage roughly offsets the cost ratio against MI355X at current MI355X pricing ($2.50-2.95/GPU-hr), making them cost-equivalent per token served. The MI355X cost-efficiency advantage largely disappears if B300 price drops to $4.50 while MI355X stays flat.

Prediction 2: The next two or three frontier open-weights models after K3 will exceed 3 trillion parameters. This continues the trajectory and keeps the 288 GB HBM3e per device requirement relevant for at least 18 more months, extending MI355X’s structural relevance for the large-model segment.

Prediction 3: ROCm day-0 support for major models from Chinese labs (Moonshot, ByteDance, Z.ai, DeepSeek) will continue improving, while ROCm support for models from US labs (Meta, Mistral, Cohere) in popular inference frameworks will remain 4-8 weeks behind CUDA for optimization features. AMD has specific business incentives to support the Chinese lab models that are driving the open-weights frontier.

Prediction 4: No major US-based inference API provider (Together AI, Fireworks AI, Replicate, Anyscale) will deploy their primary K3 serving on MI355X by end of 2026. The software stack risk plus B300 supply uncertainty makes it a harder sell to infrastructure teams optimizing for operational reliability over marginal cost efficiency.

Prediction 5: AMD will make a hardware announcement by mid-2027 specifically targeting memory capacity — either significantly higher HBM per device than MI355X or a disaggregated memory architecture for multi-node configurations. The K3-class model trajectory and the competitive pressure from B300 on throughput means AMD’s next competitive move is almost certainly on the memory axis rather than the compute axis.

The Actual Conclusion

The MI355X is a viable choice for Kimi K3 inference under specific conditions: B300 unavailability, high-concurrency batch workloads, engineering teams with ROCm debugging experience, and cost sensitivity that justifies the software overhead. Those conditions exist at a small number of specialized inference providers right now. They are unlikely to describe the majority of teams evaluating K3 infrastructure.

The interesting part of this story is not “AMD wins on cost” — it is that Kimi K3 is the first model in the open-weights space that specifically breaks the B200 single-node assumption in a way that creates a structural hardware opening. Every model at or above K3’s 2.8T parameter count with 1M context requirements puts B200 at a real disadvantage. AMD and the B300 are the two options that fit. AMD is cheaper today. B300 is faster and has better software. B300 supply is constrained. AMD is available.

In the twelve months I covered building inference for large language models at a startup, we made the wrong hardware choice twice by optimizing for the benchmark that was easiest to measure rather than the constraint that actually bottlenecked our deployment. The K3 situation is the same kind of trap. The benchmark people are arguing about is throughput and cost-per-token. The constraint that will actually drive decisions is memory capacity and procurement lead time. If you only have 30 minutes to make the hardware decision, optimize for whether the model fits and whether you can get the hardware. Everything else is refinement.

A Word on Why the “AI Slop” Criticism of Wafer’s Post Is Counterproductive

I want to spend a paragraph on the tone of the HN thread because I think it is bad for the community and worth naming. Multiple top-level comments dismissed the Wafer post as “AI slop” and “lazily written” within minutes of it hitting the front page. Some of these comments accumulated significant upvotes before anyone had read far enough into the post to evaluate whether the technical substance was sound.

The stylistic markers people are using to identify AI-generated writing — em-dashes as connectives, certain sentence rhythms, the structure of paragraphs that walk through a problem step by step — are stylistic markers that also appear in good technical writing by humans. I use em-dashes constantly. I write paragraphs that walk through problems step by step because that is how you explain engineering decisions. If the standard for “not AI slop” is “does not use any writing patterns that AI also uses,” then the standard is useless as a quality signal because it incorrectly flags human writing.

The legitimate criticisms of the Wafer post are: the prefill latency fix was described without updating the headline numbers; the price comparison uses market spot prices that are contestable; the code to reproduce the benchmarks was not published; the WaferPass history raises some credibility questions about the company’s framing choices. These are substantive objections. “Contains em-dashes” is not.

The HN pattern of front-page style critique (aesthetic dismissal before content evaluation) is, ironically, the same kind of low-effort response the critics are accusing the writers of producing. If you are going to spend time engaging with a technical benchmark post, spend it on the methodology, not the punctuation. The MI355X vs B300 question is interesting and the answer is not obvious. That was true regardless of whether a human or an AI helped write the sentences.

The Inference Provider Landscape Implications

Zooming out from the K3-specific decision: if the open-weights model size frontier continues growing past 2.8T parameters, the set of hardware that can serve these models on a single node is going to remain restricted to the high-memory tier — MI355X equivalents and B300 equivalents — for at least the next 18 months while the B200 remains the dominant deployed base.

This creates an interesting market dynamic. Inference providers who are primarily deployed on B200 (which includes most of the large ones, given B200 was the dominant GPU during 2025 buildouts) will either need to add B300 or MI355X nodes specifically for K3-class models, or skip K3 serving and wait for a B200-compatible model. The capital expenditure of adding high-memory nodes for a single model tier is not obviously justified at current open-weights model adoption rates, which remain modest compared to closed-source API alternatives.

The providers most likely to serve K3 at scale are the ones who: built on AMD from the beginning (like Wafer) or have existing B300 allocations for other reasons; operate in the research/enterprise segment where customers specifically want K3 for its 2.8T parameter reasoning capability; or have cost structures where the MI355X economics work even at lower utilization rates.

The providers least likely to serve K3 at scale are those optimizing primarily for consumer API economics, where the cost of high-memory GPU nodes is hard to recover from individual API calls at competitive pricing. A 1M-context K3 inference request that takes 51 seconds of prefill time (or even the improved 20-25 seconds after AITER MLA fixes) is a resource-intensive operation, and the pricing model for long-context inference at K3 scale is not yet established in the market.

I think what we are watching with the K3 / MI355X story is the early indicator of a hardware segmentation in the inference market: commodity inference (B200-class, models under 1T parameters, established CUDA software) versus frontier inference (high-memory tier, 2T+ parameter models, more hardware diversity). The CUDA moat matters more in the commodity segment. Memory capacity and procurement access matter more in the frontier segment. AMD’s MI355X case is strongest in the frontier segment, which is currently small but is likely to grow as frontier model capabilities increase.