Nvidia buying Groq tells you everything you need to know about where [[Inference]] is heading. This isn't a defensive move. It's Nvidia building the complete stack for a world where inference disaggregates into specialized workloads, each with its own optimal architecture. ![[Screenshot 2025-12-27 at 01.12.05.png]] Let me be clear about what's happening here. Inference is splitting into [[Prefill and Decode]]. [[SRAM]] architectures have unique advantages in decode for workloads where performance is primarily a function of [[Memory Bandwidth]]. Nvidia now has three Rubin variants, each optimized for specific patterns. Rubin CPX handles massive [[Context Window]] during prefill with super high memory capacity using relatively low bandwidth [[GDDR]] DRAM. Standard Rubin is the workhorse for training and high-density batched inference, with [[HBM]] DRAM striking that balance between memory bandwidth and capacity. And the Groq-derived Rubin SRAM variant? That's built for ultra-low latency Agentic AI reasoning workloads, using SRAM's extremely high memory bandwidth at the cost of lower memory capacity. > This is the playbook: mix and match chips to create the optimal balance of performance versus cost for each workload. It's elegant and it's devastating to competitors. ![[Screenshot 2025-12-27 at 01.12.24.png]] The second reason matters even more for market dynamics. SRAM architectures can hit token-per-second metrics much higher than GPUs, [[TPU]]s, or any [[ASIC]] we've seen. Extremely low latency per individual user at the expense of throughput per dollar. Eighteen months ago, it wasn't clear whether users would actually pay for this speed, given that SRAM is more expensive per token due to much smaller [[Batch Size]]. That question is now settled. Cerebras and Groq's recent results make it abundantly clear: **users are willing to pay for speed**. This has major implications for the competitive landscape. I'm increasingly confident that all ASICs except TPU, AI5, and Trainium will eventually be canceled. Good luck competing with three Rubin variants plus multiple associated networking chips. The scale and integration advantages are overwhelming. ![[Screenshot 2025-12-27 at 01.17.43.png]] OpenAI's ASIC sounds surprisingly good, much better than what Meta and Microsoft are building. Intel is moving in this direction with their prefill-optimized SKU and the SambaNova acquisition, though SambaNova was always the weakest SRAM competitor. Meta bought Rivos. These moves tell you everyone sees the same future, but Nvidia is three steps ahead. Which brings me to Cerebras. They're now in a very interesting and highly strategic position as the last independent SRAM player (per public knowledge) that was ahead of Groq on all public benchmarks. Groq's "many chip" rack architecture was much easier to integrate with Nvidia's networking stack, perhaps even within a single rack. Cerebras's WSE almost has to be an independent rack. That architectural difference matters for how these systems get deployed and integrated. The inference layer is consolidating around memory architecture optimization. My [[AI Inference Infrastructure]] thesis is playing out in real time: specialized architectures for specialized workload patterns. The question isn't whether this happens. It's who captures the value. --- #deeptech #inference Related: [[Inference]] | [[AI Inference Infrastructure]] | [[The short case for NVIDIA]] | [[Prefill and Decode]] | [[SRAM]] | [[HBM]] | [[GDDR]] | [[Memory Bandwidth]] | [[Context Window]] | [[Batch Size]] | [[ASIC]] | [[TPU]] Ref: [[Personal Investment Roundup#Memory & Edge Computing Shift]]