The Future of Inference: Inside d-Matrix’s "Raptor" 3D DRAM Architecture

At the prestigious Hot Chips 2026 conference, AI hardware startup d-Matrix unveiled "Raptor," a groundbreaking compute architecture that claims to solve the growing bottlenecks of generative AI inference. By flipping the traditional 3D-stacking paradigm on its head and integrating a custom-designed DRAM die directly beneath a high-performance compute chip, d-Matrix is making a bold play to redefine how large language models (LLMs) are served at scale.

While the industry remains locked in a desperate scramble for High Bandwidth Memory (HBM) capacity, d-Matrix is betting on a proprietary, vertical integration strategy. If the company’s projections hold, Raptor could deliver a staggering 4.7x increase in throughput compared to current HBM-based designs, potentially turning the tide for inference-heavy AI deployments.

The Technical Breakthrough: Inverting the Stack

The fundamental design of Raptor centers on a TSMC 4nm compute die bonded face-to-face with a custom-engineered DRAM die. This vertical arrangement utilizes a 36-micron pitch interface, enabling a massive 100 TB/s of aggregate memory bandwidth from a 32GB per-card configuration.

According to d-Matrix co-founder and CTO Sudeep Bhoja, the energy efficiency of this vertical interface is a critical differentiator. He cites a measured cost of 0.37 pJ/bit for data movement between the logic and memory dies, a massive improvement over the approximately 2.4 pJ/bit required to move data into an HBM4 base die.

Unlike conventional architectures where the memory is stacked on top, or peripheral to the compute logic, Raptor places the logic die on top. This allows a liquid cooling cold plate to sit directly on the compute silicon, effectively managing the thermal output. The DRAM die underneath serves a dual purpose: it acts as both a memory reservoir and an active interposer, routing PCIe and die-to-die signals downward through Through-Silicon Vias (TSVs).

Managing Thermal and Data Integrity

Running high-performance DRAM at a junction temperature of 105°C presents significant engineering hurdles. At these elevated temperatures, memory retention collapses from the standard 32ms to a mere 4ms, forcing the system to refresh eight times more frequently than traditional designs.

d-Matrix circumvented this potential performance disaster by implementing an ultra-granular memory architecture. By shrinking each microbank to 1,366 rows and roughly 5.33MB, the company ensured that a full refresh cycle consumes only 1.37% of the total available bandwidth. Furthermore, the system employs a two-level multiplexer chain—dubbed "bank chaining"—which allows the system to route around faulty banks. With 840 banks per chiplet and 72 dedicated as spares, the system maintains symmetric channels even in the face of localized silicon defects.

A Brief Chronology of Development

The path to the Raptor reveal at Hot Chips 2026 has been marked by iterative progress and ambitious engineering milestones:

• September 2025: The "Pavehawk" 3DIMC (3D Integrated Memory and Compute) test silicon officially comes online, proving the feasibility of the vertical interface architecture.
• June 2026: CEO Sid Sheth confirms to CNBC that the company is collaborating with major industry players to move toward production, with an official product launch for Raptor targeted for 2027.
• August 2026: The full technical disclosure of Raptor occurs at Hot Chips 2026, accompanied by an ISCA 2026 research paper co-authored with the University of British Columbia, detailing the energy efficiency and throughput advantages of the design.

Despite these milestones, d-Matrix has remained tight-lipped regarding specific production volumes, pricing models, and, most crucially, the identity of the DRAM foundry responsible for manufacturing their bespoke memory dies.

Supporting Data and Performance Projections

d-Matrix’s claims are built on simulations anchored by early silicon characterization. In its presentation, the company highlighted a performance figure of 988 tokens per second per user on the massive 2.8-trillion-parameter Kimi K3 model at a 1-million-token context window.

For comparison, the industry has looked toward the Cerebras WSE-3 as a benchmark for high-throughput inference, which recorded 969 tokens per second on the Llama 3.1 405B model in late 2024. If the d-Matrix figures hold up in real-world, third-party validation, Raptor would represent a monumental leap in efficiency, especially considering the Kimi K3 model is nearly seven times the size of Llama 3.1 405B.

The Power Budget

The ISCA 2026 paper outlines a per-package power budget of 422W, of which 296W is consumed by the vertical interface at peak operation. To combat the loss of power to data-bus inversion (DBI)—which was impossible due to the lack of a traditional burst structure or sideband pins—d-Matrix implemented a clever alternative: each 128-byte flit is compared to its predecessor. A 1-bit inversion tag is stored alongside the ECC metadata, allowing the system to recover approximately 20% of the I/O power that would otherwise be wasted.

The "Who Supplies the Wafers?" Dilemma

The most glaring question surrounding Raptor is its supply chain. d-Matrix has publicly credited TSMC for the N4P logic die and Alchip for the ASIC design and packaging. However, neither of these firms manufactures DRAM.

The DRAM market is an oligopoly controlled by three major players—Samsung, SK hynix, and Micron—all of whom are currently prioritizing their capacity for HBM4 production, which is largely sold out through the end of 2026. J.P. Morgan estimates that DRAM contract prices have risen by over 400% between 2024 and 2026.

This creates a high-stakes bottleneck. Even industry giants like Nvidia are reportedly struggling to secure enough memory, recently testing lower-capacity configurations for their upcoming "Rubin Ultra" chips. For a startup with $450 million in funding to negotiate for custom, bespoke DRAM die manufacturing in an environment where even standard HBM is scarce, the path to 2027 volume production remains fraught with risk.

When pressed on this during the Hot Chips Q&A, CTO Sudeep Bhoja acknowledged that the current memory density (11.4 MB/mm²) is roughly half that of HBM4 (21.9–26.3 MB/mm²). He noted that this density penalty is a result of using a "not-so-advanced" DRAM process to facilitate the initial design, suggesting that future iterations could bridge the gap if they can secure more mainline foundry support.

Implications for the AI Industry

The arrival of Raptor signals a shift in focus from raw compute power to memory-centric architecture. As LLMs grow in size and context window requirements, the "memory wall"—the latency and energy cost of moving data between memory and processors—has become the primary bottleneck for AI inference.

The Scale-Up vs. Scale-Out Debate

d-Matrix is positioning Raptor for rack-scale deployment. By linking 72 cards together, they claim to hold a 2.8-trillion-parameter model in memory at 4-bit precision, with enough headroom to support 54 concurrent users. This is a "scale-up" approach intended to minimize the need for inter-card communication, which is notoriously slow and energy-intensive.

However, as the co-presenter Aayush Ankit, formerly of d-Matrix and now with Meta, noted, "Communication becomes a bottleneck soon enough." Once the model size or the KV (Key-Value) cache requirements exceed the 2.3TB capacity of a 72-card rack, the system must inevitably spill over into inter-card synchronization. The very vertical bandwidth that Raptor optimizes becomes less relevant once the system must traverse the physical network fabric of a data center.

The Road Ahead

Raptor is an undeniably innovative response to the HBM crisis. By co-designing the logic and memory dies, d-Matrix has created a tightly coupled, highly efficient inference engine that challenges the status quo.

However, the transition from an impressive Hot Chips presentation to mass-market availability is a chasm that few hardware startups successfully cross. Whether d-Matrix can convince a major memory supplier to allocate precious wafer capacity to a custom, non-standard design remains the critical variable. As the AI industry continues to demand more efficiency per watt and per dollar, d-Matrix has provided a compelling proof of concept; now, the clock is ticking to see if they can scale that vision before the market shifts to the next generation of HBM5 and beyond.