Every model architecture before Engram was designed…

This matters more than it sounds. Let me explain precisely why.

HBM -- high bandwidth memory, the stacked DRAM on every H100 and B200 -- is optimized for one specific access pattern: large, sequential, contiguous reads. The attention kernel streaming the KV cache. The FFN kernel loading weight matrices row by row. These are predictable, large, coalesced memory accesses where HBM's 3-8 TB/s bandwidth actually saturates the compute units feeding off it. HBM was designed for this. The GPU was designed for this. The memory controller was designed for this.

CXL memory -- DDR5 DRAM hanging off a CXL controller on the PCIe bus -- has lower sequential bandwidth than HBM by about 10-20x. But it has something HBM doesn't: large capacity at reasonable random-access latency. Where HBM tops out at 192GB on a B200, a CXL pool can reach tens of terabytes. Where HBM's random-access performance degrades proportionally to the random-access fraction of the workload (because the memory controller loses efficiency on non-coalesced accesses), CXL's DRAM cells handle random access natively.

Engram (arXiv:2601.07372, DeepSeek team, January 2026) is the first large model architecture that treats this as a first-order design constraint. Not "let's put some parameters in CXL because we ran out of HBM." Let's design the memory access pattern of a new parameter type around what CXL is actually good at.

What Engram is, precisely.

Standard transformers have two types of parameters: attention weights (applied to every token, compute-intensive, sequential access) and FFN weights (applied to every token, memory-bandwidth-intensive, sequential access). MoE factorizes the FFN into experts and activates only a subset per token -- a step toward sparsity.

Engram introduces a third parameter type: conditional memory, activated by lookup rather than routing. For each token, a hash of the token's hidden state is used to look up a small subset of parameters from a large memory table. The lookup is conditional on content (what the token represents) rather than position (where it is in the sequence). Different tokens with similar content retrieve similar parameter subsets.

The memory table can be enormous -- billions or trillions of parameters that are never all loaded simultaneously. Each inference step loads a tiny fraction via content-addressed lookup. The total parameter count scales independently of the inference-time computational cost.

The access pattern this creates: random reads into a large table, with keys determined by content hashes at inference time. Non-sequential. Non-coalesced. Determined by the model's computation, not by a predefined schedule.

This is the access pattern that HBM handles badly. A random-read pattern into a 1TB Engram table on HBM would achieve a tiny fraction of HBM's theoretical bandwidth because the memory controller can't coalesce non-adjacent accesses. You'd be paying for 3.35 TB/s and using perhaps 100-300 GB/s effectively.

CXL handles this correctly because CXL's memory cells are DRAM -- designed for random-access workloads, the same DRAM that's been serving random access in CPUs for decades. The CXL controller adds latency for the interconnect hop but preserves the DRAM cell's natural access pattern. A 640ns CXL access to a random Engram entry is 640ns. An HBM access to the same entry would be faster in isolation but the memory controller inefficiency at random access patterns means the effective throughput for many concurrent random lookups is worse.

The CXL Engram pooling paper (arXiv:2603.10087).

This paper does something I haven't seen before: benchmarks a model architecture specifically on the memory tier it was designed for, rather than on the tier every other paper assumes.

The setup: Engram parameters offloaded to an XConn XC50256 CXL 2.0 switch, the same hardware Alibaba used for Beluga (which I wrote about earlier). The inference framework (SGLang) treats the CXL pool as a backing store for Engram lookups. During the forward pass, when the model needs to retrieve Engram parameters for a batch of tokens, it dispatches lookup requests to the CXL pool, overlaps the CXL memory access latency with attention computation on the GPU, and merges the retrieved parameters into the computation when they arrive.

The result: near-DRAM end-to-end performance on inference. Meaning the CXL tier's access latency is close enough to local DRAM latency (because the access pattern is what CXL/DRAM is good at) that the total inference throughput doesn't degrade meaningfully from the Engram lookup step.

The cost implication: CXL DRAM is roughly 4-5x cheaper per GB than HBM. If the Engram table is offloaded entirely to CXL, you're holding billions of parameters at 1/5 the cost of storing them in GPU HBM. For a model where Engram represents 40-60% of total parameters (a plausible fraction for a model specifically designed around this architecture), the per-GB cost savings compound directly into total serving cost.

The four-tier memory hierarchy that's emerging.

I want to draw the full picture because I think the Engram paper is the piece that completes it and nobody has said it as a unified thing.

SRAM (Groq LPX, TPU 8i): 150-500 TB/s bandwidth. Optimized for the access pattern that dominates decode: sequential streaming of FFN weight matrices through the compute pipeline. Every decode step loads the same large matrices in the same order. SRAM is on-chip, deterministic, 20x faster effective bandwidth than HBM for this specific sequential pattern.

HBM (every GPU): 3-8 TB/s. Optimized for large coalesced reads where the access pattern is predictable but not fixed -- attention over a growing KV cache, prefill over a long prompt. Sequential enough for HBM's memory controller to coalesce effectively. Not so sequential that SRAM can hold it all.

CXL DRAM (Engram table, KV cold tier): 200-600 GB/s effective for sequential, near-native DRAM latency for random access. Optimized for: content-addressed lookup (Engram), cold KV cache retrieval for long sessions (the HMA's DRAM tier), large-capacity parameter storage. The 640ns latency is manageable when overlapped with GPU computation.

NVMe (vLLM HMA tier 3, session archive): 5-50 GB/s. For KV states that won't be accessed again until a session resumes hours later. Persistent across server restarts. Cheap per GB. Unacceptably slow for hot access, acceptable for cold resumption.

Each tier is optimized for a different access pattern. Until Engram, every model architecture produced workloads that fit into the top two tiers (SRAM or HBM) -- the sequential access patterns that both HBM and SRAM are good at, just at different sizes and costs. Engram is the first architecture that explicitly targets the third tier -- CXL -- with a parameter type whose access pattern is optimized for CXL's random-access strength.

Why this changes the hardware procurement conversation.

Before Engram: adding CXL to an inference cluster was an optimization for KV cache cold storage. Useful, measurable improvement in TTFT for long-context multi-turn sessions, but not fundamental to the model architecture. You could run the same model with or without CXL.

After Engram: if your model uses conditional memory as a core architectural component, CXL isn't an optimization. It's required for the economics to work. Running Engram parameters in HBM is possible but expensive -- you're paying HBM prices for parameters that HBM is wrong for. Running Engram parameters in CXL is the architecturally correct choice.

This is the first time a model architecture change has created a demand signal for a specific memory tier that wasn't already in most inference clusters. Every CXL vendor (Tetramer, Montage, Samsung, SK Hynix) has been selling CXL on the "capacity extension" argument -- more memory than HBM. Engram gives them a new argument: "our hardware is the correct tier for this parameter type, not just a larger tier for the same parameter type."

The implication for hardware planning: if Engram-style conditional memory becomes a standard architectural component (which I expect, because the parameter efficiency is compelling and the economics scale correctly), you're planning a cluster with explicit CXL capacity allocations for model parameters, not just KV cold storage. The cluster BOM (bill of materials) changes.

The open question: what does co-design look like from here.

The CXL paper demonstrates that Engram's access pattern fits CXL. It doesn't optimize the Engram lookup algorithm for CXL's specific latency profile.

CXL latency is ~640ns with roughly 100-200ns of variability (from queuing at the CXL controller under concurrent loads). Engram lookups for a large batch arrive concurrently -- all tokens in the batch issuing their content-addressed lookups simultaneously. The CXL controller queues these and processes them. The queuing latency adds to the 640ns base.

An Engram implementation optimized for CXL would: batch lookups across the full batch of tokens, submit them to the CXL pool as a coalesced batch (minimizing controller queuing overhead), and overlap the full CXL latency window with GPU attention computation. The overlap schedule needs to be determined at model compile time -- you need to know which CXL accesses are needed at which layers before you start the forward pass so the prefetcher can issue them early enough to arrive on time.

This is the same prefetch scheduling problem that flash attention solved for HBM access -- schedule data movement to arrive when needed without stalling the compute pipeline. Solved for HBM, not yet solved for CXL. The CXL Engram prefetch scheduler is the engineering problem that sits between "Engram works on CXL" and "Engram on CXL achieves full compute utilization."

every model architecture before engram was designed for hbm.

engram was designed for a different tier.

the access pattern -- content-addressed random lookup into a large sparse table -- is exactly what cxl dram handles natively and what hbm handles badly.

the cost implications scale with how large the engram table is relative to total parameters.

the cxl prefetch scheduler -- issuing lookup requests early enough to arrive without stalling the compute pipeline -- is the engineering problem that takes "engram works on cxl" to "engram on cxl is economically dominant over engram on hbm." it hasn't been built yet. that's the next paper.

P.S. The memory price context matters here. Counterpoint Research reported memory prices up 50% in Q4 2025, with a "hyper-bull phase" continuing through 2026. HBM specifically is supply-constrained -- TSMC's advanced packaging (CoWoS) for HBM is booked years out. CXL DRAM runs on standard DDR5 packaging with normal DRAM supply. The economic argument for Engram on CXL isn't just cost-per-GB -- it's availability. If your model can run its conditional memory on DDR5 instead of HBM, you're not competing with every AI infrastructure build for CoWoS packaging allocation. That's a supply chain argument, not just a cost argument. Both point the same direction.