Overview
DeepSeek introduces ‘Engram,’ a conditional memory module that modernizes N-gram embeddings to provide O(1) knowledge lookup. By separating static knowledge retrieval from dynamic neural computation, the architecture improves efficiency and performance across reasoning, coding, and long-context tasks.
Abstract
While Mixture-of-Experts (MoE) scales capacity via conditional computation, Transformers lack a native primitive for knowledge lookup, forcing expensive runtime reconstruction of static patterns. This paper introduces conditional memory as a complementary sparsity axis, instantiated via Engram — a module that modernizes classic N-gram embedding for O(1) lookup. A U-shaped scaling law optimizes the trade-off between neural computation (MoE) and static memory (Engram).
Methodology
Engram Architecture: A conditional memory module that augments Transformer backbones by separating static pattern storage from dynamic computation. Uses tokenizer compression, multi-head hashing, context-aware gating, and multi-branch integration.
Sparse Retrieval: Local context (N-grams) maps to static memory entries via deterministic hashing. Lightweight multiplicative-XOR hash with multiple heads mitigates collisions.
Context-Aware Gating: Retrieved static embeddings are dynamically modulated by current hidden states, resolving ambiguity from hash collisions or polysemy.
Sparsity Allocation Problem: Given a fixed parameter budget, the paper optimizes capacity distribution between MoE experts and Engram memory, uncovering a U-shaped scaling law.
Key Contributions
- Engram-27B achieves superior performance over iso-parameter/iso-FLOPs MoE baselines
- Gains extend beyond knowledge retrieval (MMLU +3.4) to general reasoning (BBH +5.0) and code/math (HumanEval +3.0)
- Mechanistic analysis reveals Engram relieves early backbone layers from static reconstruction, deepening effective network for complex reasoning
- Offloading 100B-parameter table to host memory incurs <3% overhead via deterministic prefetching
- Published January 2026, code at github.com/deepseek-ai/Engram
Significance
Conditional memory represents a new architectural primitive for next-generation sparse models. By structurally separating knowledge lookup from computation, Engram demonstrates that static pattern retrieval is a fundamentally different operation from dynamic reasoning — and both deserve dedicated architectural support.
Related
- dr-zero-self-evolution — complementary research on self-evolving agent architectures
- ai-engineering-foundation-models — production context for foundation model architecture decisions
- ngram-machines-knowledge-qa — earlier N-gram work on knowledge organization and QA