Overview
This research introduces the N-Gram Machine (NGM), an end-to-end question-answering framework designed to overcome the scalability limits of neural networks by using symbolic knowledge storage. By representing text as indexed n-grams, the system achieves query complexity that is independent of the corpus size, allowing for efficient reasoning over massive datasets.
Abstract
This paper (Yang et al., 2019) proposes N-Gram Machines (NGM), a scalable end-to-end QA system that uses symbolic meaning representations to overcome the scalability limitations of deep neural QA models. Existing end-to-end models process the entire text after observing the question, making their complexity linear in text size — prohibitive for Wikipedia-scale or web-scale corpora. NGM uses n-gram-based knowledge storage that can be indexed and retrieved with complexity independent of the text size.
Methodology
Architecture: An end-to-end trainable system combining:
- A text auto-encoding component that converts text into symbolic knowledge storage (structured n-grams)
- A memory-enhanced sequence-to-sequence component for answering questions from the encoded knowledge
- A deterministic executor that runs programs against the knowledge storage
Key Innovation: Both the schema (structure) and content of knowledge storage are treated as discrete hidden variables, inferred automatically from weak supervision (QA pairs). The system learns what information to store and how to organize it.
N-Gram Representation: Unlike conventional contiguous text n-grams, NGM n-grams can be any combination of arbitrary words and symbols, enabling representation of complex semantics.
Key Contributions
- Successfully solves bAbI synthetic text tasks as proof-of-concept
- Scales to millions of sentences in “life-long bAbI” experiments
- Induces latent structure (schema) from natural language Wikipedia text using only QA pairs as weak supervision
- Achieves O(1) lookup complexity independent of corpus size via indexable knowledge storage
Significance
NGM bridges the gap between the scalability of classical information retrieval (indexable, fast) and the learning capability of neural networks (flexible, end-to-end trainable). This work anticipates later developments in RAG architectures and the Engram conditional memory module (DeepSeek, 2026).
Related
- deepseek-conditional-memory — later work (2026) extending N-gram concepts into modern LLM architectures
- compact-guide-to-rag — RAG as the production evolution of knowledge-augmented QA
- zettelkasten-implementation — knowledge organization principles connecting to the KM aspects of this work