Overview
Two companion papers (2019) on making machine reading comprehension faster without sacrificing accuracy, by the same research group at Cornell University with Ni Lao.
Paper 1: FastFusionNet (arXiv:1902.11291)
Efficient variant of FusionNet that removes expensive CoVe layers and substitutes BiLSTMs with faster SRU (Simple Recurrent Unit) layers. Achieves state-of-the-art on DAWNBench for SQuAD with the lowest training and inference time at time of publication.
Key technique: SRU layers reduce sequential computation while maintaining accuracy, making them the most efficient RNN variant available.
Paper 2: Integrated Triaging (arXiv:1909.13128)
Framework that inserts a lightweight “Triage Module” between early network layers, pruning most context early so deeper layers scan only a tiny fraction of the corpus. Benefits:
- Prevents overfitting to short training paragraphs
- Enables training on golden paragraphs but testing on full documents
- 20% speedup on BERT with 1.8 F1 improvement
- 20% speedup on FastFusionNet with 4.3 F1 improvement
Uses two mechanisms: early-exit (return answer if Triage Module is confident) and context-pruning (remove irrelevant passages for later layers).
Source File Note
Both source files have misleading filenames that don’t match their content. The first is labeled “Contextual Graph Attention…” but contains FastFusionNet. The second is labeled “and Future Directions” but contains Integrated Triaging.
Related
- path-ranking-algorithm — Ni Lao’s earlier work on knowledge graph inference, connecting to this NLP research
- neural-symbolic-machines — Another approach to combining neural and symbolic reasoning for QA tasks