Knowledge Vault

Utility-Based Information Distillation Over Temporally Sequenced Documents

2 min read

Overview

Yang, Lad, Lao et al. (SIGIR 2007) — Introduces utility-based information distillation, a framework that combines adaptive filtering (AF), novelty detection (ND), and anti-redundant passage ranking into a unified system for tracking evolving information needs over temporal document streams.

The Problem

Conventional adaptive filtering has three limitations:

  1. Passive user role — users can only confirm/reject system decisions, not actively refine queries or highlight relevant spans
  2. Document-level granularity — feedback is all-or-nothing on entire documents, not fine-grained text passages
  3. Redundancy — major events generate repeated coverage across sources, wasting user attention

The CAFE System

CMU Adaptive Filtering Engine (CAFE) has four core components:

  • Adaptive Filtering: Regularized logistic regression learns query profiles from fine-grained user feedback (highlighted text spans as positive examples)
  • Passage Retrieval: TF-IDF scoring against learned query profiles with relevance thresholds
  • Novelty Detection: Cosine similarity against user history — removes passages too similar to previously seen content
  • Anti-Redundant Ranking: Simplified Maximal Marginal Relevance (MMR) ensures within-list diversity

Evaluation Innovation

Introduces two evaluation contributions:

  • Nugget-matching rules — semi-automatic Boolean queries that reliably detect whether a passage contains a specific information nugget, enabling automated evaluation of arbitrary-span system responses
  • Discounted Cumulated Utility (DCU) — extends NDCG to penalize redundancy via a dampening factor gamma and include a reading cost term, measuring utility as gain minus loss

Results

On the extended TDT4 corpus (90K+ news articles, 12 tasks, 120 queries):

  • Feedback + novelty detection + anti-redundancy yields best NDCU scores
  • User feedback more beneficial under gamma=0.1 (small redundancy tolerance) than gamma=0 (zero tolerance), revealing that feedback-driven retrieval tends to surface same-term passages

Source Note

Two copies of this file exist in the source pool with different filename casing — identical content (534 lines each).

  • path-ranking-algorithm — Ni Lao’s later work on relational retrieval using constrained random walks, extending the IR foundations from this paper
  • never-ending-language-learner — NELL system at CMU where these information extraction techniques were further developed

Graph View

Backlinks