Knowledge Vault

Proposer-Solver-Evaluator Pattern

3 min read

Proposer-Solver-Evaluator Pattern

Problem

How do you build a knowledge base through self-directed learning? Traditional approaches require manual curation, but we want a system that can:

  • Generate its own learning challenges
  • Attempt to solve them using available resources
  • Evaluate its own performance
  • Adapt difficulty based on results

Solution

Implement a four-component co-evolution loop:

┌──────────────┐   Challenge    ┌──────────────┐
│   PROPOSER   │ ────────────→  │    SOLVER    │
│              │                │              │
└──────────────┘ ←──────────────└──────────────┘
       │            Answer             │
       │                               ↓
       │                        ┌──────────────┐
       │                        │  KNOWLEDGE   │
       │                        │   SOURCE     │
       │                        └──────────────┘
       │                               │
       └────────────┬──────────────────┘
                    ↓
             ┌──────────────┐
             │  EVALUATOR   │
             └──────────────┘
                    │
                    ↓
             ┌──────────────┐
             │  CURRICULUM  │
             └──────────────┘

Components

Proposer: Generates challenges of varying types and difficulties

  • Tracks which challenge types cause struggle
  • Adapts generation based on curriculum feedback
  • Can use templates or AI generation

Solver: Attempts to answer using RAG against knowledge source

  • Query extraction from challenge
  • Retrieval from local/offline sources
  • Answer generation with citations
  • Self-assessed confidence

Evaluator: Multi-dimensional scoring

  • Domain-specific metrics (accuracy, correctness, clarity)
  • Weighted combination for overall score
  • Human-readable feedback generation

Curriculum Manager: Controls difficulty progression

  • Promotion when recent average exceeds threshold
  • Demotion when performance drops
  • Minimum iterations at each level before changes

Usage

TypeScript Interfaces

interface Challenge {
  id: string;
  text: string;
  type: ChallengeType;
  difficulty: DifficultyLevel;
  topics: string[];
  expectedAspects?: string[];
}
 
interface Answer {
  challengeId: string;
  text: string;
  confidence: number;
  sources: SourceCitation[];
  reasoning?: string;
}
 
interface EvaluationResult {
  challengeId: string;
  scores: Record<string, number>;
  overall: number;
  feedback: string;
}
 
interface CurriculumState {
  currentDifficulty: DifficultyLevel;
  iterationsAtLevel: number;
  recentScores: number[];
  promotionThreshold: number;
  demotionThreshold: number;
}

Evolution Loop

async function runIteration(state: CurriculumState): Promise<CurriculumState> {
  // 1. Generate challenges at current difficulty
  const challenges = await proposer.generate(state.currentDifficulty, CHALLENGES_PER_ITERATION);
 
  // 2. Solve each challenge
  const answers = await Promise.all(challenges.map((c) => solver.solve(c)));
 
  // 3. Evaluate each answer
  const evaluations = await Promise.all(
    answers.map((a, i) => evaluator.evaluate(challenges[i], a))
  );
 
  // 4. Update curriculum
  const avgScore = evaluations.reduce((sum, e) => sum + e.overall, 0) / evaluations.length;
 
  return curriculum.update(state, avgScore);
}

Curriculum Adaptation

function updateCurriculum(state: CurriculumState, score: number): CurriculumState {
  const newScores = [...state.recentScores.slice(-9), score];
  const avgRecent = newScores.reduce((a, b) => a + b) / newScores.length;
 
  let newDifficulty = state.currentDifficulty;
  let newIterations = state.iterationsAtLevel + 1;
 
  if (newIterations >= 3) {
    if (avgRecent >= state.promotionThreshold) {
      newDifficulty = promoteDifficulty(state.currentDifficulty);
      newIterations = 0;
    } else if (avgRecent <= state.demotionThreshold) {
      newDifficulty = demoteDifficulty(state.currentDifficulty);
      newIterations = 0;
    }
  }
 
  return {
    ...state,
    currentDifficulty: newDifficulty,
    iterationsAtLevel: newIterations,
    recentScores: newScores,
  };
}

Examples

Dr. Zero Local (Wikipedia Q&A)

  • Knowledge Source: Wikipedia via Kiwix (offline)
  • Challenge Types: Factual, Comparative, Causal, Temporal, Multi-hop
  • Evaluation: Accuracy (50%), Completeness (30%), Relevance (20%)
  • Status: 10 iterations completed, calibrating retrieval

JP Code Academy (Coding Encyclopedia)

  • Knowledge Source: freeCodeCamp, language docs (offline)
  • Challenge Types: Conceptual, Comparison, Implementation, Debugging, Refactoring, Best Practice
  • Evaluation: Correctness (40%), Completeness (30%), Clarity (20%), Best Practices (10%)
  • Status: POC scaffolding complete

Trade-offs

Advantages

  • Self-directed learning without manual curation
  • Automatic difficulty adaptation
  • Checkpoint/resume capability
  • Structured knowledge output
  • Works with local/offline resources

Challenges

  • Requires quality offline resources
  • Calibration period to tune thresholds
  • Evaluation metrics need domain tuning
  • RAG retrieval quality critical to success

When to Use

  • Building domain-specific knowledge bases
  • Testing LLM capabilities systematically
  • Creating structured learning progressions
  • Generating training/evaluation datasets

When Not to Use

  • Real-time applications (loop is iterative)
  • When ground truth is unavailable
  • Domains without retrievable sources

Source

Validated through Dr. Zero Local POC (10 iterations, 100 questions).

Inspired by Meta’s Dr. Zero paper on self-evolving reasoning models.

Graph View

Backlinks