Overview
Sutton & Barto (2018/2020, 2nd Edition, MIT Press) — The foundational textbook on reinforcement learning. CC BY-NC-ND 2.0 licensed. Part of the Adaptive Computation and Machine Learning series.
Structure
Part I: Tabular Solution Methods
- Introduction — RL definition, examples, elements (agent, environment, policy, reward, value function, model), tic-tac-toe extended example, early history
- Multi-armed Bandits — Exploration vs. exploitation, epsilon-greedy, upper confidence bound, gradient bandits
- Finite Markov Decision Processes — MDPs, returns, value functions, Bellman equations
- Dynamic Programming — Policy evaluation, policy improvement, policy/value iteration
- Monte Carlo Methods — First-visit/every-visit MC, MC control, importance sampling
- Temporal-Difference Learning — TD(0), SARSA, Q-learning, expected SARSA
- n-step Bootstrapping — Bridging MC and TD methods
- Planning and Learning — Dyna-Q, model-based integration
Part II: Approximate Solution Methods
9-13. Function approximation, policy gradient methods, eligibility traces, etc.
Part III: Looking Deeper
14-17. Psychology connections, neuroscience connections, applications, frontiers
Significance
This textbook underlies virtually all modern RL research, including GRPO (used in R-Zero, Dr. Zero), RLHF (used in LLM alignment), and PPO (used in game AI). Every self-evolving LLM framework references Sutton & Barto’s formulations.
Key Concepts
| Concept | Definition |
|---|---|
| Agent | Learner and decision maker |
| Environment | Everything the agent interacts with |
| Policy | Mapping from states to actions |
| Reward Signal | Immediate feedback from environment |
| Value Function | Expected cumulative future reward |
| Model | Agent’s representation of environment dynamics |
Related
- r-zero-self-evolving-reasoning — Uses GRPO (derived from PPO) for self-evolution
- dr-zero-self-evolution — Applies RL to search agent evolution
- alignment-problem — Christian’s treatment of RL’s role in AI alignment