Knowledge Vault

Overfitting

2 min read

Overview

Overfitting occurs when a model learns the noise and idiosyncrasies of its training data rather than the underlying patterns, resulting in excellent training performance but poor generalization to new data. It is the most common failure mode in machine learning. A model that memorizes training examples instead of learning generalizable features will produce confident but incorrect predictions on unseen inputs. Overfitting risk increases with model complexity, small training datasets, and long training duration. The opposite problem — underfitting — occurs when the model is too simple to capture the underlying patterns.

Key Concepts

  • Bias-variance trade-off — Underfitting (high bias) means the model is too simple. Overfitting (high variance) means it’s too complex. The goal is the sweet spot.
  • Regularization — Techniques that penalize model complexity. L1 (Lasso) drives weights to zero for feature selection. L2 (Ridge) shrinks weights to reduce sensitivity.
  • Dropout — Randomly disabling neurons during training, forcing the network to learn redundant representations. A form of implicit ensemble.
  • Early stopping — Monitoring validation loss during training and stopping when it begins to increase, even if training loss continues decreasing.
  • Data augmentation — Generating synthetic training examples through transformations (rotation, cropping, noise). Effectively increases dataset size.
  • Cross-validation — Using multiple train/validation splits to get a more robust estimate of generalization performance.
  • Learning curves — Plotting training vs validation loss over time. Diverging curves signal overfitting.

Trade-offs

  • Regularization improves generalization but can hurt performance on complex patterns
  • Early stopping requires choosing the right patience/threshold
  • More data is the most reliable fix but not always available
  • Model ensembles reduce overfitting but increase inference cost

Graph View

Backlinks