Bagging means “Boostrap Aggregation”. It refers to an approach to ensemble models in which models are trained independently and then aggregated. It takes a model which tends to overfit and helps correct for that overfitting. Random forest uses it.
- Uses random sampling with replacement
- Trains models independently
- Combines predictions using voting or averaging
- Reduces overfitting by reducing variance, averaging leaves bias unchanged and cuts variance. So this helps only if the model overfits through high variance.
- Needs an unstable base learner; if all learners are the same then aggregation accomplishes nothing.
- Used in Random Forest, which is bagging + random feature subsets at each split
Boosting is an iterative approach which combines many weak learners in series.
- train one model
- calculate the residuals (error) made by the first model
- train another model to predict those errors
- repeat
- mainly reduces bias