Ensemble Methods for Machine Learning MEAP V06
Copyright
Welcome
Brief contents
Chapter 1: Ensemble Learning: Hype or Hallelujah?
1.1 Ensemble Methods: The Wisdom of the Crowds
1.2 Why You Should Care About Ensemble Learning
1.3 Fit vs. Complexity in Individual Models
1.3.1 Regression with Decision Trees
1.3.2 Regression with support vector machines
1.4 Our First Ensemble
1.5 Summary
Chapter 2: Homogeneous Parallel Ensembles: Bagging and Random Forests
2.1 Parallel Ensembles
2.2 Bagging: Bootstrap Aggregating
2.2.1 Intuition: Resampling and Model Aggregation
2.2.2 Implementing Bagging
2.2.3 Bagging with scikit-learn
2.2.4 Faster Training with Parallelization
2.3 Random Forests
2.3.1 Randomized Decision Trees
2.3.2 Random Forests with scikit-learn
2.3.3 Feature Importances
2.4 More Homogeneous Parallel Ensembles
2.4.1 Pasting
2.4.2 Random Subspaces and Random Patches
2.4.3 ExtraTrees
2.5 Case Study: Breast Cancer Diagnosis
2.5.1 Loading and pre-processing
2.5.2 Bagging, Random Forests and ExtraTrees
2.5.3 Feature importances with Random Forests
2.6 Summary
Chapter 3: Heterogeneous Parallel Ensembles: Combining Strong Learners
3.1 Base estimators for heterogeneous ensembles
3.1.1 Fitting base estimators
3.1.2 Individual predictions of base estimators
3.2 Combining predictions by weighting
3.2.1 Majority Vote
3.2.2 Accuracy weighting
3.2.3 Entropy weighting
3.2.4 Dempster-Shafer Combination
3.3 Combining predictions by meta-learning
3.3.1 Stacking
3.3.2 Stacking with cross validation
3.4 Case Study: Sentiment Analysis
3.4.1 Pre-processing
3.4.2 Dimensionality Reduction
3.4.3 Stacking classifiers
3.5 Summary
Chapter 4: Sequential Ensembles: Boosting
4.1 Sequential Ensembles of Weak Learners
4.2 AdaBoost: ADAptive BOOSTing
4.2.1 Intuition: Learning with Weighted Examples
4.2.2 Implementing AdaBoost
4.2.3 AdaBoost with scikit-learn
4.3 AdaBoost in Practice
4.3.1 Learning Rate
4.3.2 Early Stopping and Pruning
4.4 Case Study: Handwritten Digit Classification
4.4.1 Dimensionality Reduction with t-SNE
4.4.2 Boosting
4.5 LogitBoost: Boosting with the Logistic Loss
4.6 Summary
Chapter 5: Sequential Ensembles: Gradient Boosting
5.1 Gradient Descent for Minimization
5.1.1 Gradient Descent with an Illustrative Example
5.1.2 Gradient Descent over Loss Functions for Training
5.2 Gradient Boosting: Gradient Descent + Boosting
5.2.1 Intuition: Learning with Residuals
5.2.2 Implementing Gradient Boosting
5.2.3 Gradient Boosting with scikit-learn
5.2.4 Histogram-based Gradient Boosting
5.3 LightGBM: A Framework for Gradient Boosting
5.3.1 What Makes LightGBM โLightโ?
5.3.2 Gradient Boosting with LightGBM
5.4 LightGBM in Practice
5.4.1 Learning Rate
5.4.2 Early Stopping
5.4.3 Custom Loss Functions
5.5 Case Study: Document Retrieval
5.5.1 The LETOR Data Set
5.5.2 Document Retrieval with LightGBM
5.6 Summary
Chapter 6: Sequential Ensembles: Newton Boosting
6.1 Newtonโs Method for Minimization
6.1.1 Newtonโs Method with an Illustrative Example
6.1.2 Newton Descent over Loss Functions for Training
6.2 Newton Boosting: Newtonโs Method + Boosting
6.2.1 Intuition: Learning with Weighted Residuals
6.2.2 Intuition: Learning with Regularized Loss Functions
6.2.3 Implementing Newton Boosting
6.3 XGBoost: A Framework for Newton Boosting
6.3.1 What Makes XGBoost โExtremeโ?
6.3.2 Newton Boosting with XGBoost
6.4 XGBoost in Practice
6.4.1 Learning Rate
6.4.2 Early Stopping
6.5 Case Study Redux: Document Retrieval
6.5.1 The LETOR Data Set
6.5.2 Document Retrieval with XGBoost
6.6 Summary
Chapter 7: Learning with Continuous and Count Labels
7.1 A Brief Review of Regression
7.1.1 Linear Regression for Continuous Labels
7.1.2 Poisson Regression for Count Labels
7.1.3 Logistic Regression for Classification Labels
7.1.4 Generalized Linear Models
7.1.5 Nonlinear Regression
7.2 Parallel Ensembles for Regression
7.2.1 Random Forest and ExtraTrees
7.2.2 Combining Regression Models
7.2.3 Stacking Regression Models
7.3 Sequential Ensembles for Regression
7.3.1 Loss and Likelihood Functions for Regression
7.3.2 Gradient Boosting with LightGBM and XGBoost
7.4 Case Study: Demand Forecasting
7.4.1 The UCI Bike Rental Data Set
7.4.2 Generalized Linear Models and Stacking
7.4.3 Random Forest and ExtraTrees
7.4.4 XGBoost and LightGBM
7.5 Summary
Chapter 8: Learning with Categorical Features
8.1 Encoding Categorical Features
8.1.1 Types of Categorical Features
8.1.2 Ordinal and One-Hot Encoding
Ordinal Encoding
One-Hot Encoding
8.1.3 Encoding with Target Statistics
Greedy Target Encoding
Information Leakage and Distribution Shift
Hold-out & Leave-One-Out Target Encoding
8.1.4 The category_encoders Package
8.2 CatBoost: A Framework for Ordered Boosting
8.2.1 Ordered Target Statistics and Ordered Boosting
Ordered Target Statistics
Ordered Boosting
8.2.2 Oblivious Decision Trees
8.2.3 CatBoost in Practice
Cross Validation with CatBoost
Early Stopping with CatBoost
8.3 Case Study: Income Prediction
8.3.1 The Adult Census Data Set
8.3.2 Creating Preprocessing and Modeling Pipelines
8.3.3 Category Encoding and Ensembling
Random Forest
LightGBM
XGBoost
8.3.4 Ordered Encoding and Boosting with CatBoost
8.4 Encoding High-Cardinality String Features
The dirty-cat package
8.5 Summary
Chapter 9: Explaining Your Ensembles
9.1 What is Interpretability?
9.1.1 Black-Box vs. Glass-Box Models
9.1.2 Decision Trees (and Decision Rules)
Interpreting Decision Trees in Practice
Feature Importances
9.1.3 Generalized Linear Models
Feature Weights
Interpreting GLMs in Practice
9.2 Case Study: Data-driven Marketing
9.2.1 The Bank Telemarketing Data Set
9.2.2 Training Ensembles
9.2.3 Feature Importances in Tree Ensembles
9.3 Black-Box Methods for Global Explainability
9.3.1 Permutation Feature Importance
Permutation Feature Importance in Practice
9.3.2 Partial Dependence Plots
Partial Dependence Plots in Practice
9.3.3 Global Surrogate Models
The Fidelity-Interpretability Tradeoff
Training Global Surrogate Models in Practice
9.4 Black-Box Methods for Local Explainability
9.4.1 Local Surrogate Models with LIME
The Fidelity-Interpretability Tradeoff Again
Sampling Surrogate Examples for Local Explainability
LIME in Practice
9.4.2 Local Interpretability with SHAP
Understanding Shapley Values
Shapley Values as Feature Importance
SHapley Additive exPlanations (SHAP)
SHAP in Practice
9.5 Glass-Box Ensembles: Training for Interpretability
9.5.1 Explainable Boosting Machines (EBMs)
Generalized Additive Models with Feature Interactions
Training EBMs
9.5.2 EBMs in Practice
9.6 Summary