XGBoost: The Definitive Information (Half 2) | by Dr. Roi Yehoshua | Aug, 2023

Implementation of the XGBoost algorithm in Python from scratch

Within the previous article we mentioned the XGBoost algorithm and confirmed its implementation in pseudocode. On this article we’re going to implement the algorithm in Python from scratch.

The offered code is a concise and light-weight implementation of the XGBoost algorithm (with solely about 300 strains of code), meant to display its core performance. As such, it’s not optimized for pace or reminiscence utilization, and doesn’t embody the complete spectrum of choices offered by the XGBoost library (see https://xgboost.readthedocs.io/ for extra particulars on the options of the library). Extra particularly:

1. The code is written in pure Python, whereas the core of the XGBoost library is written in C++ (its Python courses are solely skinny wrappers over the C++ implementation).
2. It doesn’t embody numerous optimizations that enable XGBoost to take care of large quantities of information, resembling weighted quantile sketch, out-of-core tree studying, and parallel and distributed processing of the information. These optimizations shall be mentioned in additional element within the subsequent article within the sequence.
3. The present implementation helps solely regression and binary classification duties, whereas the XGBoost library additionally helps multi-class classification and rating issues.
4. Our implementation helps solely a small subset of the hyperparameters that exist within the XGBoost library. Particularly, it helps the next hyperparameters:

• n_estimators (default = 100): the variety of regression timber within the ensemble (which can also be the variety of boosting iterations).
• max_depth (default = 6): the utmost depth (variety of ranges) of every tree.
• learning_rate (default = 0.3): the step measurement shrinkage utilized to the timber.
• reg_lambda (default = 1): L2 regularization time period utilized to the weights of the leaves.
• gamma (default = 0): minimal loss discount required to separate a given node.

For consistency, I’ve stored the identical names and default values of those hyperparameters as they’re outlined within the XGBoost library.