XGBoost
XGBoost is an optimized distributed gradient boosting library designed to be highly efficient, flexible, and portable. It implements machine learning algorithms under the Gradient Boosting framework.
Neptune + XGBoost integration, lets you automatically log many types of metadata during training.
What is logged?
metrics,
parameters,
learning rate,
pickled model,
visualizations (feature importance chart and tree visualizations),
hardware consumption (CPU, GPU, Memory),
stdoutandstderrlogs, andtraining code and git commit information
To get started with this integration, follow the quickstart below. You can also check the more options section to see what you can do with this integration.
If you want to try it out now you can either:
This quickstart will show you how to:
install required libraries,
log metadata during training (metrics, parameters, pickled model, etc.),
check results in the Neptune app.
At the end of this quickstart, you will be able to add Neptune to your XGBoost scripts and use it in your experimentation.
Before you start, make sure that:
You have
python 3.6+in your system,You are already registered user, so that you can log metadata to your private projects.
You have your
NEPTUNE_API_TOKENset to the environment variable: check this docs page.
Depending on your operating system open a terminal or CMD and run this command. All required libraries are available via pip and conda:
pip install neptune-client xgboost>=1.3.0 neptune-xgboost
conda install -c conda-forge neptune-client xgboost>=1.3.0 neptune-xgboost
This integration is tested with neptune-client==0.9.18, xgboost==1.4.0, and neptune-xgboost==0.9.10.
For more help see installing neptune-client.
If you want to have hardware monitoring logged (recommended) you should additionally install psutil.
pip install psutil
conda install psutil
If you want to log visualized trees after training (recommended), you need to install graphviz.
The below installation is only for the pure Python interface to the graphviz software. You need to install graphviz separately. Check graphviz docs for installation help.
pip install graphviz
conda install -c conda-forge python-graphviz
To start logging metadata (metrics, parameters, etc.) during training you need to use NeptuneCallback.
import neptune.new as neptunefrom neptune.new.integrations.xgboost import NeptuneCallback# Create runrun = neptune.init(project="my_workspace/my_project")# Create neptune callbackneptune_callback = NeptuneCallback(run=run, log_tree=[0, 1, 2, 3])# Prepare data, parameters, etc....# Train the model and log metadata to the run in Neptunexgb.train(params=model_params,dtrain=dtrain,callbacks=[neptune_callback],)
import neptune.new as neptuneimport xgboost as xgbfrom neptune.new.integrations.xgboost import NeptuneCallbackfrom sklearn.datasets import load_bostonfrom sklearn.model_selection import train_test_split# Create runrun = neptune.init(project="common/xgboost-integration",api_token="ANONYMOUS",name="xgb-train",tags=["xgb-integration", "train"],)# Create neptune callbackneptune_callback = NeptuneCallback(run=run, log_tree=[0, 1, 2, 3])# Prepare dataX, y = load_boston(return_X_y=True)X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.2,random_state=123)dtrain = xgb.DMatrix(X_train, label=y_train)dval = xgb.DMatrix(X_test, label=y_test)# Define parametersmodel_params = {"eta": 0.7,"gamma": 0.001,"max_depth": 9,"objective": "reg:squarederror","eval_metric": ["mae", "rmse"]}evals = [(dtrain, "train"), (dval, "valid")]num_round = 57# Train the model and log metadata to the run in Neptunexgb.train(params=model_params,dtrain=dtrain,num_boost_round=num_round,evals=evals,callbacks=[neptune_callback,xgb.callback.LearningRateScheduler(lambda epoch: 0.99**epoch),xgb.callback.EarlyStopping(rounds=30)],)
Read docstrings of the NeptuneCallback to learn more about parameters.
import
NeptuneCallbackthat you will use to handle metadata logging,Create a new run in Neptune,
Pass run object to the
NeptuneCallback,Pass the created
neptune_callbackto the train function.
At this point, your script is ready to use Neptune as a logger.
Now, you can run your script and have metadata logged to Neptune for further inspection, comparison, and sharing:
python main.py
In Neptune app it will look similar to this:
Run has the following metadata:
Name | Description |
booster_config | All parameters for the booster. |
early_stopping |
|
epoch | Epochs (visualized as a chart from 1 to last epoch). |
learning_rate | Learning rate visualized as a chart. |
pickled_model | Trained model logged as a pickled file. |
plots | Feature importance and visualized trees. |
train | Training metrics |
valid | Validation metrics |
Note that all metadata above is logged under common namespace "training". You can change this namespace when creating neptune_callback, like this:
import neptune.new as neptunefrom neptune.new.integrations.xgboost import NeptuneCallback# Create runrun = neptune.init(project="my_workspace/my_project")# Create neptune callback with custom base_namespaceneptune_callback = NeptuneCallback(run=run,base_namespace="my_custom_name",)
import neptune.new as neptuneimport xgboost as xgbfrom neptune.new.integrations.xgboost import NeptuneCallbackfrom sklearn.datasets import load_bostonfrom sklearn.model_selection import train_test_split# Create runrun = neptune.init(project="common/xgboost-integration",api_token="ANONYMOUS",name="xgb-train",tags=["xgb-integration", "train"],)# Create neptune callbackneptune_callback = NeptuneCallback(run=run,base_namespace="my_custom_name",log_tree=[0, 1, 2, 3],)# Prepare dataX, y = load_boston(return_X_y=True)X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.2,random_state=123)dtrain = xgb.DMatrix(X_train, label=y_train)dval = xgb.DMatrix(X_test, label=y_test)# Define parametersmodel_params = {"eta": 0.7,"gamma": 0.001,"max_depth": 9,"objective": "reg:squarederror","eval_metric": ["mae", "rmse"]}evals = [(dtrain, "train"), (dval, "valid")]num_round = 57# Train the model and log metadata to the run in Neptunexgb.train(params=model_params,dtrain=dtrain,num_boost_round=num_round,evals=evals,callbacks=[neptune_callback,xgb.callback.LearningRateScheduler(lambda epoch: 0.99**epoch),xgb.callback.EarlyStopping(rounds=30)],)
You can run the example presented above by yourself or see it in Neptune:
You can use NeptuneCallback in the XGBoost.cv function:
import neptune.new as neptunefrom neptune.new.integrations.xgboost import NeptuneCallback# Create runrun = neptune.init(project="my_workspace/my_project")# Create neptune callbackneptune_callback = NeptuneCallback(run=run, log_tree=[0, 1, 2, 3])# Prepare data, parameters, etc....# Run cross validation and log metadata to the run in Neptunexgb.cv(params=model_params,dtrain=dtrain,callbacks=[neptune_callback],)
import neptune.new as neptuneimport xgboost as xgbfrom neptune.new.integrations.xgboost import NeptuneCallbackfrom sklearn.datasets import load_bostonfrom sklearn.model_selection import train_test_split# Create runrun = neptune.init(project="common/xgboost-integration",api_token="ANONYMOUS",name="xgb-cv",tags=["xgb-integration", "cv"],)# Create neptune callbackneptune_callback = NeptuneCallback(run=run, log_tree=[0, 1, 2, 3])# Prepare dataX, y = load_boston(return_X_y=True)X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.2,random_state=123)dtrain = xgb.DMatrix(X_train, label=y_train)dval = xgb.DMatrix(X_test, label=y_test)# Define parametersmodel_params = {"eta": 0.7,"gamma": 0.001,"max_depth": 9,"objective": "reg:squarederror","eval_metric": ["mae", "rmse"]}evals = [(dtrain, "train"), (dval, "valid")]num_round = 57# Run cross validation and log metadata to the run in Neptunexgb.cv(params=model_params,dtrain=dtrain,num_boost_round=num_round,nfold=7,callbacks=[neptune_callback],)
Read docstrings of the NeptuneCallback to learn more about parameters.
Import
NeptuneCallbackthat you will use to handle metadata logging,Create a new run in Neptune,
Pass run object to the
NeptuneCallback,Pass created
neptune_callbackto the XGBoost.cv function.
At this point, your script is ready to use Neptune as a logger.
Now, you can run your script and have metadata logged to Neptune for further inspection, comparison, and sharing:
python main.py
In Neptune it will look similar to this:
The run above has metadata:
Name | Description |
epoch | Epochs (visualized as a chart from 1 to last epoch) |
learning_rate | Learning rate visualized as a chart. |
test | Test metrics |
train | Train metrics |
Also, this run has fold_0 to fold_6 namespaces. This is because for each fold in CV (7-fold CV in this case), Neptune logs metadata:
fold_N|—— booster_config|—— pickled_model|—— plots|—— importance|—— trees
Name | Description |
booster_config | All parameters for the booster. |
pickled_model | Trained model logged as a pickled file. |
plots | Feature importance and visualized trees. |
You can run the example presented above by yourself or see it in Neptune:
You can use NeptuneCallback with the XGBoost sklearn API:
import neptune.new as neptunefrom neptune.new.integrations.xgboost import NeptuneCallback# Create runrun = neptune.init(project="my_workspace/my_project")# Create neptune callbackneptune_callback = NeptuneCallback(run=run, log_tree=[0, 1, 2, 3])# Prepare data, parameters, etc....# Create regressor objectreg = xgb.XGBRegressor(**model_params)# Fit the model and log metadata to the run in Neptunereg.fit(X_train,y_train,callbacks=[neptune_callback],)
import neptune.new as neptuneimport xgboost as xgbfrom neptune.new.integrations.xgboost import NeptuneCallbackfrom sklearn.datasets import load_bostonfrom sklearn.model_selection import train_test_split# Create runrun = neptune.init(project="common/xgboost-integration",api_token="ANONYMOUS",name="xgb-sklearn-api",tags=["xgb-integration", "sklearn-api"],)# Create neptune callbackneptune_callback = NeptuneCallback(run=run, log_tree=[0, 1, 2, 3])# Prepare databoston = load_boston()y = boston['target']X = boston['data']X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.2,random_state=123)# Define parametersmodel_params = {"n_estimators": 70,"eta": 0.7,"gamma": 0.001,"max_depth": 9,"objective": "reg:squarederror","eval_metric": ["mae", "rmse"],}reg = xgb.XGBRegressor(**model_params)# Fit the model and log metadata to the run in Neptunereg.fit(X_train,y_train,early_stopping_rounds=30,eval_metric=['mae', 'rmse'],eval_set=[(X_train, y_train), (X_test, y_test)],callbacks=[neptune_callback,xgb.callback.LearningRateScheduler(lambda epoch: 0.99**epoch),])
Read docstrings of the NeptuneCallback to learn more about parameters.
import
NeptuneCallbackthat you will use to handle metadata logging,Create a new run in Neptune,
Pass run object to the
NeptuneCallback,Pass the created
neptune_callbackto the.fit()method of the regressor (sklearn API).
At this point, your script is ready to use Neptune as a logger.
Now, you can run your script and have metadata logged to Neptune for further inspection, comparison, and sharing:
python main.py
In Neptune it will look similar to this:
Run has the following metadata:
Name | Description |
booster_config | All parameters for the booster. |
early_stopping |
|
epoch | Epochs (visualized as a chart from 1 to last epoch). |
learning_rate | Learning rate visualized as a chart. |
pickled_model | Trained model logged as a pickled file. |
plots | Feature importance and visualized trees. |
validation_0 | Metrics on the 1st validation set passed to the |
validation_1 | Metrics on the 2nd validation set passed to the |
You can run the example presented above by yourself or see it in Neptune:
You can resume a run that you created before and continue logging to it. It comes useful when you train an XGBoost model in multiple training sessions. Here is how to do it:
If you have other types of metadata that are not covered in this integration, you can still log them by using the neptune-client. When you create the run, you have the my_run handle:
# Create new Neptune runmy_run = neptune.init(project="my_workspace/my_project")
You can use my_run object to log metadata. Here is more info about it:
Please visit the Getting help page. Everything regarding support is there:
You may also find the following pages useful:
