XGBoost

​​
​​
​​
​​
​​
​​
​See code example on GitHub​
​See run logged to Neptune​
​Run example in Google Colab​
What will you get with this integration?
​XGBoost is an optimized distributed gradient boosting library designed to be highly efficient, flexible, and portable. It implements machine learning algorithms under the Gradient Boosting framework.
Neptune + XGBoost integration, lets you automatically log many types of metadata during training.
What is logged?
metrics,
parameters,
learning rate,
pickled model,
visualizations (feature importance chart and tree visualizations),
hardware consumption (CPU, GPU, Memory),
stdout and stderr logs, and
training code and git commit information
Example dashboard with train-valid metrics and selected parameters
Where to start?
To get started with this integration, follow the quickstart below. You can also check the more options section to see what you can do with this integration.
If you want to try it out now you can either:
​​
​​
​​
​​
​​
​​
​See code example on GitHub​
​See run logged to Neptune​
​Run example in Google Colab​
Quickstart
This quickstart will show you how to:
install required libraries,
log metadata during training (metrics, parameters, pickled model, etc.),
check results in the Neptune app.
At the end of this quickstart, you will be able to add Neptune to your XGBoost scripts and use it in your experimentation.
Install requirements
Before you start, make sure that:
You have python 3.6+ in your system,
You are already registered user, so that you can log metadata to your private projects.
You have your NEPTUNE_API_TOKEN set to the environment variable: check this docs page.
Install neptune-client, xgboost, and neptune-xgboost
Depending on your operating system open a terminal or CMD and run this command. All required libraries are available via pip and conda:
pip
conda
1
pip install neptune-client xgboost>=1.3.0 neptune-xgboost
Copied!
1
conda install -c conda-forge neptune-client xgboost>=1.3.0 neptune-xgboost
Copied!
This integration is tested with neptune-client==0.9.18, xgboost==1.4.0, and neptune-xgboost==0.9.10.
For more help see installing neptune-client.
Install graphviz (optional)
If you want to log visualized trees after training (recommended), you need to install graphviz.
The below installation is only for the pure Python interface to the graphviz software. You need to install graphviz separately. Check graphviz docs for installation help.
pip
conda
1
pip install graphviz
Copied!
1
conda install -c conda-forge python-graphviz
Copied!
Log metadata during training
To start logging metadata (metrics, parameters, etc.) during training you need to use NeptuneCallback.
core code
full script
1
import neptune.new as neptune
2
from neptune.new.integrations.xgboost import NeptuneCallback
3
​
4
# Create run
5
run = neptune.init(project="my_workspace/my_project")
6
​
7
# Create neptune callback
8
neptune_callback = NeptuneCallback(run=run, log_tree=[0, 1, 2, 3])
9
​
10
# Prepare data, parameters, etc.
11
...
12
​
13
# Train the model and log metadata to the run in Neptune
14
xgb.train(
15
    params=model_params,
16
    dtrain=dtrain,
17
    callbacks=[neptune_callback],
18
)
Copied!
1
import neptune.new as neptune
2
import xgboost as xgb
3
from neptune.new.integrations.xgboost import NeptuneCallback
4
from sklearn.datasets import load_boston
5
from sklearn.model_selection import train_test_split
6
​
7
# Create run
8
run = neptune.init(
9
    project="common/xgboost-integration",
10
    api_token="ANONYMOUS",
11
    name="xgb-train",
12
    tags=["xgb-integration", "train"],
13
)
14
​
15
# Create neptune callback
16
neptune_callback = NeptuneCallback(run=run, log_tree=[0, 1, 2, 3])
17
​
18
# Prepare data
19
X, y = load_boston(return_X_y=True)
20
X_train, X_test, y_train, y_test = train_test_split(
21
    X,
22
    y,
23
    test_size=0.2,
24
    random_state=123
25
)
26
dtrain = xgb.DMatrix(X_train, label=y_train)
27
dval = xgb.DMatrix(X_test, label=y_test)
28
​
29
# Define parameters
30
model_params = {
31
    "eta": 0.7,
32
    "gamma": 0.001,
33
    "max_depth": 9,
34
    "objective": "reg:squarederror",
35
    "eval_metric": ["mae", "rmse"]
36
}
37
evals = [(dtrain, "train"), (dval, "valid")]
38
num_round = 57
39
​
40
# Train the model and log metadata to the run in Neptune
41
xgb.train(
42
    params=model_params,
43
    dtrain=dtrain,
44
    num_boost_round=num_round,
45
    evals=evals,
46
    callbacks=[
47
        neptune_callback,
48
        xgb.callback.LearningRateScheduler(lambda epoch: 0.99**epoch),
49
        xgb.callback.EarlyStopping(rounds=30)
50
    ],
51
)
Copied!
Read docstrings of the NeptuneCallback to learn more about parameters.
In the snippet above
import NeptuneCallback that you will use to handle metadata logging,
Create a new run in Neptune,
Pass run object to the NeptuneCallback,
Pass the created neptune_callback to the train function.
At this point, your script is ready to use Neptune as a logger.
Now, you can run your script and have metadata logged to Neptune for further inspection, comparison, and sharing:
1
python main.py
Copied!
In Neptune app it will look similar to this:
Logged metadata include parameters and train/valid metrics.
More about the run above
Run has the following metadata:
Name
Description
booster_config
All parameters for the booster.
early_stopping
best_score and best_iteration (logged only if early stopping was activated).
epoch
Epochs (visualized as a chart from 1 to last epoch).
learning_rate
Learning rate visualized as a chart.
pickled_model
Trained model logged as a pickled file.
plots
Feature importance and visualized trees.
train
Training metrics
valid
Validation metrics
Note that all metadata above is logged under common namespace "training". You can change this namespace when creating neptune_callback, like this:
core code
full script
1
import neptune.new as neptune
2
from neptune.new.integrations.xgboost import NeptuneCallback
3
​
4
# Create run
5
run = neptune.init(project="my_workspace/my_project")
6
​
7
# Create neptune callback with custom base_namespace
8
neptune_callback = NeptuneCallback(
9
    run=run,
10
    base_namespace="my_custom_name",
11
)
Copied!
1
import neptune.new as neptune
2
import xgboost as xgb
3
from neptune.new.integrations.xgboost import NeptuneCallback
4
from sklearn.datasets import load_boston
5
from sklearn.model_selection import train_test_split
6
​
7
# Create run
8
run = neptune.init(
9
    project="common/xgboost-integration",
10
    api_token="ANONYMOUS",
11
    name="xgb-train",
12
    tags=["xgb-integration", "train"],
13
)
14
​
15
# Create neptune callback
16
neptune_callback = NeptuneCallback(
17
    run=run,
18
    base_namespace="my_custom_name",
19
    log_tree=[0, 1, 2, 3],
20
)
21
​
22
# Prepare data
23
X, y = load_boston(return_X_y=True)
24
X_train, X_test, y_train, y_test = train_test_split(
25
    X,
26
    y,
27
    test_size=0.2,
28
    random_state=123
29
)
30
dtrain = xgb.DMatrix(X_train, label=y_train)
31
dval = xgb.DMatrix(X_test, label=y_test)
32
​
33
# Define parameters
34
model_params = {
35
    "eta": 0.7,
36
    "gamma": 0.001,
37
    "max_depth": 9,
38
    "objective": "reg:squarederror",
39
    "eval_metric": ["mae", "rmse"]
40
}
41
evals = [(dtrain, "train"), (dval, "valid")]
42
num_round = 57
43
​
44
# Train the model and log metadata to the run in Neptune
45
xgb.train(
46
    params=model_params,
47
    dtrain=dtrain,
48
    num_boost_round=num_round,
49
    evals=evals,
50
    callbacks=[
51
        neptune_callback,
52
        xgb.callback.LearningRateScheduler(lambda epoch: 0.99**epoch),
53
        xgb.callback.EarlyStopping(rounds=30)
54
    ],
55
)
Copied!
What next?
You can run the example presented above by yourself or see it in Neptune:
​​
​​
​​
​​
​​
​​
​See code example on GitHub​
​See run logged to Neptune​
​Run example in Google Colab​
More options
CV
You can use NeptuneCallback in the XGBoost.cv function:
core code
full script
1
import neptune.new as neptune
2
from neptune.new.integrations.xgboost import NeptuneCallback
3
​
4
# Create run
5
run = neptune.init(project="my_workspace/my_project")
6
​
7
# Create neptune callback
8
neptune_callback = NeptuneCallback(run=run, log_tree=[0, 1, 2, 3])
9
​
10
# Prepare data, parameters, etc.
11
...
12
​
13
# Run cross validation and log metadata to the run in Neptune
14
xgb.cv(
15
    params=model_params,
16
    dtrain=dtrain,
17
    callbacks=[neptune_callback],
18
)
Copied!
1
import neptune.new as neptune
2
import xgboost as xgb
3
from neptune.new.integrations.xgboost import NeptuneCallback
4
from sklearn.datasets import load_boston
5
from sklearn.model_selection import train_test_split
6
​
7
# Create run
8
run = neptune.init(
9
    project="common/xgboost-integration",
10
    api_token="ANONYMOUS",
11
    name="xgb-cv",
12
    tags=["xgb-integration", "cv"],
13
)
14
​
15
# Create neptune callback
16
neptune_callback = NeptuneCallback(run=run, log_tree=[0, 1, 2, 3])
17
​
18
# Prepare data
19
X, y = load_boston(return_X_y=True)
20
X_train, X_test, y_train, y_test = train_test_split(
21
    X,
22
    y,
23
    test_size=0.2,
24
    random_state=123
25
)
26
dtrain = xgb.DMatrix(X_train, label=y_train)
27
dval = xgb.DMatrix(X_test, label=y_test)
28
​
29
# Define parameters
30
model_params = {
31
    "eta": 0.7,
32
    "gamma": 0.001,
33
    "max_depth": 9,
34
    "objective": "reg:squarederror",
35
    "eval_metric": ["mae", "rmse"]
36
}
37
evals = [(dtrain, "train"), (dval, "valid")]
38
num_round = 57
39
​
40
# Run cross validation and log metadata to the run in Neptune
41
xgb.cv(
42
    params=model_params,
43
    dtrain=dtrain,
44
    num_boost_round=num_round,
45
    nfold=7,
46
    callbacks=[neptune_callback],
47
)
Copied!
Read docstrings of the NeptuneCallback to learn more about parameters.
In the snippet above
Import NeptuneCallback that you will use to handle metadata logging,
Create a new run in Neptune,
Pass run object to the NeptuneCallback,
Pass created neptune_callback to the XGBoost.cv function.
At this point, your script is ready to use Neptune as a logger.
Now, you can run your script and have metadata logged to Neptune for further inspection, comparison,  and sharing:
1
python main.py
Copied!
In Neptune it will look similar to this:
Example dashboard with cross-validation results.
More about this run
The run above has metadata:
Name
Description
epoch
Epochs (visualized as a chart from 1 to last epoch)
learning_rate
Learning rate visualized as a chart.
test
Test metrics
train
Train metrics
Also, this run has fold_0 to fold_6 namespaces. This is because for each fold in CV (7-fold CV in this case), Neptune logs metadata:
1
fold_N
2
  |—— booster_config
3
  |—— pickled_model
4
  |—— plots
5
        |—— importance
6
        |—— trees
Copied!
Name
Description
booster_config
All parameters for the booster.
pickled_model
Trained model logged as a pickled file.
plots
Feature importance and visualized trees.
What next?
You can run the example presented above by yourself or see it in Neptune:
​​
​​
​​
​​
​​
​​
​See code example on GitHub​
​See run logged to Neptune​
​Run example in Google Colab​
Scikit-learn API
You can use NeptuneCallback with the XGBoost sklearn API:
core code
full script
1
import neptune.new as neptune
2
from neptune.new.integrations.xgboost import NeptuneCallback
3
​
4
# Create run
5
run = neptune.init(project="my_workspace/my_project")
6
​
7
# Create neptune callback
8
neptune_callback = NeptuneCallback(run=run, log_tree=[0, 1, 2, 3])
9
​
10
# Prepare data, parameters, etc.
11
...
12
​
13
# Create regressor object
14
reg = xgb.XGBRegressor(**model_params)
15
​
16
# Fit the model and log metadata to the run in Neptune
17
reg.fit(
18
    X_train,
19
    y_train,
20
    callbacks=[neptune_callback],
21
)
Copied!
1
import neptune.new as neptune
2
import xgboost as xgb
3
from neptune.new.integrations.xgboost import NeptuneCallback
4
from sklearn.datasets import load_boston
5
from sklearn.model_selection import train_test_split
6
​
7
# Create run
8
run = neptune.init(
9
    project="common/xgboost-integration",
10
    api_token="ANONYMOUS",
11
    name="xgb-sklearn-api",
12
    tags=["xgb-integration", "sklearn-api"],
13
)
14
​
15
# Create neptune callback
16
neptune_callback = NeptuneCallback(run=run, log_tree=[0, 1, 2, 3])
17
​
18
# Prepare data
19
boston = load_boston()
20
y = boston['target']
21
X = boston['data']
22
X_train, X_test, y_train, y_test = train_test_split(
23
    X,
24
    y,
25
    test_size=0.2,
26
    random_state=123
27
)
28
​
29
# Define parameters
30
model_params = {
31
    "n_estimators": 70,
32
    "eta": 0.7,
33
    "gamma": 0.001,
34
    "max_depth": 9,
35
    "objective": "reg:squarederror",
36
    "eval_metric": ["mae", "rmse"],
37
}
38
​
39
reg = xgb.XGBRegressor(**model_params)
40
​
41
# Fit the model and log metadata to the run in Neptune
42
reg.fit(
43
    X_train,
44
    y_train,
45
    early_stopping_rounds=30,
46
    eval_metric=['mae', 'rmse'],
47
    eval_set=[(X_train, y_train), (X_test, y_test)],
48
    callbacks=[
49
        neptune_callback,
50
        xgb.callback.LearningRateScheduler(lambda epoch: 0.99**epoch),
51
    ]
52
)
Copied!
Read docstrings of the NeptuneCallback to learn more about parameters.
In the snippet above
import NeptuneCallback that you will use to handle metadata logging,
Create a new run in Neptune,
Pass run object to the NeptuneCallback,
Pass the created neptune_callback to the .fit() method of the regressor (sklearn API).
At this point, your script is ready to use Neptune as a logger.
Now, you can run your script and have metadata logged to Neptune for further inspection, comparison,  and sharing:
1
python main.py
Copied!
In Neptune it will look similar to this:
Example dashboard, where sklearn API was used.
More about the run above
Run has the following metadata:
Name
Description
booster_config
All parameters for the booster.
early_stopping
best_score and best_iteration (logged only if early stopping was activated).
epoch
Epochs (visualized as a chart from 1 to last epoch).
learning_rate
Learning rate visualized as a chart.
pickled_model
Trained model logged as a pickled file.
plots
Feature importance and visualized trees.
validation_0
Metrics on the 1st validation set passed to the eval_set parameter of the model.fit() (sklearn API).
validation_1
Metrics on the 2nd validation set passed to the eval_set parameter of the model.fit() (sklearn API).
What next?
You can run the example presented above by yourself or see it in Neptune:
​​
​​
​​
​​
​​
​​
​See code example on GitHub​
​See run logged to Neptune​
​Run example in Google Colab​
Resume run
You can resume a run that you created before and continue logging to it. It comes useful when you train an XGBoost model in multiple training sessions. Here is how to do it:
Resume run
Log other metadata
If you have other types of metadata that are not covered in this integration, you can still log them by using the neptune-client. When you create the run, you have the my_run handle:
1
# Create new Neptune run
2
my_run = neptune.init(project="my_workspace/my_project")
Copied!
You can use my_run object to log metadata. Here is more info about it:
What can you log and display?
How to ask for help?
Please visit the Getting help page. Everything regarding support is there:
Getting help
Other pages you may like
You may also find the following pages useful:
Hyperparameter Optimization
What can you log and display?
Broken link