Neptune - Scikit-Optimize Integration¶

What will you get with this integration?¶

Scikit-Optimize, or skopt, is a simple and efficient library to minimize (very) expensive and noisy black-box functions. It is often used to Machine Learning model hyperparameter optimization.

With Neptune integration, you can:

visualize the runs as they are running,
see charts of logged run scores,
log the parameters tried at every run,
log figures from plots module: plot_evaluations, plot_convergence, plot_objective, and plot_regret,
monitor hardware consumption during the run,
save the pickled results object.

Tip

You can log many other experiment metadata like interactive charts, video, audio, and more. See the full list.

Note

This integration is tested with neptune-client==0.4.132, neptune-contrib==0.25.0, and scikit-optimize==0.8.1

Where to start?¶

To get started with this integration, follow the quickstart below.

You can also see how to log BayesSearchCV parameter sweeps in the more options section.

If you want to try things out and focus only on the code you can either:

Run in Google Colab

View source on GitHub

See example in Neptune

Quickstart¶

This quickstart will show you how to:

Install the necessary neptune packages
Connect Neptune to your Skopt hyperparameter tuning code and create the first experiment
Log run metrics and parameters for every search configuration
Log best metric and best parameter configuration
Log diagnostic plots: convergence plot, evaluations plot, and objective plot from skopt.plotting
Explore them in the Neptune UI.

Before you start¶

You have Python 3.x and following libraries installed:

neptune-client, and neptune-contrib. See neptune-client installation guide.
scikit-optimize==0.8.1. See skopt installation guide.

You also need minimal familiarity with skopt. Have a look at the skopt guide to get started.

pip install --quiet scikit-optimize==0.8.1 neptune-client neptune-contrib['monitoring']

Step 1: Initialize Neptune¶

import neptune

neptune.init(api_token='ANONYMOUS',
             project_qualified_name='shared/scikit-optimize-integration')

Tip

You can also use your personal API token. Read more about how to securely set the Neptune API token.

Step 2: Create an Experiment¶

neptune.create_experiment(name='skopt sweep')

This also creates a link to the experiment. Open the link in a new tab. The charts will currently be empty, but keep the window open. You will be able to see live metrics once logging starts.

Step 3: Run skopt with the Neptune Callback¶

NeptuneCallback() will log the run metrics, run parameters and results pickle after every iteration. Everything can be inspected live.

# Create Neptune Callback
import neptunecontrib.monitoring.skopt as skopt_utils
neptune_callback = skopt_utils.NeptuneCallback()

# Run the skopt minimize function with the Neptune Callback
results = skopt.forest_minimize(objective,
                                space,
                                base_estimator='ET',
                                n_calls=100,
                                n_random_starts=10,
                                callback=[neptune_callback],)

Step 4: Log best parameter configuration, best score and diagnostic plots¶

You can log additional information from skopt results after the tuning has completed with the log_results() function.

This will log:

Best score for the sweep as ‘best_score’ metric
Best parameter set as ‘best_parameters’ property
Fog figures from plots module: plot_evaluations, plot_convergence, plot_objective, and plot_regret to the ‘diagnostics’ log.

skopt_utils.log_results(results)

Note

You can change the Neptune experiment to which the results are logged with the experiment parameter, and choose whether or not you want to log plots and the pickle objects with the log_plots and log_pickle parameters.

Step 5: See your Skopt tuning in Neptune¶

Now you can switch to the Neptune tab which you had opened previously to watch the tuning live!

More Options¶

Use Neptune with BayesSearchCV¶

Step 1: Initialize Neptune and create an experiment¶

import neptune

neptune.init(api_token='ANONYMOUS',
             project_qualified_name='shared/scikit-optimize-integration')

neptune.create_experiment(name='skopt sweep')

Tip

You can also use your personal API token. Read more about how to securely set the Neptune API token.

Step 2: Initialize BayesSearchCV¶

opt = BayesSearchCV(
    SVC(),
    {
        'C': Real(1e-6, 1e+6, prior='log-uniform'),
        'gamma': Real(1e-6, 1e+1, prior='log-uniform'),
        'degree': Integer(1,8),
        'kernel': Categorical(['linear', 'poly', 'rbf']),
    },
    n_iter=32,
    random_state=0
)

Step 2: Pass Neptune callback to the `.fit` method¶

import neptunecontrib.monitoring.skopt as skopt_utils

opt.fit(X_train, y_train,
        callback=skopt_utils.NeptuneCallback())

Step 3: Log best parameter set and diagnostic plots to Neptune¶

You can log best parameter, diagnostic plots and results pickle to Neptune with the log_results() function. To access the optimization results object you should use the ._optim_results attribute of the BayesSearchCV object.

skopt_utils.log_results(opt._optim_results[0])

Remember that you can try it out with zero setup: