Neptune - Scikit-Optimize Integration

What will you get with this integration?

Scikit-Optimize, or skopt, is a simple and efficient library to minimize (very) expensive and noisy black-box functions. It is often used to Machine Learning model hyperparameter optimization.

With Neptune integration, you can:

  • visualize the runs as they are running,

  • see charts of logged run scores,

  • log the parameters tried at every run,

  • log figures from plots module: plot_evaluations, plot_convergence, plot_objective, and plot_regret,

  • monitor hardware consumption during the run,

  • save the pickled results object.

Tip

You can log many other experiment metadata like interactive charts, video, audio, and more. See the full list.

Note

This integration is tested with neptune-client==0.4.132, neptune-contrib==0.25.0, and scikit-optimize==0.8.1

Where to start?

To get started with this integration, follow the quickstart below.

You can also see how to log BayesSearchCV parameter sweeps in the more options section.

If you want to try things out and focus only on the code you can either:

Quickstart

This quickstart will show you how to:

  • Install the necessary neptune packages

  • Connect Neptune to your Skopt hyperparameter tuning code and create the first experiment

  • Log run metrics and parameters for every search configuration

  • Log best metric and best parameter configuration

  • Log diagnostic plots: convergence plot, evaluations plot, and objective plot from skopt.plotting

  • Explore them in the Neptune UI.

Before you start

You have Python 3.x and following libraries installed:

You also need minimal familiarity with skopt. Have a look at the skopt guide to get started.

pip install --quiet scikit-optimize==0.8.1 neptune-client neptune-contrib['monitoring']

Step 1: Initialize Neptune

import neptune

neptune.init(api_token='ANONYMOUS',
             project_qualified_name='shared/scikit-optimize-integration')

Tip

You can also use your personal API token. Read more about how to securely set the Neptune API token.

Step 2: Create an Experiment

neptune.create_experiment(name='skopt sweep')

This also creates a link to the experiment. Open the link in a new tab. The charts will currently be empty, but keep the window open. You will be able to see live metrics once logging starts.

Step 3: Run skopt with the Neptune Callback

NeptuneCallback() will log the run metrics, run parameters and results pickle after every iteration. Everything can be inspected live.

# Create Neptune Callback
import neptunecontrib.monitoring.skopt as skopt_utils
neptune_callback = skopt_utils.NeptuneCallback()

# Run the skopt minimize function with the Neptune Callback
results = skopt.forest_minimize(objective,
                                space,
                                base_estimator='ET',
                                n_calls=100,
                                n_random_starts=10,
                                callback=[neptune_callback],)

Step 4: Log best parameter configuration, best score and diagnostic plots

You can log additional information from skopt results after the tuning has completed with the log_results() function.

This will log:

  • Best score for the sweep as ‘best_score’ metric

  • Best parameter set as ‘best_parameters’ property

  • Fog figures from plots module: plot_evaluations, plot_convergence, plot_objective, and plot_regret to the ‘diagnostics’ log.

skopt_utils.log_results(results)

Note

You can change the Neptune experiment to which the results are logged with the experiment parameter, and choose whether or not you want to log plots and the pickle objects with the log_plots and log_pickle parameters.

Step 5: See your Skopt tuning in Neptune

Now you can switch to the Neptune tab which you had opened previously to watch the tuning live!

Neptune-Skopt Integration

More Options

Use Neptune with BayesSearchCV

Step 1: Initialize Neptune and create an experiment

import neptune

neptune.init(api_token='ANONYMOUS',
             project_qualified_name='shared/scikit-optimize-integration')

neptune.create_experiment(name='skopt sweep')

Tip

You can also use your personal API token. Read more about how to securely set the Neptune API token.

Step 2: Initialize BayesSearchCV

opt = BayesSearchCV(
    SVC(),
    {
        'C': Real(1e-6, 1e+6, prior='log-uniform'),
        'gamma': Real(1e-6, 1e+1, prior='log-uniform'),
        'degree': Integer(1,8),
        'kernel': Categorical(['linear', 'poly', 'rbf']),
    },
    n_iter=32,
    random_state=0
)

Step 2: Pass Neptune callback to the .fit method

import neptunecontrib.monitoring.skopt as skopt_utils

opt.fit(X_train, y_train,
        callback=skopt_utils.NeptuneCallback())

Step 3: Log best parameter set and diagnostic plots to Neptune

You can log best parameter, diagnostic plots and results pickle to Neptune with the log_results() function. To access the optimization results object you should use the ._optim_results attribute of the BayesSearchCV object.

skopt_utils.log_results(opt._optim_results[0])

Remember that you can try it out with zero setup:

How to ask for help?

Please visit the Getting help page. Everything regarding support is there.