Neptune - Scikit-Optimize Integration¶
What will you get with this integration?¶
Scikit-Optimize, or skopt, is a simple and efficient library to minimize (very) expensive and noisy black-box functions. It is often used to Machine Learning model hyperparameter optimization.
With Neptune integration, you can:
visualize the runs as they are running,
see charts of logged run scores,
log the parameters tried at every run,
log figures from plots module: plot_evaluations, plot_convergence, plot_objective, and plot_regret,
monitor hardware consumption during the run,
save the pickled results object.
Tip
You can log many other experiment metadata like interactive charts, video, audio, and more. See the full list.
Note
This integration is tested with neptune-client==0.4.132
, neptune-contrib==0.25.0
, and scikit-optimize==0.8.1
Where to start?¶
To get started with this integration, follow the quickstart below.
You can also see how to log BayesSearchCV
parameter sweeps in the more options section.
If you want to try things out and focus only on the code you can either:
Quickstart¶
This quickstart will show you how to:
Install the necessary neptune packages
Connect Neptune to your Skopt hyperparameter tuning code and create the first experiment
Log run metrics and parameters for every search configuration
Log best metric and best parameter configuration
Log diagnostic plots: convergence plot, evaluations plot, and objective plot from
skopt.plotting
Explore them in the Neptune UI.
Before you start¶
You have Python 3.x
and following libraries installed:
neptune-client
, andneptune-contrib
. See neptune-client installation guide.scikit-optimize==0.8.1
. See skopt installation guide.
You also need minimal familiarity with skopt. Have a look at the skopt guide to get started.
pip install --quiet scikit-optimize==0.8.1 neptune-client neptune-contrib['monitoring']
Step 1: Initialize Neptune¶
import neptune
neptune.init(api_token='ANONYMOUS',
project_qualified_name='shared/scikit-optimize-integration')
Tip
You can also use your personal API token. Read more about how to securely set the Neptune API token.
Step 2: Create an Experiment¶
neptune.create_experiment(name='skopt sweep')
This also creates a link to the experiment. Open the link in a new tab. The charts will currently be empty, but keep the window open. You will be able to see live metrics once logging starts.
Step 3: Run skopt with the Neptune Callback¶
NeptuneCallback()
will log the run metrics, run parameters and results pickle after every iteration.
Everything can be inspected live.
# Create Neptune Callback
import neptunecontrib.monitoring.skopt as skopt_utils
neptune_callback = skopt_utils.NeptuneCallback()
# Run the skopt minimize function with the Neptune Callback
results = skopt.forest_minimize(objective,
space,
base_estimator='ET',
n_calls=100,
n_random_starts=10,
callback=[neptune_callback],)
Step 4: Log best parameter configuration, best score and diagnostic plots¶
You can log additional information from skopt results after the tuning has completed with the log_results()
function.
This will log:
Best score for the sweep as ‘best_score’ metric
Best parameter set as ‘best_parameters’ property
Fog figures from plots module: plot_evaluations, plot_convergence, plot_objective, and plot_regret to the ‘diagnostics’ log.
skopt_utils.log_results(results)
Note
You can change the Neptune experiment to which the results are logged with the experiment
parameter, and choose whether or not you want to log plots and the pickle objects with the log_plots
and log_pickle
parameters.
Step 5: See your Skopt tuning in Neptune¶
Now you can switch to the Neptune tab which you had opened previously to watch the tuning live!

More Options¶
Use Neptune with BayesSearchCV¶
Step 1: Initialize Neptune and create an experiment¶
import neptune
neptune.init(api_token='ANONYMOUS',
project_qualified_name='shared/scikit-optimize-integration')
neptune.create_experiment(name='skopt sweep')
Tip
You can also use your personal API token. Read more about how to securely set the Neptune API token.
Step 2: Initialize BayesSearchCV¶
opt = BayesSearchCV(
SVC(),
{
'C': Real(1e-6, 1e+6, prior='log-uniform'),
'gamma': Real(1e-6, 1e+1, prior='log-uniform'),
'degree': Integer(1,8),
'kernel': Categorical(['linear', 'poly', 'rbf']),
},
n_iter=32,
random_state=0
)
Step 2: Pass Neptune callback to the .fit
method¶
import neptunecontrib.monitoring.skopt as skopt_utils
opt.fit(X_train, y_train,
callback=skopt_utils.NeptuneCallback())
Step 3: Log best parameter set and diagnostic plots to Neptune¶
You can log best parameter, diagnostic plots and results pickle to Neptune with the log_results()
function.
To access the optimization results object you should use the ._optim_results
attribute of the BayesSearchCV
object.
skopt_utils.log_results(opt._optim_results[0])
Remember that you can try it out with zero setup:
How to ask for help?¶
Please visit the Getting help page. Everything regarding support is there.
Other pages you may like¶
You may also find the following pages useful: