Tracking hyperparameter optimization jobs with Neptune#

When optimizing or tuning hyperparameters, you can use Neptune to track the metadata from the study as well as each trial.

In this guide, we'll show you how to configure Neptune for your HPO job in two ways:

By logging the metadata from all trials to the same Neptune run
By creating a separate Neptune run for each trial

Integration tip

Neptune integrates directly with Optuna, a hyperparameter optimization framework.

For a detailed guide, see Optuna integration guide.

See example in Neptune Code examples

Before you start#

Sign up at neptune.ai/register.
Create a project for storing your metadata.
Install Neptune:
```
pip install neptune
```
Passing your Neptune credentials

Once you've registered and created a project, set your Neptune API token and full project name to the NEPTUNE_API_TOKEN and NEPTUNE_PROJECT environment variables, respectively.
```
export NEPTUNE_API_TOKEN="h0dHBzOi8aHR0cHM.4kl0jvYh3Kb8...6Lc"
```
To find your API token: In the bottom-left corner of the Neptune app, expand the user menu and select Get my API token.
```
export NEPTUNE_PROJECT="ml-team/classification"
```
Your full project name has the form workspace-name/project-name. You can copy it from the project settings: Click the menu in the top-right → Details & privacy.

On Windows, navigate to Settings → Edit the system environment variables, or enter the following in Command Prompt: setx SOME_NEPTUNE_VARIABLE 'some-value'

While it's not recommended especially for the API token, you can also pass your credentials in the code when initializing Neptune.
```
run = neptune.init_run(
    project="ml-team/classification",  # your full project name here
    api_token="h0dHBzOi8aHR0cHM6Lkc78ghs74kl0jvYh...3Kb8",  # your API token here
)
```
For more help, see Set Neptune credentials.

Setting up the training script#

We'll use a skeleton code to demonstrate how to set up a training script for hyperparameter optimization with Neptune. For an end-to-end working example, see the code examples.

Import the needed libraries:

import neptune
...

from neptune.utils import stringify_unsupported
...

Define hyperparameters and the search space:
```
parameters = {...}

PARAM_TO_OPTIMIZE = [...]  # (1)!
```
1. Search space for the parameters to optimize. For example, lr = [0.001, 0.01, 0.1]
Initialize your model and data.

Next, set up the training loop depending on your approach:

Log all trials to the same run
Log each trial to a separate run

Logging all trials to the same run#

In this approach, we'll create a global Neptune run for logging metadata across the trials.

Initialize a Neptune run:
```
run = neptune.init_run()
```
If you haven't saved your credentials as environment variables, you can pass them as arguments when initializing Neptune:
```
neptune.init_run(
    project="workspace-name/project-name",
    api_token="YourNeptuneApiTokenHere",
)
```
How do I save my credentials as environment variables?

Set your Neptune API token and full project name to the NEPTUNE_API_TOKEN and NEPTUNE_PROJECT environment variables, respectively.
Linux macOS Windows Jupyter Notebook
export NEPTUNE_API_TOKEN="h0dHBzOi8aHR0cHM.4kl0jvYh3Kb8...ifQ=="

export NEPTUNE_PROJECT="ml-team/classification"
export NEPTUNE_API_TOKEN="h0dHBzOi8aHR0cHM.4kl0jvYh3Kb8...ifQ=="

export NEPTUNE_PROJECT="ml-team/classification"
setx NEPTUNE_API_TOKEN "h0dHBzOi8aHR0cHM.4kl0jvYh3Kb8...ifQ=="

setx NEPTUNE_PROJECT "ml-team/classification"

You can also navigate to Settings → Edit the system environment variables and add the variables there.
%env NEPTUNE_API_TOKEN="h0dHBzOi8aHR0cHM.4kl0jvYh3Kb8...ifQ=="

%env NEPTUNE_PROJECT="ml-team/classification"
To find your credentials:
- API token: In the bottom-left corner of the Neptune app, expand your user menu and select Get your API token. If you need the token of a service account, go to the workspace or project settings and enter the Service accounts settings.
- Project name: Your full project name has the form workspace-name/project-name. You can copy it from the project menu ( → Details & privacy).
If you're working in Google Colab, you can set your credentials with the os and getpass libraries:
```
import os
from getpass import getpass
os.environ["NEPTUNE_API_TOKEN"] = getpass("Enter your Neptune API token: ")
os.environ["NEPTUNE_PROJECT"] = "workspace-name/project-name"
```
Haven't registered yet?

You can also try Neptune anonymously by passing the following credentials:
```
run = neptune.init_run(
    api_token=neptune.ANONYMOUS_API_TOKEN,
    project="common/pytorch-integration",
    tags=["sweep-level"],
)
```

Set up the training loop:

for (i, param) in enumerate(PARAM_TO_OPTIMIZE):

    # Log hyperparameters
    run[f"trials/{i}/params"] = stringify_unsupported(parameters)
    run[f"trials/{i}/params/<PARAM_TO_OPTIMIZE>"] = param

    # Initialize fields for best values across all trials
    best_value = None

    # Training loop
    for _ in range(parameters["epochs"]):
        ...

        metric = ...

        # Log metrics
        run[f"trials/{i}/metrics/<METRIC>"].append(metric)
        ...

        # Log best value and parameter so far
        if best_value is None or metric > best_value:
            run["best/trial"] = i
            run["best/metrics/<METRIC>"] = best_value = value
            run["best/params"] = stringify_unsupported(parameters)
            run["best/params/<PARAM_TO_OPTIMIZE>"] = param

To stop the connection to Neptune and sync all data, call the stop() method:
```
run.stop()
```

Analyzing results in Neptune#

When browsing the metadata of the sweep level run, you can see a namespace called trials. It contains the metadata (params and metrics) logged for each trial. The best trial, along with its parameter and metric values, is stored in the best namespace.

To organize all relevant metadata in one view, create a custom dashboard.

See example in Neptune

To view the best trials across different runs, create saved table views.

See example in Neptune

Logging each trial to a separate run#

You can also log metadata from each trial into separate runs. This way, you can track metadata from each trial separately.

Aggregated values can be logged to a parent sweep-level run. Sweep-level identifiers can be used to group all trials from the same sweep.

To log each trial to a separate run:

Set up the training script.

Create a sweep ID:

import uuid

sweep_id = str(uuid.uuid4())

Initialize a sweep-level run:

sweep_run = neptune.init_run(tags=["sweep-level"])

Add the sweep ID as a group tag to the sweep-level run:
```
sweep_run["sys/group_tags"].add(sweep_id)
```

Set up the training loop:

for i, param in enumerate(PARAM_TO_OPTIMIZE):

# Create a trial-level run
with neptune.init_run(
    name=f"trial-{i}",
    tags=["trial-level",],
) as trial_run:

    # Add sweep_id to the trial-level run
    trial_run["sys/group_tags"].add(sweep_id)

    # Log hyperparameters
    trial_run["params"] = stringify_unsupported(parameters)
    trial_run["params/<PARAM_TO_OPTIMIZE>"] = param

    # Initialize fields for best values across all trials
    best_value = None

    # Training loop
    for _ in range(parameters["epochs"]):
        ...

        metric = ...

        # Log trial metrics
        trial_run["metrics/<METRIC>"].append(metric)
        ...

        # Log best values across all trials to sweep-level run
        if best_loss is None or metric < best_value:
            sweep_run["best/trial"] = i
            sweep_run["best/metrics/<METRIC>"] = best_value = metric
            sweep_run["best/params"] = stringify_unsupported(parameters)
            sweep_run["best/params/<PARAM_TO_OPTIMIZE>"] = param

5. Stop the sweep-level run:

sweep_run.stop()

Analyzing results in Neptune#

Follow the link to the runs and explore the logged metadata (such as metrics and hyperparameters) in Neptune:

The best trial, with its metrics and parameters, is available in the best namespace of the sweep-level run.
Metadata (metrics and parameters) across all trials are available in the trial-level runs.

To group all trials under a sweep together, use run groups.

See example in Neptune

API reference ≫ append()
What you can log and display
Integrations ≫ Optuna integration guide