Skip to content

ZenML integration guide#

Related

ZenML on GitHub

ZenML is a technology-agnostic, open source pipelines framework. With the Neptune-ZenML integration, you can log and visualize information from your ZenML pipeline steps (such as models, parameters, metrics).

Neptune setup#

Installing Neptune#

Install the Neptune client library:

pip install neptune

conda install -c conda-forge neptune
Installing through Anaconda Navigator

To find neptune, you may need to update your channels and index.

  1. In the Navigator, select Environments.
  2. In the package view, click Channels.
  3. Click Add..., enter conda-forge, and click Update channels.
  4. In the package view, click Update index... and wait until the update is complete. This can take several minutes.
  5. You should now be able to search for neptune.

Note: The displayed version may be outdated. The latest version of the package will be installed.

Note: On Bioconda, there is a "neptune" package available which is not the neptune.ai client library. Make sure to specify the "conda-forge" channel when installing neptune.ai.

Storing your Neptune credentials#

As a best practice, you should save your Neptune credentials as environment variables.

Setting your API token#

  1. In the bottom-left corner of the app, expand your user menu.

  2. Select Get Your API token.

    How to find your Neptune API token

  3. Depending on your system:

    From the API token dialog in Neptune, copy the export command and append the line to your .profile or other shell initialization file.

    Example line
    export NEPTUNE_API_TOKEN="uyVrZXkiOiIzNTd..."

    From the API token dialog in Neptune, copy the export command and append the line to your .profile or other shell initialization file.

    Example line
    export NEPTUNE_API_TOKEN="uyVrZXkiOiIzNTd..."

    1. From the API token dialog in Neptune, copy the setx command.

      Example line
      setx NEPTUNE_API_TOKEN "uyVrZXkiOiIzNTd..."

    2. Open a terminal app, such as PowerShell or Command Prompt.

    3. Paste the line you copied and press enter.
    4. To activate the change, restart the terminal app.

    You can use the os library to set the token as an environment variable.

    Add the following to a notebook cell:

    import os
from getpass import getpass
os.environ["NEPTUNE_API_TOKEN"] = getpass("Enter your Neptune API token: ")

    From the API token dialog, copy your token, paste it in the input box, and hit Enter.

    Note that any environment variables declared this way won't persist after the notebook kernel shuts down. If you start a new kernel, they need to be set again.

Setting your project name#

  1. In the top-right corner, click the settings menu ().

  2. Select Edit project details.

    How to access project details

  3. Find the copy button () next to the project name.

  4. Assign the project name to an environment variable named NEPTUNE_PROJECT:

    Append the following line to your .profile (or other shell configuration file):

    export NEPTUNE_PROJECT="workspace-name/project-name"

    where workspace-name/project-name is the full project name you just copied.

    Append the following line to your .profile (or other shell configuration file):

    export NEPTUNE_PROJECT="workspace-name/project-name"

    where workspace-name/project-name is the full project name you just copied.

    To set the project name permanently:

    1. Open a terminal app, such as PowerShell or Command Prompt.

    2. Paste in the following command and press enter:

      setx NEPTUNE_PROJECT "workspace-name/project-name"

      where workspace-name/project-name is the full project name you just copied.

    3. To activate the change, restart the terminal.

    You can use the os library to set the project name as an environment variable:

    import os
os.environ["NEPTUNE_PROJECT"] = "workspace-name/project-name"

    where workspace-name/project-name is the full project name you just copied.

    Note that any environment variables declared this way won't persist after the notebook kernel shuts down. If you start a new kernel, they need to be set again.

More help with getting started

Registering the Neptune Experiment Tracker#

When registering neptune as the Experiment Tracker flavor, pass the API token and project name as arguments to the options of the experiment-tracker register command:

zenml experiment-tracker register neptune_experiment_tracker --flavor=neptune \
    --project="$NEPTUNE_PROJECT" --api_token="$NEPTUNE_API_TOKEN"

zenml stack register neptune_stack \
    -a default \
    -o default \
    -e neptune_experiment_tracker \
    --set

Using the Neptune-ZenML integration#

For detailed configuration and examples, see the ZenML resources:

Manually logging metadata to the run object#

You can access the Neptune Run object with the get_neptune_run() function. Use the object to track metadata from your training pipeline in a structure of your choosing.

from zenml.integrations.neptune.experiment_trackers.run_state import (
    get_neptune_run
)

@step(experiment_tracker="",)

def my_step() ...:
    neptune_run = get_neptune_run()
    neptune_run["sys/name"] = "My custom run name"  # (1)!
    neptune_run["params/lr"] = params.lr  # (2)!
    ...
  1. Set a custom name for the run by assigning it to the name field of the system namespace. For more, see System namespace overview: Name
  2. Assign any value to a path of your choice. This would store the learning rate under a field called lr inside a namespace called params.

For more logging help and tips, see: