ZenML integration guide#
Related
ZenML on GitHub
ZenML is a technology-agnostic, open source pipelines framework. With the Neptune-ZenML integration, you can log and visualize information from your ZenML pipeline steps (such as models, parameters, metrics).
Neptune setup#
Installing Neptune#
Install the Neptune client library:
Installing through Anaconda Navigator
To find neptune, you may need to update your channels and index.
- In the Navigator, select Environments.
- In the package view, click Channels.
- Click Add..., enter
conda-forge
, and click Update channels. - In the package view, click Update index... and wait until the update is complete. This can take several minutes.
- You should now be able to search for neptune.
Note: The displayed version may be outdated. The latest version of the package will be installed.
Note: On Bioconda, there is a "neptune" package available which is not the neptune.ai client library. Make sure to specify the "conda-forge" channel when installing neptune.ai.
Storing your Neptune credentials#
As a best practice, you should save your Neptune credentials as environment variables.
Setting your API token#
- In the bottom-left corner of the app, expand your user menu.
-
Select Get Your API token.
-
Depending on your system:
From the API token dialog in Neptune, copy the
export
command and append the line to your.profile
or other shell initialization file.From the API token dialog in Neptune, copy the
export
command and append the line to your.profile
or other shell initialization file.-
From the API token dialog in Neptune, copy the
setx
command. -
Open a terminal app, such as PowerShell or Command Prompt.
- Paste the line you copied and press enter.
- To activate the change, restart the terminal app.
You can use the os library to set the token as an environment variable.
Add the following to a notebook cell:
import os from getpass import getpass os.environ["NEPTUNE_API_TOKEN"] = getpass("Enter your Neptune API token: ")
From the API token dialog, copy your token, paste it in the input box, and hit Enter.
Note that any environment variables declared this way won't persist after the notebook kernel shuts down. If you start a new kernel, they need to be set again.
-
Setting your project name#
- In the top-right corner, click the settings menu ().
-
Select Edit project details.
-
Find the copy button () next to the project name.
-
Assign the project name to an environment variable named
NEPTUNE_PROJECT
:Append the following line to your
.profile
(or other shell configuration file):where
workspace-name/project-name
is the full project name you just copied.Append the following line to your
.profile
(or other shell configuration file):where
workspace-name/project-name
is the full project name you just copied.To set the project name permanently:
- Open a terminal app, such as PowerShell or Command Prompt.
-
Paste in the following command and press enter:
where
workspace-name/project-name
is the full project name you just copied. -
To activate the change, restart the terminal.
You can use the os library to set the project name as an environment variable:
where
workspace-name/project-name
is the full project name you just copied.Note that any environment variables declared this way won't persist after the notebook kernel shuts down. If you start a new kernel, they need to be set again.
More help with getting started
Registering the Neptune Experiment Tracker#
When registering neptune
as the Experiment Tracker flavor, pass the API token and project name as arguments to the options of the experiment-tracker register
command:
zenml experiment-tracker register neptune_experiment_tracker --flavor=neptune \
--project="$NEPTUNE_PROJECT" --api_token="$NEPTUNE_API_TOKEN"
zenml stack register neptune_stack \
-a default \
-o default \
-e neptune_experiment_tracker \
--set
Using the Neptune-ZenML integration#
For detailed configuration and examples, see the ZenML resources:
- Experiment Trackers ≫ Neptune in the ZenML documentation
- Neptune-ZenML example on GitHub
Manually logging metadata to the run object#
You can access the Neptune Run
object with the get_neptune_run()
function. Use the object to track metadata from your training pipeline in a structure of your choosing.
from zenml.integrations.neptune.experiment_trackers.run_state import (
get_neptune_run
)
@step(experiment_tracker="",)
def my_step() ...:
neptune_run = get_neptune_run()
neptune_run["sys/name"] = "My custom run name" # (1)!
neptune_run["params/lr"] = params.lr # (2)!
...
- Set a custom name for the run by assigning it to the
name
field of the system namespace. For more, see System namespace overview: Name - Assign any value to a path of your choice. This would store the learning rate under a field called
lr
inside a namespace calledparams
.
For more logging help and tips, see: