Best practices#

By following our best practices, you ensure smooth and secure working with Neptune.

Configuring your credentials#

Best practice

Save your Neptune API token and project name as environment variables, rather than placing them in plain text in your model-training script.

While it is possible to pass your API token and project name as arguments to the init functions, it's both more secure and more convenient to save them as environment variables in your system.

This way, Neptune picks your credentials up automatically and you avoid storing them in the source code of your training scripts.

Setting your Neptune API token#

To save your API token, assign it to the NEPTUNE_API_TOKEN environment variable:

In the top-right corner of the app, click your avatar icon.
Select Get Your API token.
Depending on your system:
Linux macOS Windows Jupyter Notebook
From the API token dialog in Neptune, copy the export command and append the line to your .profile or other shell initialization file.
Example line
export NEPTUNE_API_TOKEN="uyVrZXkiOiIzNTd..."
From the API token dialog in Neptune, copy the export command and append the line to your .profile or other shell initialization file.
Example line
export NEPTUNE_API_TOKEN="uyVrZXkiOiIzNTd..."
1. From the API token dialog in Neptune, copy the setx command.
  
  Example line
  setx NEPTUNE_API_TOKEN "uyVrZXkiOiIzNTd..."
2. Open a terminal app, such as PowerShell or Command Prompt.
3. Paste the line you copied and press enter.
4. To activate the change, restart the terminal app.
You can use the os library to set the token as an environment variable.

Add the following to a notebook cell:
import os from getpass import getpass os.environ["NEPTUNE_API_TOKEN"] = getpass("Enter your Neptune API token: ")
From the API token dialog, copy your token, paste it in the input box, and hit Enter.

Note that any environment variables declared this way won't persist after the notebook kernel shuts down. If you start a new kernel, they need to be set again.

Setting your project name#

To save your project name, assign it to the NEPTUNE_PROJECT environment variable:

Navigate to Settings → Properties
Copy the full name of the project, or keep the page open to copy the name in a moment.
Next, assign the project name to an environment variable named NEPTUNE_PROJECT:
Linux macOS Windows Jupyter Notebook
Append the following line to your .profile (or other shell configuration file):
export NEPTUNE_PROJECT="workspace-name/project-name"
where workspace-name/project-name is the full project name you just copied.
Append the following line to your .profile (or other shell configuration file):
export NEPTUNE_PROJECT="workspace-name/project-name"
where workspace-name/project-name is the full project name you just copied.
To set the project name permanently:
1. Open a terminal app, such as PowerShell or Command Prompt.
2. Paste in the following command and press enter:
  
  setx NEPTUNE_PROJECT "workspace-name/project-name"
  
  where workspace-name/project-name is the full project name you just copied.
3. To activate the change, restart the terminal.
You can use the os library to set the project name as an environment variable:
import os os.environ["NEPTUNE_PROJECT"] = "workspace-name/project-name"
where workspace-name/project-name is the full project name you just copied.

Note that any environment variables declared this way won't persist after the notebook kernel shuts down. If you start a new kernel, they need to be set again.
To set the project only for the current session
Linux macOS Windows
On the command line, enter the following:

export NEPTUNE_PROJECT="workspace-name/project-name"
In Terminal, enter the following:

export NEPTUNE_PROJECT="workspace-name/project-name"
In a terminal app, such as PowerShell or Command Prompt, enter the following:

set NEPTUNE_PROJECT="workspace-name/project-name"

Stopping runs and other objects#

Best practice

When you're done logging to a Neptune object, stop it with the stop() method in your code.

When a script finishes executing, runs and other objects are stopped automatically.

However, to avoid logging metadata for longer than intended, it's best to stop the Neptune object explicitly with the stop() method. This is especially important for Run objects, which (by default¹) log system metrics in the background for as long as they're active.

Stopping a run

import neptune

run = neptune.init_run()

run["namespace/field"] = "some metadata"
...

run.stop()

Note for Jupyter notebooks

When working in an interactive notebook environment, you should always call stop().

This is because the connection to Neptune is not stopped when the cell has finished executing, but rather only when you stop the notebook.

Other cases for using stop() include (but aren't limited to):

Creating multiple runs in a single script
Continuous training flows
Model registry flows

While model and project objects are synchronous and do not monitor things in the background as runs do, it's also best to stop them when the connection to Neptune is not needed.

Maximum number of fields#

Best practice

Make sure you create no more than 9000 fields in a single Neptune object.

To ensure optimal performance, the number of user-defined fields is limited to 9000. This limit concerns any one Neptune object that can contain metadata fields, such as a Run or Model object.

Note

The field limit is not about metadata limits per se.

For example, rather than assigning 100 values to 100 fields (run["field1"] = 0, run["field2"] = 1, ..., ) you could construct a series of those 100 values and log them under a single field:

for i in range(100):
    run["field"].append(i)

You can log numerical values, strings, and files as a series. For details, see Field types overview.

If you exceed the limit while logging, Neptune stops the synchronization to the server and stores the data locally. To continue uploading the metadata, you need to delete some excess fields from the object in question, then sync the data manually.

For detailed instructions, see Troubleshooting ≫ Too many fields in object.

Reserved namespaces#

Best practice

Avoid creating fields in the sys or monitoring namespaces.

sys - The system namespace is reserved, that is, you cannot create new fields in this namespace.
- Note: You can manually edit several field values of the system namespace, such as the description, stage, and tags. For details, see System namespace overview.
monitoring - The monitoring namespace is intended specifically for system monitoring rather than model monitoring. Examples of system monitoring include hardware consumption and standard streams (stdout and stderr). For more, see Logging system metrics.

See also: What Neptune logs automatically ↩