Skip to content

Best practices#

By following our best practices, you ensure smooth and secure working with Neptune.

Configuring your credentials#

Best practice

Save your Neptune API token and project name as environment variables, rather than placing them in plain text in your model-training script.

While it is possible to pass your API token and project name as arguments to the init functions, it's both more secure and more convenient to save them as environment variables in your system.

This way, Neptune picks your credentials up automatically and you avoid storing them in the source code of your training scripts.

Setting your Neptune API token#

To save your API token, assign it to the NEPTUNE_API_TOKEN environment variable:

  1. In the bottom-left corner of the app, expand your user menu.

  2. Select Get Your API token.

    How to find your Neptune API token

  3. Depending on your system:

    From the API token dialog in Neptune, copy the export command and append the line to your .profile or other shell initialization file.

    Example line
    export NEPTUNE_API_TOKEN="uyVrZXkiOiIzNTd..."

    From the API token dialog in Neptune, copy the export command and append the line to your .profile or other shell initialization file.

    Example line
    export NEPTUNE_API_TOKEN="uyVrZXkiOiIzNTd..."

    1. From the API token dialog in Neptune, copy the setx command.

      Example line
      setx NEPTUNE_API_TOKEN "uyVrZXkiOiIzNTd..."

    2. Open a terminal app, such as PowerShell or Command Prompt.

    3. Paste the line you copied and press enter.
    4. To activate the change, restart the terminal app.

    You can use the os library to set the token as an environment variable.

    Add the following to a notebook cell:

    import os
from getpass import getpass
os.environ["NEPTUNE_API_TOKEN"] = getpass("Enter your Neptune API token: ")

    From the API token dialog, copy your token, paste it in the input box, and hit Enter.

    Note that any environment variables declared this way won't persist after the notebook kernel shuts down. If you start a new kernel, they need to be set again.

Setting your project name#

To save your project name, assign it to the NEPTUNE_PROJECT environment variable:

  1. In the top-right corner, click the settings menu ().

  2. Select Edit project details.

    How to access project details

  3. Find the copy button () next to the project name.

  4. Assign the project name to an environment variable named NEPTUNE_PROJECT:

    Append the following line to your .profile (or other shell configuration file):

    export NEPTUNE_PROJECT="workspace-name/project-name"

    where workspace-name/project-name is the full project name you just copied.

    Append the following line to your .profile (or other shell configuration file):

    export NEPTUNE_PROJECT="workspace-name/project-name"

    where workspace-name/project-name is the full project name you just copied.

    To set the project name permanently:

    1. Open a terminal app, such as PowerShell or Command Prompt.

    2. Paste in the following command and press enter:

      setx NEPTUNE_PROJECT "workspace-name/project-name"

      where workspace-name/project-name is the full project name you just copied.

    3. To activate the change, restart the terminal.

    You can use the os library to set the project name as an environment variable:

    import os
os.environ["NEPTUNE_PROJECT"] = "workspace-name/project-name"

    where workspace-name/project-name is the full project name you just copied.

    Note that any environment variables declared this way won't persist after the notebook kernel shuts down. If you start a new kernel, they need to be set again.

Stopping runs and other objects#

Best practice

When you're done logging to a Neptune object, stop it with the stop() method in your code.

When a script finishes executing, runs and other Neptune objects are stopped automatically. However, in interactive sessions (such as Jupyter notebooks) the connection to the object is stopped only when the kernel stops.

To avoid logging metadata for longer than intended, it's best to stop each Neptune object explicitly with the stop() method. This is especially important for Run objects, which by default1 log system metrics in the background for as long as they're active.

Stopping a run
import neptune

run = neptune.init_run()

run["namespace/field"] = "some metadata"
...
run.stop()

Other cases for using stop() include (but aren't limited to):

  • Creating multiple runs in a single script
  • Continuous training flows
  • Model registry flows

While model and project objects are synchronous and do not monitor things in the background as runs do, it's also best to stop them when the connection to Neptune is not needed.

Enabling background monitoring in interactive sessions

To mitigate undesired background logging, in interactive sessions, some monitoring options are disabled by default.

You can turn them on with:

import neptune

run = neptune.init_run(
    capture_hardware_metrics=True,
    capture_stderr=True,
    capture_stdout=True,
)

Maximum number of fields#

Best practice

Make sure you create no more than 9000 fields in a single Neptune object.

To ensure optimal performance, the number of user-defined fields is limited to 9000. This limit concerns any one Neptune object that can contain metadata fields, such as a Run or Model object.

Note

The field limit is not about metadata limits per se.

For example, rather than assigning 100 values to 100 fields (run["field1"] = 0, run["field2"] = 1, ..., ) you could construct a series of those 100 values and log them under a single field:

for i in range(100):
    run["field"].append(i)

You can log numerical values, strings, and files as a series. For details, see Field types overview.

If you exceed the field limit while logging, Neptune stops the synchronization to the server and stores the data locally. To continue uploading the metadata, you need to delete some excess fields from the object in question, then sync the data manually.

For detailed instructions, see Troubleshooting ≫ Too many fields in object.

Reserved namespaces#

Best practice

Avoid creating custom fields in the sys or monitoring namespaces.

  • sys - The system namespace is reserved, that is, you cannot create new fields in this namespace.
    • Note: You can manually edit several field values of the system namespace, such as the description, stage, and tags. For details, see System namespace overview.
  • monitoring - The monitoring namespace is intended specifically for system monitoring rather than model monitoring. Examples of system monitoring include hardware consumption and standard streams (stdout and stderr). For more, see Log system metrics.

  1. See also: What Neptune logs automatically ↩