Skip to content

Best practices#

By following our best practices, you ensure smooth and secure working with Neptune.

Configuring your credentials#

Best practice

Save your Neptune API token and project name as environment variables, rather than placing them in plain text in your model-training script.

While it is possible to pass your API token and project name as arguments to the init methods, it's both more secure and more convenient to save them as environment variables in your system.

This way, Neptune picks your credentials up automatically and you avoid storing them in the source code of your training scripts.

Setting your Neptune API token#

To save your API token, assign it to the NEPTUNE_API_TOKEN environment variable:

  1. In the top-right corner of the app, click your avatar icon.
  2. Select Get Your API token.

    How to find your Neptune API token

  3. Depending on your system:

    1. From the API token dialog in Neptune, copy the export command.

      Example:

      export NEPTUNE_API_TOKEN="uyVrZXkiOiIzNTd..."
      
    2. Append the line to your .profile or other shell configuration file.

    1. From the API token dialog in Neptune, copy the export command.

      Example:

      export NEPTUNE_API_TOKEN="uyVrZXkiOiIzNTd..."
      
    2. Append the line to your .profile or other shell configuration file.

    1. From the API token dialog in Neptune, copy the setx command.

      Example:

      setx NEPTUNE_API_TOKEN "uyVrZXkiOiIzNTd..."
      
    2. Open a terminal app, such as PowerShell or Command Prompt.

    3. Paste the line you copied and press enter.
    4. To activate the change, restart the terminal app.
    from getpass import getpass
    
    run = neptune.init_run(
        project = ...
        api_token = getpass("Enter your Neptune API token: "),
    )
    

Setting your project name#

To save your project name, assign it to the NEPTUNE_PROJECT environment variable:

  1. Navigate to Settings → Properties
  2. Copy the full name of the project, or keep the page open to copy the name in a moment.

    Where to copy your project name

  3. Next, assign the project name to an environment variable named NEPTUNE_PROJECT:

Append the following line to your .profile (or other shell configuration file):

export NEPTUNE_PROJECT="workspace-name/project-name"

where workspace-name/project-name is the full project name you just copied.

Append the following line to your .profile (or other shell configuration file):

export NEPTUNE_PROJECT="workspace-name/project-name"

where workspace-name/project-name is the full project name you just copied.

To set the project name permanently:

  1. Open a terminal app, such as PowerShell or Command Prompt.
  2. Paste in the following command and press enter:

    setx NEPTUNE_PRPOJECT "workspace-name/project-name"
    

    where workspace-name/project-name is the full project name you just copied.

  3. To activate the change, restart the terminal.

To set the project only for the current session

On the command line, enter the following:

export NEPTUNE_PROJECT="workspace-name/project-name"

In Terminal, enter the following:

export NEPTUNE_PROJECT="workspace-name/project-name"

In a terminal app, such as PowerShell or Command Prompt, enter the following:

set NEPTUNE_PRPOJECT="workspace-name/project-name"

Stopping runs and other objects#

Best practice

When you're done logging to a Neptune object, stop it with the stop() method in your code.

When a script finishes executing, runs and other objects are stopped automatically.

However, to avoid logging metadata for longer than intended, it's best to stop the Neptune object explicitly with the stop() method. This is especially important for Run objects, which (by default1) log system metrics in the background for as long as they're active.

Stopping a run
import neptune.new as neptune

run = neptune.init_run()

run["namespace/field"] = "some metadata"
...

run.stop()
Note for Jupyter notebooks

When working in an interactive notebook environment, you should always call stop().

This is because the connection to Neptune is not stopped when the cell has finished executing, but rather only when you stop the notebook.

Other cases for using stop() include (but aren't limited to):

  • Creating multiple runs in a single script
  • Continuous training flows
  • Model registry flows

While model and project objects are synchronous and do not monitor things in the background as runs do, it's also best to stop them when the connection to Neptune is not needed.

Maximum number of fields#

Best practice

Make sure you create no more than 9000 fields in a single Neptune object.

To ensure optimal performance, the number of user-defined fields is limited to 9000. This limit concerns any one Neptune object that can contain metadata fields, such as a Run or Model object.

Note

The field limit is not about metadata limits per se.

For example, rather than assigning 100 values to 100 fields (run["field1"] = 0, run["field2"] = 1, ..., ) you could construct a series of those 100 values and log them under a single field:

for i in range(100):
    run["field"].log(i)

You can log numerical values, strings, and files as a series. For details, see Field types overview.

If you exceed the limit while logging, Neptune stops the synchronization to the server and stores the data locally. To continue uploading the metadata, you need to delete some excess fields from the object in question, then sync the data manually.

For detailed instructions, see TroubleshootingToo many fields in object.

Reserved namespaces#

Best practice

Avoid creating fields in the sys or monitoring namespaces.

  • sys - The system namespace is reserved, that is, you cannot create new fields in this namespace.
    • Note: You can manually edit several field values of the system namespace, such as the description, stage, and tags. For details, see System namespace overview.
  • monitoring - The monitoring namespace is intended specifically for system monitoring rather than model monitoring. Examples of system monitoring include hardware consumption and standard streams (stdout and stderr). For more, see Logging system metrics.