Best practices#

By following our best practices, you ensure smooth and secure working with Neptune.

Configuring credentials#

Best practice

Save your Neptune API token and project name as environment variables, rather than placing them in plain text in your model training script.

While it is possible to pass your API token and project name as arguments to the init functions, it's both more secure and more convenient to save them as environment variables in your system.

This way, Neptune picks your credentials up automatically and you avoid storing them in the source code of your training scripts.

Setting Neptune API token#

To save your API token, assign it to the NEPTUNE_API_TOKEN environment variable:

In the bottom-left corner of the app, expand your user menu.
Select Get Your API token.
Depending on your system:
Linux macOS Windows Jupyter Notebook
From the API token dialog in Neptune, copy the export command and append the line to your .profile or other shell initialization file.
Example line
export NEPTUNE_API_TOKEN="uyVrZXkiOiIzNTd..."
From the API token dialog in Neptune, copy the export command and append the line to your .profile or other shell initialization file.
Example line
export NEPTUNE_API_TOKEN="uyVrZXkiOiIzNTd..."
1. From the API token dialog in Neptune, copy the setx command.
  
  Example line
  setx NEPTUNE_API_TOKEN "uyVrZXkiOiIzNTd..."
2. Open a terminal app, such as PowerShell or Command Prompt.
3. Paste the line you copied and press enter.
4. To activate the change, restart the terminal app.
You can use the os library to set the token as an environment variable.

Add the following to a notebook cell:
import os from getpass import getpass os.environ["NEPTUNE_API_TOKEN"] = getpass("Enter your Neptune API token: ")
From the API token dialog, copy your token, paste it in the input box, and hit Enter.

Note that any environment variables declared this way won't persist after the notebook kernel shuts down. If you start a new kernel, they need to be set again.

Setting project name#

To save your project name, assign it to the NEPTUNE_PROJECT environment variable:

In the top-right corner, click the settings menu ().
Select Edit project details.
Find the copy button () next to the project name.
Assign the project name to an environment variable named NEPTUNE_PROJECT:
Linux macOS Windows Jupyter Notebook
Append the following line to your .profile (or other shell configuration file):
export NEPTUNE_PROJECT="workspace-name/project-name"
where workspace-name/project-name is the full project name you just copied.
Append the following line to your .profile (or other shell configuration file):
export NEPTUNE_PROJECT="workspace-name/project-name"
where workspace-name/project-name is the full project name you just copied.
To set the project name permanently:
1. Open a terminal app, such as PowerShell or Command Prompt.
2. Paste in the following command and press enter:
  
  setx NEPTUNE_PROJECT "workspace-name/project-name"
  
  where workspace-name/project-name is the full project name you just copied.
3. To activate the change, restart the terminal.
You can use the os library to set the project name as an environment variable:
import os os.environ["NEPTUNE_PROJECT"] = "workspace-name/project-name"
where workspace-name/project-name is the full project name you just copied.

Note that any environment variables declared this way won't persist after the notebook kernel shuts down. If you start a new kernel, they need to be set again.

Stopping runs and other objects#

Best practice

When you're done logging to a Neptune object, stop it with the stop() method in your code.

When a script finishes executing, runs and other Neptune objects are stopped automatically. However, in interactive sessions (such as Jupyter notebooks) the connection to the object is stopped only when the kernel stops.

To avoid logging metadata for longer than intended, it's best to stop each Neptune object explicitly with the stop() method. This is especially important for Run objects, which by default¹ log system metrics in the background for as long as they're active.

Stopping a run

import neptune

run = neptune.init_run()

run["namespace/field"] = "some metadata"
...
run.stop()

Other cases for using stop() include (but aren't limited to):

Creating multiple runs in a single script
Continuous training flows
Model registry flows

While model and project objects are synchronous and do not monitor things in the background as runs do, it's also best to stop them when the connection to Neptune is not needed.

Enabling background monitoring in interactive sessions

To mitigate undesired background logging, in interactive sessions, some monitoring options are disabled by default.

You can turn them on with:

import neptune

run = neptune.init_run(
    capture_hardware_metrics=True,
    capture_stderr=True,
    capture_stdout=True,
)

Maximum number of fields#

Best practice

Make sure you create no more than 9000 fields in a single Neptune object.

To ensure optimal performance, the number of user-defined fields is limited to 9000. This limit concerns any one Neptune object that can contain metadata fields, such as a Run or Model object.

Note

The field limit is not about metadata limits per se.

For example, rather than assigning 100 values to 100 fields (run["field1"] = 0, run["field2"] = 1, ..., ) you could construct a series of those 100 values and log them under a single field:

for i in range(100):
    run["field"].append(i)

You can log numerical values, strings, and files as a series. For details, see Field types overview.

If you exceed the field limit while logging, Neptune stops the synchronization to the server and stores the data locally. To continue uploading the metadata, you need to delete some excess fields from the object in question, then sync the data manually.

For detailed instructions, see API ≫ Errors and warnings ≫ Field count limit exceeded.

Reserved namespaces#

Best practice

Avoid creating custom fields in the sys or monitoring namespaces.

sys: The system namespace is reserved, that is, you cannot create new fields in this namespace.

Note: You can manually edit several field values of the system namespace, such as the description, stage, and tags. For details, see System namespace overview.
monitoring: The monitoring namespace is intended specifically for system monitoring rather than model monitoring. Examples of system monitoring include hardware consumption and standard streams (stdout and stderr). For more, see Logging system metrics.

Pipelining libraries#

Neptune provides tracking and visualization for pipelining libraries such as KubeFlow. You generally just need to ensure that all the steps of the pipeline (in practice, scripts) are tracking data to the same Neptune run.

To access the same run object in multiple scripts, you have a few options:

Set a custom run ID
- You can create a custom identifier for the run and use that to access the same run from multiple locations.
- You can also export the custom run ID as an environment variable (NEPTUNE_CUSTOM_RUN_ID). This tells Neptune that scripts started with the same NEPTUNE_CUSTOM_RUN_ID value should be treated as one and the same run.
Pass the run object between files
- You can use the Run object as a parameter in functions you import from other scripts.
- This method applies to other Neptune objects as well: Model, ModelVersion, and Project.

On top of that, you might want to use namespaces to organize tracked metadata into meaningful steps of your pipeline.

For example:

# in the preparation script
run["preparation/input_dataset"].upload("test.csv")

# in the training script
run["training/accuracy"].append(0.96)

# in the validation script
run["validation/accuracy"].append(0.89)

Setting a base namespace
Tutorial ≫ Logging with Neptune in a sequential pipeline
Namespaces and fields

Parallel computing#

The Neptune client library is thread-safe. This means that you can track data to your run from multiple threads within one Python process.

Within one Python script, calls that log metadata from a run are executed in "First In, First Out" (FIFO) order. However, to avoid potential race condition issues, we advise against modifying a variable from multiple threads.

Info

You can track metadata to several runs at the same time – for example, from a Jupyter notebook.

Each run has a separate FIFO queue and a separate synchronization thread.

Distributed computing#

You can track run metadata from several processes, running on the same or different machines.

Neptune is fully compatible with distributed computing frameworks, such as Apache Spark. Neptune also provides some synchronization methods that will help you handle more sophisticated workflows:

wait() – wait for all the metadata tracking calls to finish before executing the call. See below for an example.
sync() – synchronize the local representation of the run or other object with the Neptune servers.

Neptune is optimized for rapid metadata tracking and does not perform API calls to Neptune servers if not needed. As such, Neptune stores its own local representation of the run structure and assumes that no other process is modifying the run at the same time in a conflicting way.

Examples of such conflicting cases would be:

Using the same field name with a different field type, such as File versus StringSeries.
Removing or renaming variables or fields that are being used by another process.

In the case of a conflict, tracking methods will throw an exception and some of the tracked data may not be stored on Neptune servers.

In particular, you should respect the type of a field even within a simple script:

run["learning_rate"] = 0.5
run["learning_rate"] = "let's try a string instead"
# Error: Float variable cannot be assigned with a str.

run["accuracy"].append(0.95)
run["accuracy"] = 0.99
# Error: FloatSeries does not support the "=" operator. Use append() instead.

Like in parallel computing, to avoid potential race condition issues, we advise against modifying a variable from multiple processes.

Fetching metadata logged within the same script#

When working in asynchronous mode (default), the metadata you track is periodically synchronized with the servers in the background. Because of this, the data may not be immediately available to fetch from the server, even if it appears in the local representation.

To work around that, you can use the wait() method.

Rule of thumb

If you fetch metadata that you logged from within the same script, use wait() on the object before querying:

run["learning_rate"] = 0.5

# Wait for the data to be synchronized
run.wait()

# Fetching the metadata we just logged will now work
learning_rate = run["learning_rate"].fetch()

Log from multiple scripts
Setting a base namespace
Tutorial ≫ Tracking distributed training jobs

See also: What Neptune logs automatically ↩