Best practices#
By following our best practices, you ensure smooth and secure working with Neptune.
Configuring your credentials#
Best practice
Save your Neptune API token and project name as environment variables, rather than placing them in plain text in your model-training script.
While it is possible to pass your API token and project name as arguments to the init
functions, it's both more secure and more convenient to save them as environment variables in your system.
This way, Neptune picks your credentials up automatically and you avoid storing them in the source code of your training scripts.
Setting your Neptune API token#
To save your API token, assign it to the NEPTUNE_API_TOKEN
environment variable:
- In the bottom-left corner of the app, expand your user menu.
-
Select Get Your API token.
-
Depending on your system:
From the API token dialog in Neptune, copy the
export
command and append the line to your.profile
or other shell initialization file.From the API token dialog in Neptune, copy the
export
command and append the line to your.profile
or other shell initialization file.-
From the API token dialog in Neptune, copy the
setx
command. -
Open a terminal app, such as PowerShell or Command Prompt.
- Paste the line you copied and press enter.
- To activate the change, restart the terminal app.
You can use the os library to set the token as an environment variable.
Add the following to a notebook cell:
import os from getpass import getpass os.environ["NEPTUNE_API_TOKEN"] = getpass("Enter your Neptune API token: ")
From the API token dialog, copy your token, paste it in the input box, and hit Enter.
Note that any environment variables declared this way won't persist after the notebook kernel shuts down. If you start a new kernel, they need to be set again.
-
Setting your project name#
To save your project name, assign it to the NEPTUNE_PROJECT
environment variable:
- In the top-right corner, click the settings menu ().
-
Select Edit project details.
-
Find the copy button () next to the project name.
-
Assign the project name to an environment variable named
NEPTUNE_PROJECT
:Append the following line to your
.profile
(or other shell configuration file):where
workspace-name/project-name
is the full project name you just copied.Append the following line to your
.profile
(or other shell configuration file):where
workspace-name/project-name
is the full project name you just copied.To set the project name permanently:
- Open a terminal app, such as PowerShell or Command Prompt.
-
Paste in the following command and press enter:
where
workspace-name/project-name
is the full project name you just copied. -
To activate the change, restart the terminal.
You can use the os library to set the project name as an environment variable:
where
workspace-name/project-name
is the full project name you just copied.Note that any environment variables declared this way won't persist after the notebook kernel shuts down. If you start a new kernel, they need to be set again.
Stopping runs and other objects#
Best practice
When you're done logging to a Neptune object, stop it with the stop()
method in your code.
When a script finishes executing, runs and other Neptune objects are stopped automatically. However, in interactive sessions (such as Jupyter notebooks) the connection to the object is stopped only when the kernel stops.
To avoid logging metadata for longer than intended, it's best to stop each Neptune object explicitly with the stop()
method. This is especially important for Run
objects, which by default1 log system metrics in the background for as long as they're active.
import neptune
run = neptune.init_run()
run["namespace/field"] = "some metadata"
...
run.stop()
Other cases for using stop()
include (but aren't limited to):
- Creating multiple runs in a single script
- Continuous training flows
- Model registry flows
While model and project objects are synchronous and do not monitor things in the background as runs do, it's also best to stop them when the connection to Neptune is not needed.
Enabling background monitoring in interactive sessions
To mitigate undesired background logging, in interactive sessions, some monitoring options are disabled by default.
You can turn them on with:
Maximum number of fields#
Best practice
Make sure you create no more than 9000 fields in a single Neptune object.
To ensure optimal performance, the number of user-defined fields is limited to 9000. This limit concerns any one Neptune object that can contain metadata fields, such as a Run
or Model
object.
Note
The field limit is not about metadata limits per se.
For example, rather than assigning 100 values to 100 fields (run["field1"] = 0
, run["field2"] = 1
, ..., ) you could construct a series of those 100 values and log them under a single field:
You can log numerical values, strings, and files as a series. For details, see Field types overview.
If you exceed the field limit while logging, Neptune stops the synchronization to the server and stores the data locally. To continue uploading the metadata, you need to delete some excess fields from the object in question, then sync the data manually.
For detailed instructions, see Troubleshooting ≫ Too many fields in object.
Reserved namespaces#
Best practice
Avoid creating custom fields in the sys or monitoring namespaces.
- sys - The system namespace is reserved, that is, you cannot create new fields in this namespace.
- Note: You can manually edit several field values of the system namespace, such as the description, stage, and tags. For details, see System namespace overview.
- monitoring - The monitoring namespace is intended specifically for system monitoring rather than model monitoring. Examples of system monitoring include hardware consumption and standard streams (stdout and stderr). For more, see Log system metrics.
-
See also: What Neptune logs automatically ↩