neptune
#
You can use the global neptune
object to initialize new Neptune objects or resume existing ones.
init_project()
#
Starts a connection to an existing Neptune project. You can use it to fetch metadata of runs in the project.
You can also use the project object to manage metadata common to the whole project, such as information about datasets, links to documents, or key project metrics.
Parameters
Name | Type | Default | Description |
---|---|---|---|
project |
str , optional |
None |
Name of a project in the form workspace-name/project-name . If None , the value of the NEPTUNE_PROJECT environment variable is used. |
api_token |
str , optional |
None |
Your Neptune API token (or a service account's API token). If None , the value of the NEPTUNE_API_TOKEN environment variable is used.To keep your token secure, avoid placing it in source code. Instead, save it as an environment variable. |
mode |
str , optional |
async |
Connection mode in which the logging will work. Possible values are async , sync , read-only , and debug .If you leave it out, the value of the |
flush_period |
float , optional |
5 (seconds) |
In asynchronous (default) connection mode, how often Neptune should trigger disk flushing. |
proxies |
dict , optional |
None |
Argument passed to HTTP calls made via the Requests library. For details on proxies, see the Requests documentation. |
async_lag_callback |
NeptuneObjectCallback , optional |
None |
Custom callback function which is called if the lag between a queued operation and its synchronization with the server exceeds the duration defined by async_lag_threshold . The callback should take a Project object as the argument and can contain any custom code, such as calling stop() on the object.Note: Instead of using this argument, you can use Neptune's default callback by setting the |
async_lag_threshold |
float , optional |
1800.0 (seconds) |
Duration between the queueing and synchronization of an operation. If a lag callback (default callback enabled via environment variable or custom callback passed to the async_lag_callback argument) is enabled, the callback is called when this duration is exceeded. |
async_no_progress_callback |
NeptuneObjectCallback , optional |
None |
Custom callback function which is called if there has been no synchronization progress whatsoever for the duration defined by async_no_progress_threshold . The callback should take a Project object as the argument and can contain any custom code, such as calling stop() on the object.Note: Instead of using this argument, you can use Neptune's default callback by setting the |
async_no_progress_threshold |
float , optional |
300.0 (seconds) |
For how long there has been no synchronization progress. If a no-progress callback (default callback enabled via environment variable or custom callback passed to the async_no_progress_callback argument) is enabled, the callback is called when this duration is exceeded. |
Deprecated name
parameter
The name
parameter was changed to project
in neptune-client 0.16.16
.
Returns
Project
object that can be used to interact with the project as a whole, like logging or fetching project-level metadata.
Example
Connect to the project "classification" in the workspace "ml-team":
import neptune
project = neptune.init_project(project="ml-team/classification")
project["general/data_analysis"].upload("data_analysis.ipynb")
Note on collaboration
The project
object follows the same logic as other Neptune objects: If you assign a new value to an existing field, the new value overwrites the previous one.
In a given project, you always initialize and work with the same project
object, so take care not to accidentally overwrite each other's entries if your team is collaborating on project metadata.
Tip: Recall that the append()
method appends the logged value to a series. It works for text strings as well as numerical values.
Connect to a project in read-only mode:
init_run()
#
Starts a new tracked run and adds it to the top of the experiments table.
Since all parameters are optional, the minimal invocation is:
See also: Defining a custom init_run()
function, for keeping the logging consistent within your team.
Note for interactive sessions
If you initialize a run in an interactive session (Python interpreter or Jupyter Notebook), the following monitoring options are disabled by default:
capture_stdout
capture_stderr
capture_traceback
capture_hardware_metrics
To enable them, set each parameter to True
when initializing the run. Note: The monitoring will continue until you call run.stop()
or the kernel stops.
Also note: Your source files can only be tracked if you pass the path(s) to the source_code
argument. For help, see Logging source code.
Parameters
Name | Type | Default | Description |
---|---|---|---|
project |
str , optional |
None |
Name of a project in the form workspace-name/project-name . If None , the value of the NEPTUNE_PROJECT environment variable is used. |
api_token |
str , optional |
None |
Your Neptune API token (or a service account's API token). If None , the value of the NEPTUNE_API_TOKEN environment variable is used.To keep your token secure, avoid placing it in source code. Instead, save it as an environment variable. |
with_id |
str , optional |
None |
The Neptune identifier of an existing run to resume, such as "CLS-11". The identifier is stored in the object's sys/id field. If omitted or None is passed, a new tracked run is created. |
custom_run_id |
str , optional |
None |
A unique identifier that can be used to log metadata to a single run from multiple locations. Max length: 36 characters. If None and the NEPTUNE_CUSTOM_RUN_ID environment variable is set, Neptune will use that as the custom_run_id value. For details, see Set custom run ID. |
mode |
str , optional |
async |
Connection mode in which the logging will work. Possible values are async , sync , offline , read-only , and debug .If you leave it out, the value of the |
name |
str , optional |
Neptune ID | Custom name for the run. You can use it as a human-readable ID and add it as a column in the experiments table (sys/name ). If left empty, once the run is synchronized with the server, Neptune sets the auto-generated identifier (sys/id ) as the name. |
description |
str , optional |
"" |
Editable description of the run. You can add it as a column in the experiments table (sys/description ). |
tags |
list , optional |
[] |
Must be a list of str which represent the tags for the run. You can edit them after run is created, either in the run information or experiments table. |
source_files |
list or str , optional |
None |
List of source files to be uploaded. Must be list of If Unix style pathname pattern expansion is supported. For example, you can pass |
capture_stdout |
Boolean , optional |
True |
Whether to log the standard output stream. Is logged in the monitoring namespace. |
capture_stderr |
Boolean , optional |
True |
Whether to log the standard error stream. Is logged in the monitoring namespace. |
capture_hardware_metrics |
Boolean , optional |
True |
Whether to track hardware consumption (CPU, GPU, memory utilization). Logged in the monitoring namespace. |
fail_on_exception |
Boolean , optional |
True |
If an uncaught exception occurs, whether to set run's Failed state to True . |
monitoring_namespace |
str , optional |
"monitoring" |
Namespace inside which all monitoring logs will be stored. |
flush_period |
float , optional |
5 (seconds) |
In asynchronous (default) connection mode, how often Neptune should trigger disk flushing. |
proxies |
dict , optional |
None |
Argument passed to HTTP calls made via the Requests library. For details on proxies, see the Requests documentation. |
capture_traceback |
Boolean , optional |
True |
In case of an exception, whether to log the traceback of the run. |
git_ref |
GitRef or Boolean |
None |
GitRef object containing information about the Git repository path.If To specify a different location, set to To turn off Git tracking for the run, set to |
dependencies |
str , optional |
None |
Tracks environment requirements. If you pass "infer" to this argument, Neptune logs dependencies installed in the current environment. You can also pass a path to your dependency file directly. If left empty, no dependency file is uploaded. |
async_lag_callback |
NeptuneObjectCallback , optional |
None |
Custom callback function which is called if the lag between a queued operation and its synchronization with the server exceeds the duration defined by async_lag_threshold . The callback should take a Run object as the argument and can contain any custom code, such as calling stop() on the object.Note: Instead of using this argument, you can use Neptune's default callback by setting the |
async_lag_threshold |
float , optional |
1800.0 (seconds) |
Duration between the queueing and synchronization of an operation. If a lag callback (default callback enabled via environment variable or custom callback passed to the async_lag_callback argument) is enabled, the callback is called when this duration is exceeded. |
async_no_progress_callback |
NeptuneObjectCallback , optional |
None |
Custom callback function which is called if there has been no synchronization progress whatsoever for the duration defined by async_no_progress_threshold . The callback should take a Run object as the argument and can contain any custom code, such as calling stop() on the object.Note: Instead of using this argument, you can use Neptune's default callback by setting the |
async_no_progress_threshold |
float , optional |
300.0 (seconds) |
For how long there has been no synchronization progress. If a no-progress callback (default callback enabled via environment variable or custom callback passed to the async_no_progress_callback argument) is enabled, the callback is called when this duration is exceeded. |
Deprecated run
parameter
The run
parameter was changed to with_id
in neptune-client 0.16.7
.
Returns
Run
object that is used to manage the tracked run and log metadata to it.
Examples
Create a new run:
Create a run with a custom name:
Create a run with:
- a name and description
- custom monitoring namespace (recommended)
- no sources files or Git info uploaded
run = neptune.init_run(
name="jolly-butter",
description="neural net trained on MNIST",
monitoring_namespace="monitoring",
source_files=[],
git_ref=False,
)
Resume an existing run, to continue logging to it:
import neptune
run = neptune.init_run(with_id="CLAS-134")
run["namespace/field"] = some_metadata
...
How do I find the ID?
The Neptune ID is a unique identifier for the run. The Experiments tab displays it in the leftmost column.
In the run structure, the ID is stored in the system namespace (sys
).
-
If the run is active, you can obtain its ID with
run["sys/id"].fetch()
. For example: -
If you set a custom run ID, it's stored at
sys/custom_run_id
:
Resume a run in read-only mode, to only read metadata from it:
Resume a run in order to delete data:
run = neptune.init_run(
with_id="CLAS-134",
capture_hardware_metrics=False,
capture_stderr=False,
capture_stdout=False,
capture_traceback=False,
git_ref=False,
source_code=[],
)
del run["namespace"]
Start a new run and log all Python files in the current working directory (excluding hidden files with names beginning with a dot):
Log all Python files from all subdirectories (excluding hidden files):
Log all files and directories in the current working directory (excluding hidden files):
Log all files and directories in the current working directory, including hidden files
Larger example:
run = neptune.init_run(
name="first-pytorch-ever",
description="Write a longer description here.",
tags=["pytorch", "test", "do-not-delete"],
source_files=["training_with_pytorch.py", "net.py"],
capture_hardware_metrics=False,
dependencies="infer",
)
While it's not a recommended practice, you can also pass your Neptune credentials in the code when initializing Neptune.
run = neptune.init_run(
project="ml-team/classification", # your full project name here
api_token="h0dHBzOi8aHR0cHM6Lkc78ghs74kl0jvYh3Kb8", # your API token here
)
How do I save my credentials as environment variables?
Set your Neptune API token and full project name to the NEPTUNE_API_TOKEN
and NEPTUNE_PROJECT
environment variables, respectively.
You can also navigate to Settings → Edit the system environment variables and add the variables there.
To find your credentials:
- API token: In the bottom-left corner of the Neptune app, expand your user menu and select Get your API token. If you need the token of a service account, go to the workspace or project settings and enter the Service accounts settings.
- Project name: Your full project name has the form
workspace-name/project-name
. You can copy it from the project menu ( → Details & privacy).
If you're working in Google Colab, you can set your credentials with the os and getpass libraries:
ANONYMOUS_API_TOKEN
#
API token for anonymous logging.
You can use this value for the api_token
argument of the init
functions.
Example
Start a run as an anonymous user:
import neptune
run = neptune.init_run(
api_token=neptune.ANONYMOUS_API_TOKEN,
project="common/quickstarts",
)
If you're not using a function from the neptune
package to initialize Neptune, you can also import and use the anonymous API token separately.
from pytorch_lightning import Trainer
from pytorch_lightning.loggers import NeptuneLogger
from neptune import ANONYMOUS_API_TOKEN
neptune_logger = NeptuneLogger(
api_key=ANONYMOUS_API_TOKEN,
project=...,
)
Deprecated#
This part of the API is deprecated.
get_last_run()
#
Deprecated
This function is not available as of neptune 1.0
.
Fetches the last created run
object.
Returns
run
object last created by the global neptune
object.
Example
import neptune
# Crate a new tracked run
neptune.init_run(name="A new approach", source_files="**/*.py")
# Oops - we didn't assign it to a "run" variable.
# Not a problem! We've got you covered:
run = neptune.get_last_run()
get_project()
#
Deprecated
This function is not available as of neptune 1.0
.
Use neptune.init_project(mode="read-only")
instead.
init()
#
init_model()
#
Initializes a Model
object from an existing or new model.
You can use this function to create a new model from code or to perform actions on existing models.
Parameters
Name | Type | Default | Description |
---|---|---|---|
with_id |
str , optional |
None |
The Neptune identifier of an existing model to resume, in the form PROJECTKEY-MODELKEY . For example, "NLU-PRE1" . The identifier is stored in the object's sys/id field. If omitted or None is passed, a new model is created. |
name |
str , optional |
"Untitled" |
A custom name for the model. You can use it as a human-readable ID and add it to the models table as a column (sys/name ). |
key |
str , optional |
None |
Key for a model. Required when creating a new model.
|
project |
str , optional |
None |
Name of a project in the form workspace-name/project-name . If None , the value of the NEPTUNE_PROJECT environment variable is used. |
api_token |
str , optional |
None |
Your Neptune API token (or a service account's API token). If None , the value of the NEPTUNE_API_TOKEN environment variable is used.To keep your token secure, avoid placing it in source code. Instead, save it as an environment variable. |
mode |
str , optional |
async |
Connection mode in which the logging will work. Possible values are async , sync , read-only , and debug .If you leave it out, the value of the |
flush_period |
float , optional |
5 (seconds) |
In asynchronous (default) connection mode, how often Neptune should trigger disk flushing. |
proxies |
dict , optional |
None |
Argument passed to HTTP calls made via the Requests library. For details on proxies, see the Requests documentation. |
async_lag_callback |
NeptuneObjectCallback , optional |
None |
Custom callback function which is called if the lag between a queued operation and its synchronization with the server exceeds the duration defined by async_lag_threshold . The callback should take a Model object as the argument and can contain any custom code, such as calling stop() on the object.Note: Instead of using this argument, you can use Neptune's default callback by setting the |
async_lag_threshold |
float , optional |
1800.0 (seconds) |
Duration between the queueing and synchronization of an operation. If a lag callback (default callback enabled via environment variable or custom callback passed to the async_lag_callback argument) is enabled, the callback is called when this duration is exceeded. |
async_no_progress_callback |
NeptuneObjectCallback , optional |
None |
Custom callback function which is called if there has been no synchronization progress whatsoever for the duration defined by async_no_progress_threshold . The callback should take a Model object as the argument and can contain any custom code, such as calling stop() on the object.Note: Instead of using this argument, you can use Neptune's default callback by setting the |
async_no_progress_threshold |
float , optional |
300.0 (seconds) |
For how long there has been no synchronization progress. If a no-progress callback (default callback enabled via environment variable or custom callback passed to the async_no_progress_callback argument) is enabled, the callback is called when this duration is exceeded. |
Deprecated model
parameter
The model
parameter was changed to with_id
in neptune-client 0.16.7
.
Returns
Model
object that is used to manage the model and log metadata to it.
Examples
Create a new model and log the signature:
import neptune
model = neptune.init_model(key="PRE")
model["signature"].upload("model_signature.json")
You can provide the project parameter as an environment variable or directly in the init_model()
function:
When creating a model, you can give it a name:
Initialize existing model with identifier "CLS-PRE":
To prevent modifications when connecting to an existing model, you can connect in read-only mode:
model = neptune.init_model(with_id="CLS-PRE", mode="read-only")
model["validation/data/v0.1"].download()
init_model_version()
#
Initializes a ModelVersion
object from an existing or new model version.
You can use this function to create a new model version from code or to perform actions on existing model versions.
Parameters
Name | Type | Default | Description |
---|---|---|---|
with_id |
str , optional |
None |
The Neptune identifier of an existing model version to resume, such as "CLS-PRE-3". The identifier is stored in the object's sys/id field. If omitted or None is passed, a new model version is created. |
name |
str , optional |
"Untitled" |
A custom name for the model version. You can use it as a human-readable ID and add it to the model versions table as a column (sys/name ). |
model |
str , optional |
None |
Identifier of the model for which the new version should be created. Required when creating a new model version. The identifier is stored in the model's sys/id field. |
project |
str , optional |
None |
Name of a project in the form workspace-name/project-name . If None , the value of the NEPTUNE_PROJECT environment variable is used. |
api_token |
str , optional |
None |
Your Neptune API token (or a service account's API token). If None , the value of the NEPTUNE_API_TOKEN environment variable is used.To keep your token secure, avoid placing it in source code. Instead, save it as an environment variable. |
mode |
str , optional |
async |
Connection mode in which the logging will work. Possible values are async , sync , read-only , and debug .If you leave it out, the value of the |
flush_period |
float , optional |
5 (seconds) |
In asynchronous (default) connection mode, how often Neptune should trigger disk flushing. |
proxies |
dict , optional |
None |
Argument passed to HTTP calls made via the Requests library. For details on proxies, see the Requests documentation. |
async_lag_callback |
NeptuneObjectCallback , optional |
None |
Custom callback function which is called if the lag between a queued operation and its synchronization with the server exceeds the duration defined by async_lag_threshold . The callback should take a ModelVersion object as the argument and can contain any custom code, such as calling stop() on the object.Note: Instead of using this argument, you can use Neptune's default callback by setting the |
async_lag_threshold |
float , optional |
1800.0 (seconds) |
Duration between the queueing and synchronization of an operation. If a lag callback (default callback enabled via environment variable or custom callback passed to the async_lag_callback argument) is enabled, the callback is called when this duration is exceeded. |
async_no_progress_callback |
NeptuneObjectCallback , optional |
None |
Custom callback function which is called if there has been no synchronization progress whatsoever for the duration defined by async_no_progress_threshold . The callback should take a ModelVersion object as the argument and can contain any custom code, such as calling stop() on the object.Note: Instead of using this argument, you can use Neptune's default callback by setting the |
async_no_progress_threshold |
float , optional |
300.0 (seconds) |
For how long there has been no synchronization progress. If a no-progress callback (default callback enabled via environment variable or custom callback passed to the async_no_progress_callback argument) is enabled, the callback is called when this duration is exceeded. |
Deprecated version
parameter
The version
parameter was changed to with_id
in neptune-client 0.16.7
.
Returns
ModelVersion
object that is used to manage the model version and log metadata to it.
Examples
Create a new model version for a model with identifier "CLS-PRE":
You can provide the project parameter as an environment variable or directly in the init_model_version()
function:
Initialize an existing model version with identifier "CLS-PRE-12" and log additional artifact metadata:
model_version = neptune.init_model_version(with_id="CLS-PRE-12")
model_version["dataset"].track_files("path/to/more/files")
To prevent modifications when connecting to an existing model version, you can connect in read-only mode:
model_version = neptune.init_model_version(with_id="CLS-PRE-12", mode="read-only")
val_acc = model_version["validation/metrics/acc"].fetch()