Skip to content

Connection modes reference#

The connection mode determines how the metadata tracking will work when you initialize a run or other object.

Available connection modes:

Mode Value Description
Asynchronous (default) async Neptune stores the logged data locally and periodically synchronizes it with the server in the background.
Debug debug No data is stored or sent anywhere.
Offline offline Data is not sent to Neptune but stored locally.
Read only read-only No data can be logged, only fetched.
Synchronous sync Neptune immediately synchronizes the data each time a tracking call (such as append() or upload()) is made.

Setting the connection mode#

You can set the mode in the following ways:

The environment variable is used if the mode parameter is omitted from the init function.

export NEPTUNE_MODE='debug'

export NEPTUNE_MODE='debug'

setx NEPTUNE_MODE 'debug'

Tip

This can be a handy way to debug or "switch off" Neptune without modifying your code.

When initializing a run or other object, provide the mode parameter to the initialization function:

import neptune

run = neptune.init_run(
    name="My debug run",
    mode="debug",  # "async", "sync", "offline", "read-only", or "debug"
)

Available modes#

Asynchronous mode#

By default, metadata tracking works asynchronously: The tracked data is temporarily stored on the local disk and periodically synchronized by a separate thread in the background. All tracking calls (like append(), assign(), upload()) are non-blocking.

To be able to restore tracked data in case you experience connectivity issues or your machine restarts (for example, if you use spot instances), you should use Neptune with persistent disks.

Disk flushing#

Neptune triggers disk flushing:

  • When you call wait() on a Neptune object.
  • When the connection to the object stops (the object context is destroyed).
  • Every 5 seconds (default). You can set a different interval with the flush_period parameter.

Note

The OS can trigger disk flushing implicitly. As such, setting flush_period=600 will flush the disk at least every 600 seconds, but it can happen more frequently for other reasons.

Error handling#

Related

Best practices

In asynchronous mode, metadata tracking calls do not throw exceptions related to connectivity or metadata consistency issues. Such issues are printed to stderr.

Connectivity issues#

In asynchronous mode, if there is a problem with the connection to Neptune servers (for example, due to internet connectivity issues), the Neptune client library will track your metadata locally and continuously try to re-establish a connection with the Neptune servers.

Once your run is finished, if the connection has not been re-established, Neptune will continue trying for another 5 minutes. After this period, the process will end. All the unsuccessful tracking calls are stored on the local disk.

You can upload the unsynchronized metadata later via the Neptune Command Line Interface, using the neptune sync command.

Related

Debug mode#

Debug mode essentially turns off all the Neptune-related logging. It can come in handy when you're debugging your code and don't want to pollute your Neptune project.

In contrast to offline mode, debug mode doesn't save any data locally.

import neptune

run = neptune.init_run(mode="debug")

Offline mode#

In this mode, no connection to Neptune servers is established. Instead, all the tracked metadata is stored on the local disk, in the .neptune folder of the current working directory.

Good to know

  • You can upload the data to the Neptune servers manually with the neptune sync command.
  • It's not possible to use a custom run ID in offline mode.
import neptune

run = neptune.init_run(mode="offline")

Read-only mode#

Read only mode is useful when you only want to fetch metadata rather than record it. For example, when you're connecting to Neptune to perform specific analysis on your run metadata.

In read only mode:

  • No write operation is possible. Any write-like operation will be ignored.
  • No metadata is sent to Neptune. In particular, stdout and stderr are not captured, and no hardware consumption is logged.

Only pre-existing objects can be initialized in read only mode.

import neptune

# Connecting to run with identifier NER-2
run = neptune.init_run(with_id="NER-2", mode="read-only")

Synchronous mode#

In synchronous mode, Neptune executes a tracking call only after all other calls have reached the Neptune servers.

In this mode, tracking methods throw exceptions in case of issues related to connectivity or the consistency of an object's local representation. Neptune will retry each request for 60 seconds before throwing the exception.

import neptune

run = neptune.init_run(name="My synchronous", mode="sync")

Tip

If there's an issue with your code but you're not sure where, synchronous mode can help you locate the problematic part.

Note that this mode may slow down your logging.

For an example of ensuring synchronous tracking across multiple processes, see How to use wait() and sync() to ensure synchronous logging.