Skip to content

How to use wait() and sync() to ensure synchronous logging#

To ensure synchronous communication between multiple processes, you can make use of the wait and sync operations. These are available to any Neptune objects.

How they work#

The local representation is formed based on metadata tracking calls made with Neptune (such as append() and upload()). The Neptune server has the true representation based on all tracking calls.

  • sync() – synchronize the local representation with the server representation.
    • In version control terms, it's like pulling changes from the remote. This is useful in cases when parallel processes are logging to the servers and you want to access a value logged by some other process.
  • wait() – wait for all tracking calls to reach Neptune before continuing execution.
    • This ensures that all of the queued Neptune logging calls reach the servers before the script continues execution.

To combine the two operations, you can use sync(wait=True) on a run or other Neptune object. It's equivalent to:

some_neptune_object.wait()
some_neptune_object.sync()

Examples#

Fetching a value that is guaranteed to exist#

If you're fetching a value from a run that was logged earlier in the same process, or from another instance in a parallel setup, first run sync(wait=True) so that the server and local object are in sync.

import neptune

run = neptune.init_run()
run["parameter"] = param
If logging from a single process
run.wait()
If logging from multiple processes
run.sync(wait=True)

You can now fetch the parameter from the run:

fetched_param = run["parameter"].fetch()

Note

  • A variable logged by another process may not be immediately visible on the server.

  • If the script is not doing model training, you can initialize Neptune in the sync connection mode. This makes every method synchronous.

    For details, see Setting the logging mode.

Copying parameters from a run to a model version#

Log the metadata and call wait() on the run:

run["parameters"] = {"lr":0.01, "gamma":0.9}
run.wait()

The server representation of the run now has the "parameters" dictionary assigned. The value of this field can be fetched and used elsewhere in your code:

model_version["parameters"] = run["parameters"].fetch()

Creating a version of a recently registered model#

Create a new model:

model = neptune.init_model(key=MODEL_KEY)

Wait for the model object to be created on the server:

model.wait()

Fetch the model ID and use it to create a model version:

model_id = model["sys/id"].fetch()
model_version = neptune.init_model_version(model=model_id)