Skip to content

How to use wait() and sync() to ensure synchronous logging#

To ensure synchronous communication between multiple processes, you can make use of the wait and sync operations. These are available to any Neptune objects.

How they work#

The local representation is formed based on metadata tracking calls made with Neptune (such as append() and upload()). The Neptune server has the true representation based on all tracking calls.

  • sync() – synchronize the local representation with the server representation.
    • In version control terms, it's like pulling changes from the remote. This is useful in cases when parallel processes are logging to the servers and you want to access a value logged by some other process.
  • wait() – wait for all tracking calls to reach Neptune before continuing execution.
    • This ensures that all of the queued Neptune logging calls reach the servers before the script continues execution.

To combine the two operations, you can use sync(wait=True) on a run or other Neptune object. It's equivalent to:

some_neptune_object.wait()
some_neptune_object.sync()

Example: Fetching a value that is guaranteed to exist#

If you're fetching a value from a run that was logged earlier in the same process, or from another instance in a parallel setup, first run sync(wait=True) so that the server and local object are in sync.

import neptune

run = neptune.init_run()
run["parameter"] = param
If logging from a single process
run.wait()
If logging from multiple processes
run.sync(wait=True)

You can now fetch the parameter from the run:

fetched_param = run["parameter"].fetch()

Note

  • A variable logged by another process may not be immediately visible on the server.

  • If the script is not doing model training, you can initialize Neptune in the sync connection mode. This makes every method synchronous.

    For details, see Setting the logging mode.