Skip to main content
App version: 3.20250908

Using Neptune in distributed environments

You can log metadata to the same run from multiple separate processes at once.

Use the custom run ID to access the same run in each process:

Process A
run = Run(
    run_id="seagull-123",
)

for step in epoch:
    # some training loop
    run.log_metrics(
        data={"train/loss": 0.14},
        step=step,
    )
Process B
run = Run(
    run_id="seagull-123",
)

for step in epoch:
    # some validation loop
    run.log_metrics(
        data={"val/loss": 0.16},
        step=step,
    )

In the above example, the two processes are logging metrics separately, to different attribute paths: train/loss and val/loss, respectively.

caution

If you call log_configs() on an existing attribute, the new value overwrites the previous one. For details, see Edit run configs.

Related

Logging the same metric across processes​

You can log the same metric across multiple processes, as long as steps aren't logged out of order within the same attribute. For details, see Log metrics: Setting step values.

Logging from multiple ranks​

Since metrics have to be logged strictly in order of steps, we recommend logging to a different namespace from each rank. This ensures that there's no collision between ranks.

Example implementation:

import os
from neptune_scale import Run

...
rank = os.getenv["RANK", 0]  # The variable depends on the framework used
...
run = Run(run_id="seagull-123", ...)  # Use the same run_id across ranks
run.log_metrics(f"{rank}/path/to/metric_name": METRIC_VALUE})
...

Multiprocessing best practices​

When using Neptune in a multiprocessing environment, we recommend using spawn as the method to start a child process. It ensures better resource separation between processes and better supports frameworks such as JAX.

spawn is the default method in Windows and MacOS.

Waiting for metadata to be saved​

To ensure that the logged metadata has been fully processed by Neptune before the script execution continues, use the wait_for_processing() method.

The method returns True once the data is saved and available on the Neptune server.