Enable or disable logging of system metrics#

By default, Neptune automatically logs the following system metrics:

Hardware consumption: memory, CPU, GPU (only NVIDIA), plus its memory and power usage.
Console logs: stdout, stderr, and traceback reports.

They're stored in the monitoring namespace of each run. By default, the metrics are organized into sub-namespaces per process:

run
|-- monitoring
    | -- <hash>
        |-- cpu
        |-- gpu
        |-- gpu_memory
        |-- gpu_power
        |-- hostname
        |-- memory
        |-- pid       # Process ID
        |-- stderr
        |-- stdout
        |-- tid       # Thread ID
    |-- <hash>
        |-- cpu
        |-- gpu
        |-- ...

where the value of <hash> is calculated based on environment information. This is to ensure that the hash is unique for each process.

See example in Neptune

How to disable

To turn off logging of system metrics, use one or more of the following options with init_run():

run = neptune.init_run(
    capture_hardware_metrics=False,
    capture_stdout=False,
    capture_stderr=False,
    capture_traceback=False,
)

Setting all of the above to False prevents the creation of a monitoring namespace.

Setting a custom monitoring namespace name#

To provide your custom monitoring namespace name, pass it to the monitoring_namespace argument at initialization:

import neptune

run = neptune.init_run(monitoring_namespace="monitoring")

This custom name replaces the monitoring/<hash>/ pattern:

run
|-- monitoring
    |-- cpu
    |-- gpu
    |-- gpu_memory
    |-- gpu_power
    |-- hostname
    |-- memory
    |-- pid
    |-- stderr
    |-- stdout
    |-- tid

Monitoring distributed runs#

When logging metrics from multiple processes, ensure that your custom name is unique for each process:

Process 1

run = neptune.init_run(
    custom_run_id="honest-mustang",
    monitoring_namespace="monitoring/<process_1_identifier>",
)

Process 2

run = neptune.init_run(
    custom_run_id="honest-mustang",
    monitoring_namespace="monitoring/<process_2_identifier>",
)

Logging custom system metrics#

To log custom monitoring metrics, you can access the monitoring namespace like any other namespace and define custom fields:

import neptune

run = neptune.init_run(monitoring_namespace="monitoring")
run["monitoring/custom-metric"].append(custom_metric_value)

Separating custom and auto-created metrics#

Neptune organizes the auto-created monitoring fields under the path that you provide to the monitoring_namespace argument.

If you have custom metrics that you'd like to structure separately from the auto-created ones, you can include a custom sub-namespace in the path:

run = neptune.init_run(monitoring_namespace="monitoring/auto-created-metrics")

Then, you can organize custom metrics or namespaces under the monitoring root namespace:

monitoring_ns = run["monitoring"]
monitoring_ns["custom-metric"].append(custom_metric_value)

Result

run
|-- monitoring
    |-- auto-created-metrics
        |-- cpu
        |-- gpu
        |-- gpu_memory
        |-- gpu_power
        |-- hostname
        |-- memory
        |-- pid
        |-- stderr
        |-- stdout
        |-- tid
    |-- custom-metric

Logging Python Logger output#

To capture logs from Python Logger , create and add a Neptune handler to your Logger:

import logging
from neptune.integrations.python_logger import NeptuneHandler

logger = logging.getLogger("my_python_logger")

logger.addHandler(NeptuneHandler(run=run))  # a previously initialized run

logger.debug("Starting training")

Tip

You can customize what level of logs you want to capture or the name of the field where the logs will be stored.

For details, see API reference → Python Logger.

Capturing external logs#

Neptune can be used in any Python script, so logs can be tracked in separate processes.

The best way to access a run in multiple places is with a custom ID. You can then access the same run and stream additional metadata to it from any Python script.

Assign a custom ID when creating the run:
Main script
```
import neptune

run = neptune.init_run(custom_run_id="adventurous-barracuda")
```
Neptune also supports setting the custom run ID as an environment variable.
In the script where you want to capture the logs, use the same code to open the run:
Separate script
```
import neptune

run = neptune.init_run(custom_run_id="adventurous-barracuda")
```
(Optional) Neptune automatically logs standard streams and some system metrics. To capture custom logs:
```
run["process_2/logs"] = ...  # Assign custom logs
```
If you're logging per iteration, you can also create a StringSeries by appending strings iteratively:
```
for e in epoch:
    run["process_2/logs"].append("some string")
```

Result

All of the logs are captured by the same run.

In the Neptune app, you can browse the logs in All metadata or display them in a custom dashboard.

Enabling monitoring in interactive sessions#

By default, monitoring of system metrics is turned off for interactive Python kernels, such as Jupyter notebooks. This includes logging of hardware consumption and standard streams (stdout and stderr).

To turn it on, you need to explicitly set the related initialization parameters to True:

>>> import neptune
>>> run = neptune.init_run(
...     capture_hardware_metrics=True,
...     capture_stderr=True,
...     capture_stdout=True,
...     capture_traceback=True,
... )

Disabling monitoring#

To turn off system monitoring, use one or more of following options with init_run():

run = neptune.init_run(
    capture_hardware_metrics=False,
    capture_stdout=False,
    capture_stderr=False,
    capture_traceback=False,
)

Setting all of the above options to False prevents the creation of a monitoring namespace altogether.