Enable or disable logging of system metrics

By default, Neptune automatically logs the following system metrics:

• Hardware consumption: CPU, GPU (only NVIDIA), and memory.
• Console logs: stdout, stderr, and traceback reports.

They're stored in the monitoring namespace of each run. By default, the metrics are organized into sub-namespaces per process:

run
|-- monitoring
    | -- <hash>
        |-- cpu
        |-- hostname
        |-- memory
        |-- pid       # Process ID
        |-- stderr
        |-- stdout
        |-- tid       # Thread ID
    |-- <hash>
        |-- cpu
        |-- hostname
        |-- ...

where the value of <hash> is calculated based on environment information. This is to ensure that the hash is unique for each process.

See example in Neptune 

How to disable

To turn off logging of system metrics, use one or more of the following options with init_run():

run = neptune.init_run(
    capture_hardware_metrics=False,
    capture_stdout=False,
    capture_stderr=False,
    capture_traceback=False,
)

Setting all of the above to False prevents the creation of a monitoring namespace.

Setting a custom monitoring namespace name

To provide your custom monitoring namespace name, pass it to the monitoring_namespace argument at initialization:

import neptune

run = neptune.init_run(monitoring_namespace="monitoring")

This custom name replaces the monitoring/<hash>/ pattern:

run
|-- monitoring
    |-- cpu
    |-- hostname
    |-- memory
    |-- pid
    |-- stderr
    |-- stdout
    |-- tid

Monitoring distributed runs

When logging metrics from multiple processes, ensure that your custom name is unique for each process:

Process 1

run = neptune.init_run(
    custom_run_id="honest-mustang",
    monitoring_namespace="monitoring/<process_1_identifier>",
)

Process 2

run = neptune.init_run(
    custom_run_id="honest-mustang",
    monitoring_namespace="monitoring/<process_2_identifier>",
)

Logging custom system metrics

To log custom monitoring metrics, you can access the monitoring namespace like any other namespace and define custom fields:

import neptune

run = neptune.init_run(monitoring_namespace="monitoring")
run["monitoring/custom-metric"].append(custom_metric_value)

Separating custom and auto-created metrics

Neptune organizes the auto-created monitoring fields under the path that you provide to the monitoring_namespace argument.

If you have custom metrics that you'd like to structure separately from the auto-created ones, you can include a custom sub-namespace in the path:

run = neptune.init_run(monitoring_namespace="monitoring/auto-created-metrics")

Then, you can organize custom metrics or namespaces under the monitoring root namespace:

monitoring_ns = run["monitoring"]
monitoring_ns["custom-metric"].append(custom_metric_value)

Result

run
|-- monitoring
    |-- auto-created-metrics
        |-- cpu
        |-- hostname
        |-- memory
        |-- pid
        |-- stderr
        |-- stdout
        |-- tid
    |-- custom-metric

Logging Python Logger output

To capture logs from Python Logger , create and add a Neptune handler to your Logger:

import logging
from neptune.integrations.python_logger import NeptuneHandler

logger = logging.getLogger("my_python_logger")

logger.addHandler(NeptuneHandler(run=run))  # a previously initialized run

logger.debug("Starting training")

Tip

You can customize what level of logs you want to capture or the name of the field where the logs will be stored.

For details, see API reference → Python Logger.

Enabling monitoring in interactive sessions

By default, monitoring of system metrics is turned off for interactive Python kernels, such as Jupyter notebooks. This includes logging of hardware consumption and standard streams (stdout and stderr).

To turn it on, you need to explicitly set the related initialization parameters to True:

>>> import neptune
>>> run = neptune.init_run(
...     capture_hardware_metrics=True,
...     capture_stderr=True,
...     capture_stdout=True,
...     capture_traceback=True,
... )

Disabling monitoring

To turn off system monitoring, use one or more of following options with init_run():

run = neptune.init_run(
    capture_hardware_metrics=False,
    capture_stdout=False,
    capture_stderr=False,
    capture_traceback=False,
)

Setting all of the above options to False prevents the creation of a monitoring namespace altogether.