What can you log and display?

Neptune supports log and display for many different types of model building metadata.

On this page, you can see how each ML metadata type can be logged via Neptune API and how it will be displayed in the Neptune UI.

Supported ML metadata types include:

Remember to create a Run before you start logging:

import neptune.new as neptune
run = neptune.init(api_token="YOUR_API_TOKEN",
project="YOUR_WORKSPACE/YOUR_PROJECT")

Metrics and losses

Log metrics and losses to Neptune using =and .log() methods:

# logging score (single value)
run['score'] = 0.97
run['test/acc'] = 0.97
# logging metric (series of values)
for epoch in range(100):
acc = ...
loss = ...
metric = ...
run["train/accuracy"].log(acc)
run["train/loss"].log(loss)
run['metric'].log(metric)
Metrics and losses in Neptune

Metric can be accuracy, MSE, or any numerical value. All metrics are visualized as Charts in the Run UI. You can also check grouped values in the All Metadata section of the Run UI and add metrics as columns to the Runs Table.

Parameters and Model Configuration

Log parameters one-by-one to Fields of a Run.

run['parameters/epoch_nr'] = 5
run['parameters/batch_size'] = 32
run['parameters/dense'] = 512
run['parameters/optimizer'] = 'sgd'
run['parameters/metrics'] = ['accuracy', 'mae']
run['parameters/activation'] = 'relu'

Define parameters as Python dictionary and log them all at once to a Namespace of a Run.

# Define parameters
PARAMS = {'epoch_nr': 5,
'batch_size': 32,
'dense': 512,
'optimizer': 'sgd',
'metrics': ['accuracy', 'binary_accuracy'],
'activation': 'relu'}
# Pass parameters
run['parameters'] = PARAMS

Logged parameters appear in the All metadata section of a single Run UI:

Parameters in the single Run UI

You can also add parameters as columns to the Runs Table:

Parameters in the Runs Table

Model checkpoints

You can log model checkpoints to a Run.

# Log PyTorch model weights
my_model = ...
torch.save(my_model, 'my_model.pt')
run['model_checkpoints/my_model'].upload('model_checkpoints/my_model.pt')

You can save model weights from any deep learning framework. Model checkpoints will appear in the All metadata section of the Run UI, for example in the ‘model_checkpoints’ Namespace.

Code

Neptune supports code versioning in three different ways.

Git information

If you start a Run from a directory that is a part of the git repo, Neptune will automatically find the .git directory and log information from it.

It creates a summary in the source_code/git section of the Run UI with the following Fields:

  • Dirty: whether the commit was dirty or not

  • Commit message: what was the commit message

  • Commit ID: Git SHA signature

  • Commit author: the author of the last commit

  • Commit date: the data of the last commit

  • Current branch: what is the current git branch

  • Remotes: list of git remotes your local .git is connected to

Code Snapshot

Neptune automatically snapshots code when you create a Run with neptune.init().

By default, it will only save the entry point file (e.g.main.py if you run python main.py), but you can pass a list of files or wildcard patterns ("glob") like: **/*.py to specify more files:

# Snapshot model.py and prep_data.py
run = neptune.init(...,
source_files=["model.py", "prep_data.py"])
# Snapshot all python files and 'config.yaml' file
run = neptune.init(...,
source_files=["**/*.py", "config.yaml"])

When using pattern expansion, such as '**/*.py', make sure that your expression does not log too many files or non-source code files.

For example, using '*' as a pattern will upload all files and directories from the current working directory (cwd). It may result in logging files that you did not want to upload and cluttering your project.

You will have all the files in the Source code section of the Run UI.

Neptune also logs the entry point filename so that you know which script was actually executed.

Notebook Code Snapshot

Neptune can automatically snapshot your Jupyter notebook checkpoint every time you create a Run in that notebook.

To get started, see Integration with Jupyter Notebook and Jupyter Lab.

Note You can also log Jupyter notebook checkpoint without creating a Neptune Run to log your Exploratory Data Analysis (EDA) or other work.

See Integration with Jupyter Notebook and Jupyter Lab.

Data versions

Neptune support many ways to log and display metadata about datasets.

User-defined fields

You can use Namespaces and basic Logging methods to organize any ML metadata in Neptune.

Typically, people log the following metadata about datasets:

  • the md5 hash of the dataset

  • the location of the dataset

  • a list of classes

  • a list of feature names (for structured data problems)

For example:

# Prepare dataset
(train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.fashion_mnist.load_data()
train_images = train_images / 255.0
test_images = test_images / 255.0
# Log data version as a Run property
run['train/images_version'] = hashlib.md5(train_images).hexdigest()
run['test/images_version'] = hashlib.md5(test_images).hexdigest()

Go to All metadata section of the Run UI and find your data versions and other dataset metadata.

You can also add data versions as columns in the Runs Table UI to group and compare Runs trained on the same versions of the dataset.

Data Version Control (DVC)

Neptune displays DVC files in the UI, you just need to specify which DVC files you would like to log.

For example, to log all the DVC files from your working directory and all subdirectories pass "**/*.dvc" to the source_files argument of neptune.init():

# Snapshot all .dvc files from any directory
run = neptune.init(...,
source_files=["**/*.dvc"])

Hardware consumption

To enable this feature you need to install psutil. Often, you just need to run:

pip install psutil

For more information, see the official psutil installation page.

Automatically monitor hardware utilization for your Runs:

  • cpu: total CPU consumption of your machine

  • memory: total memory consumption of your machine

  • gpu: GPU consumption of your machine

  • gpu_memory: GPU memory consumption of your machine

  • stderr: standard error logs from your console

  • stdout: standard output logs from your console

All that information is visualized in the Monitoring section of the Run UI.

To turn off the logging of hardware consumption use capture_hardware_ metrics argument of neptune.init():

# Turn off hardware monitoring
run = neptune.init(...,
capture_hardware_metrics=False)

To log hardware consumption to a different Namespace use monitoring_namespace argument argument of neptune.init():

# Assign custom name to the monitoring namespace
run = neptune.init(...,
monitoring_namespace='worker-7-monitoring')

If you see the Info (NVML): NVML Error: NVML Shared Library Not Found - GPU usage metrics may not be reported. your GPU consumption is not being logged.

It means that either:

  • there are no GPUs on your machine

  • your NVIDIA NVML library is not installed or configured properly. See how to install and configure NVML.

Logging GPU on Windows

On Windows, Neptune searches for the nvml.dll file in the standard locations:

  • C:\Program Files\NVIDIA Corporation\NVSMI\nvml.dll

  • C:\Windows\System32\nvml.dll

If you are having trouble logging GPU metrics on Windows, please check that your NVML installation is correct and that you have the nvml.dll file in either of those locations.

Alternatively, you can set a custom location of nvml.dll on Windows by setting the NVML_DLL_PATH environment variable.

Files

You can log any file or directory you want to Neptune by using .upload()and .upload_files()methods:

# Log file
run['aux/data'].upload('auxiliary-data.zip')
# Save multiple files under a path
run["config_files"].upload_files([path_to_config_1, path_to_config_2])
# You can also use wildcard patterns ("glob") with .upload_files()
run["preprocessing_scripts"].upload_files("./preprocessing/*.py")

You can browse and download files in the All metadata section of the Run UI.

Files are displayed in the UI based on the file format:

  • PNG, JPG, JPEG, GIF image formats are displayed as Images

  • HTML files are displayed as Interactive visualizations

  • MP4, OGG, OGV, MOV, M4V, MKV, WEBM are displayed as players for Video

  • MP3, M4A, OGA, WAVE are displayed as players for Audio

  • CSV files are displayed as Tables

  • IPYNB files are displayed as fully rendered Notebooks

  • Text files and source code files are displayed as text with appropriate syntax coloring if applicable

Make sure that you define the correct path to files that you want to upload when using wildcards like '*.png' or '**/*.yaml' as it may result in the unintended logging of a large amount of data and cluttering your storage.

You can delete files from a Run by using .pop()method:

run.pop('aux/data')

Console logs

stdout and stderr

To enable this feature you need to install psutil. Often, you just need to run:

pip install psutil

For more information, see the official psutil installation page.

Automatically save console logs for your Runs :

  • stderr: standard error logs from your console

  • stdout: standard output logs from your console

All that information is visualized in the Monitoring section of the Run UI.

To turn off the saving of console logs use capture_stdout capture_stderarguments of neptune.init():

# Turn off saving console logs
run = neptune.init(...,
capture_stdout=False,
capture_stderr=False,)

Python's Logger

You can also capture Python's Logger logs. Simply create and add Neptune handler to your Logger object:

import logging
from neptune.new.integrations.python_logger import NeptuneHandler
logger = logging.getLogger('my_python_logger')
logger.addHandler(NeptuneHandler(run=run))
logger.debug('Starting training')

When initializing the NeptuneHandler you can customize what level of logs you want to capture or the name of the field where the logs will be stored.

Images

You can log either a single image or a series of images to Neptune.

Several data formats are available:

In all cases, you will have images in the All metadata section of the Run UI (under Namespace defined by you), where you can browse and download them.

To log images use .upload() or .log(File) methods:

from neptune.new.types import File
# single image
run['train/bounding_boxes'].upload('bboxes.png')
# for series of images
for i in range(valid_nr):
ith_path = ...
run['valid/misclassified'].log(File(ith_path))

You can log an unlimited number of images either in the single image log or as a series of image logs.

Image file

Neptune currently supports PNG, JPEG, JPG, GIF formats for image files.

You can log image format files directly from the disk.

Log single image from disk with the .upload() method:

run['train/bounding_boxes'].upload('bboxes.png')

Log series of images in a for loop with the .log(File) method:

from neptune.new.types import File
for name in misclassified_images_names:
y_pred = ...
y_true = ...
run['misclassified_images'].log(File('misclassified_image.png'))

Notice the difference between .upload() for single images and .log() for series of images.

Matplotlib figure

Log Matplotlib figure (matplotlib.figure.Figure) as an image:

# Import matplotlib
import matplotlib.pyplot as plt
# Generate figure
fig = plt.figure(figsize=(7, 9))
...
# Log figure to run
run['matplotlib-fig'].upload(fig)

You can also log series of Matplotlib figures:

for epoch in range(params['iterations']):
plt_fig = get_histogram()
run['train/distribution'].log(plt_fig)

You will have Matplotlib figures displayed in the All metadata section of the Run UI, where you can browse and download them.

For more information, see How to use Neptune and Matplotlib.

PIL Image

You can log the PIL image directly from the memory:

image1 = Image.open('Representation-learning.jpg')
image2 = Image.open('RL-agents-1.jpg')
# Log image to run
run['representation_learning'].upload(image1)
run['reinforcement_learning'].upload(image2)

You can also log series of PIL images:

from neptune.new.types import File
for epoch in range(images_nr):
pil_image = Image.open('path_to_image')
run['train/distribution'].log(File(pil_image))

You will have images in the All metadata section of the Run UI, where you can browse and download them.

NumPy array

You can log NumPy arrays (2d or 3d) directly from the memory, and have them visualized as an image:

import numpy as np
from neptune.new.types import File
im_array = np.random.rand(100, 100, 3) * 255
run['np_image'].upload(File.as_image(im_array))

You can also log series of NumPy arrays:

from neptune.new.types import File
for epoch in range(data):
im_array = ...
run['train/distribution'].log(File.as_image(im_array))

You will have NumPy images in the All metadata section of the Run UI, where you can browse and download them.

TensorFlow tensor

You can log TensorFlow Tensors (2d or 3d) directly from the memory, and have them visualized as an image:

import tensorflow as tf
from neptune.new.types import File
tf_tensor = tensorflow.random.uniform(shape=[10, 20, 3])
run['tf_image'].upload(File.as_image(tf_tensorrray))

You can also log series of TensorFlow Tensors:

from neptune.new.types import File
for epoch in range(data):
tf_tensor = ...
run['train/distribution'].log(File.as_image(tf_tensor))

You will have TensorFlow Tensor images in the All metadata section of the Run UI, where you can browse and download them.

PyTorch tensor

You can log PyTorch Tensors (2d or 3d) directly from the memory, and have them visualized as an image:

import tensorflow as tf
from neptune.new.types import File
torch_tensor = torch.rand(30, 30)
run['torch_tensor'].upload(File.as_image(torch_tensor))

You can also log series of PyTorch Tensors:

from neptune.new.types import File
for epoch in range(data):
torch_tensor = ...
run['train/distribution'].log(File.as_image(torch_tensor))

You will have PyTorch Tensor images in the All metadata section of the Run UI, where you can browse and download them.

Interactive visualizations

You can log interactive charts and they will be rendered interactively in the Neptune UI. You will find it in the All metadata section of the Run UI.

Common visualization libraries are supported:

For a full-screen view, you can open visualization in the new browser tab, by clicking on the “arrow-pointing-top-right” icon, located right above your visualization.

HTML file

You can log any HTML file, and it will be rendered in the UI.

run['html'] = neptune.types.File('button_example.html')

HTML will appear in the All metadata section of the Run UI.

Matplotlib figure

This option is tested with matplotlib==3.2.0 and plotly==4.12.0. Make sure that you have the correct versions installed. See plotly installation guide.

You can log Matplotlib figure (matplotlib.figure.Figure) as an interactive visualization, by using neptune.types.File.as_html():

from neptune.new.types import File
fig = ...
run['visuals/matplotlib-fig'] = File.as_html(fig)

The interactive chart will appear in the All metadata section of the Run UI, where you can explore, open in full-screen and download it.

For more information, see How to use Neptune and Matplotlib.

Altair chart

You can log Altair charts as interactive visualizations, by using neptune.types.File.as_html():

from neptune.new.types import File
fig = ...
run['visuals/altair-fig'] = File.as_html(fig)

An interactive Altair chart will appear in the All metadata section of the Run UI, where you can explore, open in full screen and download it.

Bokeh plot

You can log Bokeh plots as interactive visualizations, by using neptune.types.File.as_html():

from neptune.new.types import File
fig = ...
run['visuals/bokeh-fig'] = File.as_html(fig)

An interactive Bokeh plot will appear in the All metadata section of the Run UI, where you can explore, open in full-screen and download it.

Plotly figure

You can log Plotly figures as interactive visualizations, by using neptune.types.File.as_html():

from neptune.new.types import File
fig = ...
run['visuals/plotly-fig'] = File.as_html(fig)

An interactive Plotly figure will appear in the All metadata section of the Run UI, where you can explore, open in full-screen and download it.

For more information, see How to use Neptune and Plotly.

Notes and comments

You can add notes and comments to your Runs by using the description argument of the neptune.init():

run = neptune.init(
description='neural net trained on Fashion-MNIST with high LR and low dropout'
)

Run description appears in the All metadata > sys section of the Run UI and it can also be added as a column to the Runs Table.

You can also edit the description directly in the Neptune UI.

Tags

You can add tags to your Runs to organize them in a meaningful way.

run = neptune.init(tags=['maskRCNN', 'finetune', 'prod_v2.0.1'])

Tags are a convenient way to organize or group runs. They appear in the All metadata > sys section of the Run UI and can be also be added as columns to the Runs Table.

You can also edit tags directly in the Neptune UI.

Text

You can log textual information to the Run by using:

# Log single String
run['aux/text'] = 'text I keep track of, like query or tokenized word'
# Log series of String to one log
for epoch in range(epochs_nr):
token = str(...)
run['train/tokens'].log(token)

You will have it in the All metadata section of the Run UI, where you can browse and download it.

A single line of text log is limited to 1000 characters but the number of lines is not limited. To log longer texts split your text into lines of under 1000 characters.

Video

Neptune currently supports MP4, OGG, OGV, MOV, M4V, MKV, and WEBM formats for video.

You can log video files and watch them in the Neptune UI.

# Log video file from disk
run['video'] = neptune.types.File('/path/to/video-file.mp4')

As a result, a video player is rendered in the All metadata section of the Run UI, where you can watch, open in full screen and download it.

Audio

Neptune currently supports MP3, M4A, OGA, and WAVE formats for audio.

Log audio files and listen to them right in the UI.

# Log audio file from disk
run['audio'] = neptune.types.File('/path/to/audio-file.mp3')

As a result, an audio player is rendered in the All metadata section of the Run UI, where you can watch, open in full screen and download it.

Tables

When you log tabular data, such as CSV files or Pandas DataFrame, Neptune will display it as a table.

CSV files

You can log CSV files and have them displayed as a table in the Neptune UI.

# CSV file
run['test/preds'].upload('path/to/test_preds.csv')

A CSV file is displayed in the All metadata section of the Run UI.

Pandas DataFrame

You can log Pandas DataFrames and have them displayed as a table in the Neptune UI.

from neptune.new.types import File
# Pandas DataFrame
run['data/sample'].upload(File.as_html(sample_df))

A Pandas DataFrame is displayed in the All metadata section of the Run UI.

What's next?