Neptune supports log and display for many different types of model building metadata.

On this page, you can see how each ML metadata type can be logged via Neptune API and how it will be displayed in the Neptune UI.

Supported ML metadata types include:

• ​Metrics and losses​

• ​Parameters and Model Configuration​

• ​Model Checkpoints​

• ​Code and Git information​

• ​Data Versions​

• ​Hardware Consumption​

• ​Files​

• ​Console logs​

• ​Images​

• ​Interactive visualizations​

• ​Notes and comments​

• ​Tags​

• ​Text​

• ​Video​

• ​Audio​

• ​Tables​

import neptune.new as neptune
​
run = neptune.init(api_token="YOUR_API_TOKEN",
                   project="YOUR_WORKSPACE/YOUR_PROJECT")

Metrics and losses

Log metrics and losses to Neptune using =and .log() methods:

# logging score (single value)
run['score'] = 0.97
run['test/acc'] = 0.97
​
# logging metric (series of values)
for epoch in range(100):
    acc = ...
    loss = ...
    metric = ...
​
    run["train/accuracy"].log(acc)
    run["train/loss"].log(loss)
    run['metric'].log(metric)

Metric can be accuracy, MSE, or any numerical value. All metrics are visualized as Charts in the Run UI. You can also check grouped values in the All Metadata section of the Run UI and add metrics as columns to the Runs Table.

Parameters and Model Configuration

Log parameters one-by-one to Fields of a Run.

run['parameters/epoch_nr'] = 5
run['parameters/batch_size'] = 32
run['parameters/dense'] = 512
run['parameters/optimizer'] = 'sgd'
run['parameters/metrics'] = ['accuracy', 'mae']
run['parameters/activation'] = 'relu'

Define parameters as Python dictionary and log them all at once to a Namespace of a Run.

# Define parameters
PARAMS = {'epoch_nr': 5,
          'batch_size': 32,
          'dense': 512,
          'optimizer': 'sgd',
          'metrics': ['accuracy', 'binary_accuracy'],
          'activation': 'relu'}
​
# Pass parameters
run['parameters'] = PARAMS

Logged parameters appear in the All metadata section of a single Run UI:

You can also add parameters as columns to the Runs Table:

Model checkpoints

You can log model checkpoints to a Run.

# Log PyTorch model weights
my_model = ...
torch.save(my_model, 'my_model.pt')
run['model_checkpoints/my_model'].upload('model_checkpoints/my_model.pt')

You can save model weights from any deep learning framework. Model checkpoints will appear in the All metadata section of the Run UI, for example in the ‘model_checkpoints’ Namespace.

Code

Neptune supports code versioning in three different ways.

Git information

If you start a Run from a directory that is a part of the git repo, Neptune will automatically find the .git directory and log information from it.

It creates a summary in the source_code/git section of the Run UI with the following Fields:

• Dirty: whether the commit was dirty or not

• Commit message: what was the commit message

• Commit ID: Git SHA signature

• Commit author: the author of the last commit

• Commit date: the data of the last commit

• Current branch: what is the current git branch

• Remotes: list of git remotes your local .git is connected to

Code Snapshot

Neptune automatically snapshots code when you create a Run with neptune.init().

By default, it will only save the entry point file (e.g.main.py if you run python main.py), but you can pass a list of files or wildcard patterns ("glob") like: **/*.py to specify more files:

# Snapshot model.py and prep_data.py
run = neptune.init(...,
                   source_files=["model.py", "prep_data.py"])
​
# Snapshot all python files and 'config.yaml' file
run = neptune.init(...,
                   source_files=["**/*.py", "config.yaml"])

You will have all the files in the Source code section of the Run UI.

Neptune also logs the entry point filename so that you know which script was actually executed.

Notebook Code Snapshot

Neptune can automatically snapshot your Jupyter notebook checkpoint every time you create a Run in that notebook.

To get started, see Integration with Jupyter Notebook and Jupyter Lab. ​

Data versions

Neptune support many ways to log and display metadata about datasets.

User-defined fields

You can use Namespaces and basic Logging methods to organize any ML metadata in Neptune.

Typically, people log the following metadata about datasets:

• the md5 hash of the dataset

• the location of the dataset

• a list of classes

• a list of feature names (for structured data problems)

For example:

# Prepare dataset
(train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.fashion_mnist.load_data()
train_images = train_images / 255.0
test_images = test_images / 255.0
​
# Log data version as a Run property
run['train/images_version'] = hashlib.md5(train_images).hexdigest()
run['test/images_version'] = hashlib.md5(test_images).hexdigest()

Go to All metadata section of the Run UI and find your data versions and other dataset metadata.

You can also add data versions as columns in the Runs Table UI to group and compare Runs trained on the same versions of the dataset.

Data Version Control (DVC)

Neptune displays DVC files in the UI, you just need to specify which DVC files you would like to log.

For example, to log all the DVC files from your working directory and all subdirectories pass "**/*.dvc" to the source_files argument of neptune.init():

# Snapshot all .dvc files from any directory
run = neptune.init(...,
                   source_files=["**/*.dvc"])

Hardware consumption

pip install psutil

Automatically monitor hardware utilization for your Runs:

• cpu: total CPU consumption of your machine

• memory: total memory consumption of your machine

• gpu: GPU consumption of your machine

• gpu_memory: GPU memory consumption of your machine

• stderr: standard error logs from your console

• stdout: standard output logs from your console

All that information is visualized in the Monitoring section of the Run UI.

# Turn off hardware monitoring
run = neptune.init(...,
                   capture_hardware_metrics=False)

# Assign custom name to the monitoring namespace
run = neptune.init(...,
                   monitoring_namespace='worker-7-monitoring')

Files

You can log any file or directory you want to Neptune by using .upload()and .upload_files()methods:

# Log file
run['aux/data'].upload('auxiliary-data.zip')
​
# Save multiple files under a path
run["config_files"].upload_files([path_to_config_1, path_to_config_2])
​
# You can also use wildcard patterns ("glob") with .upload_files()
run["preprocessing_scripts"].upload_files("./preprocessing/*.py")
​

You can browse and download files in the All metadata section of the Run UI.

Files are displayed in the UI based on the file format:

• PNG, JPG, JPEG, GIF image formats are displayed as Images​

• HTML files are displayed as Interactive visualizations​

• MP4, OGG, OGV, MOV, M4V, MKV, WEBM are displayed as players for Video​

• MP3, M4A, OGA, WAVE are displayed as players for Audio​

• CSV files are displayed as Tables​

• IPYNB files are displayed as fully rendered Notebooks

• Text files and source code files are displayed as text with appropriate syntax coloring if applicable

run.pop('aux/data')

Console logs

stdout and stderr

pip install psutil

Automatically save console logs for your Runs :

• stderr: standard error logs from your console

• stdout: standard output logs from your console

All that information is visualized in the Monitoring section of the Run UI.

# Turn off saving console logs
run = neptune.init(..., 
                   capture_stdout=False, 
                   capture_stderr=False,)

Python's Logger

You can also capture Python's Logger logs. Simply create and add Neptune handler to your Logger object:

import logging
from neptune.new.integrations.python_logger import NeptuneHandler
​
logger = logging.getLogger('my_python_logger')
​
logger.addHandler(NeptuneHandler(run=run))
​
logger.debug('Starting training')

Images

You can log either a single image or a series of images to Neptune.

Several data formats are available:

• ​Files with the image format like PNG, JPEG, JPG, GIF,​

• ​Matplotlib figure,​

• ​PIL image,​

• ​NumPy array​

• ​TensorFlow Tensor​

• ​Pytorch Tensor​

In all cases, you will have images in the All metadata section of the Run UI (under Namespace defined by you), where you can browse and download them.

To log images use .upload() or .log(File) methods:

from neptune.new.types import File
​
# single image
run['train/bounding_boxes'].upload('bboxes.png')
​
# for series of images
for i in range(valid_nr):
    ith_path = ...
    run['valid/misclassified'].log(File(ith_path))

Image file

You can log image format files directly from the disk.

Log single image from disk with the .upload() method:

run['train/bounding_boxes'].upload('bboxes.png')

Log series of images in a for loop with the .log(File) method:

from neptune.new.types import File
​
for name in misclassified_images_names:
    y_pred = ...
    y_true = ...
    run['misclassified_images'].log(File('misclassified_image.png'))

Matplotlib figure

Log Matplotlib figure (matplotlib.figure.Figure) as an image:

# Import matplotlib
import matplotlib.pyplot as plt
​
# Generate figure
fig = plt.figure(figsize=(7, 9))
...
​
# Log figure to run
run['matplotlib-fig'].upload(fig)

You can also log series of Matplotlib figures:

for epoch in range(params['iterations']):
    plt_fig = get_histogram()
    run['train/distribution'].log(plt_fig)

You will have Matplotlib figures displayed in the All metadata section of the Run UI, where you can browse and download them.

For more information, see How to use Neptune and Matplotlib.

PIL Image

You can log the PIL image directly from the memory:

image1 = Image.open('Representation-learning.jpg')
image2 = Image.open('RL-agents-1.jpg')
​
# Log image to run
run['representation_learning'].upload(image1)
run['reinforcement_learning'].upload(image2)

You can also log series of PIL images:

from neptune.new.types import File
​
for epoch in range(images_nr):
    pil_image = Image.open('path_to_image')
    run['train/distribution'].log(File(pil_image))

You will have images in the All metadata section of the Run UI, where you can browse and download them.

NumPy array

You can log NumPy arrays (2d or 3d) directly from the memory, and have them visualized as an image:

import numpy as np
from neptune.new.types import File
​
im_array = np.random.rand(100, 100, 3) * 255
run['np_image'].upload(File.as_image(im_array))

You can also log series of NumPy arrays:

from neptune.new.types import File
​
for epoch in range(data):
    im_array = ...
    run['train/distribution'].log(File.as_image(im_array))

You will have NumPy images in the All metadata section of the Run UI, where you can browse and download them.

TensorFlow tensor

You can log TensorFlow Tensors (2d or 3d) directly from the memory, and have them visualized as an image:

import tensorflow as tf
from neptune.new.types import File
​
tf_tensor = tensorflow.random.uniform(shape=[10, 20, 3])
run['tf_image'].upload(File.as_image(tf_tensorrray))

You can also log series of TensorFlow Tensors:

from neptune.new.types import File
​
for epoch in range(data):
    tf_tensor = ...
    run['train/distribution'].log(File.as_image(tf_tensor))

You will have TensorFlow Tensor images in the All metadata section of the Run UI, where you can browse and download them.

PyTorch tensor

You can log PyTorch Tensors (2d or 3d) directly from the memory, and have them visualized as an image:

import tensorflow as tf
from neptune.new.types import File
​
torch_tensor = torch.rand(30, 30)
run['torch_tensor'].upload(File.as_image(torch_tensor))

You can also log series of PyTorch Tensors:

from neptune.new.types import File
​
for epoch in range(data):
    torch_tensor = ...
    run['train/distribution'].log(File.as_image(torch_tensor))

You will have PyTorch Tensor images in the All metadata section of the Run UI, where you can browse and download them.

Interactive visualizations

You can log interactive charts and they will be rendered interactively in the Neptune UI. You will find it in the All metadata section of the Run UI.

Common visualization libraries are supported:

• ​HTML file​

• ​Matplotlib figure -> we turn it interactive automatically

• ​Altair chart​

• ​Bokeh plots​

• ​Plotly figure​

HTML file

You can log any HTML file, and it will be rendered in the UI.

run['html'] = neptune.types.File('button_example.html')

HTML will appear in the All metadata section of the Run UI.

Matplotlib figure

You can log Matplotlib figure (matplotlib.figure.Figure) as an interactive visualization, by using neptune.types.File.as_html():

from neptune.new.types import File
​
fig = ...
run['visuals/matplotlib-fig'] = File.as_html(fig)

The interactive chart will appear in the All metadata section of the Run UI, where you can explore, open in full-screen and download it.

For more information, see How to use Neptune and Matplotlib.

Altair chart

You can log Altair charts as interactive visualizations, by using neptune.types.File.as_html():

from neptune.new.types import File
​
fig = ...
run['visuals/altair-fig'] = File.as_html(fig)

An interactive Altair chart will appear in the All metadata section of the Run UI, where you can explore, open in full screen and download it.

Bokeh plot

You can log Bokeh plots as interactive visualizations, by using neptune.types.File.as_html():

from neptune.new.types import File
​
fig = ...
run['visuals/bokeh-fig'] = File.as_html(fig)

An interactive Bokeh plot will appear in the All metadata section of the Run UI, where you can explore, open in full-screen and download it.

Plotly figure

You can log Plotly figures as interactive visualizations, by using neptune.types.File.as_html():

from neptune.new.types import File
​
fig = ...
run['visuals/plotly-fig'] = File.as_html(fig)

An interactive Plotly figure will appear in the All metadata section of the Run UI, where you can explore, open in full-screen and download it.

For more information, see How to use Neptune and Plotly.

Notes and comments

You can add notes and comments to your Runs by using the description argument of the neptune.init():

run = neptune.init(
    description='neural net trained on Fashion-MNIST with high LR and low dropout'
    )

Run description appears in the All metadata > sys section of the Run UI and it can also be added as a column to the Runs Table.

Tags

You can add tags to your Runs to organize them in a meaningful way.

run = neptune.init(tags=['maskRCNN', 'finetune', 'prod_v2.0.1'])

Tags are a convenient way to organize or group runs. They appear in the All metadata > sys section of the Run UI and can be also be added as columns to the Runs Table.

Text

You can log textual information to the Run by using:

# Log single String
run['aux/text'] = 'text I keep track of, like query or tokenized word'
​
# Log series of String to one log
for epoch in range(epochs_nr):
    token = str(...)
    run['train/tokens'].log(token)

You will have it in the All metadata section of the Run UI, where you can browse and download it.

Video

You can log video files and watch them in the Neptune UI.

# Log video file from disk
run['video'] = neptune.types.File('/path/to/video-file.mp4')

As a result, a video player is rendered in the All metadata section of the Run UI, where you can watch, open in full screen and download it.

Audio

Log audio files and listen to them right in the UI.

# Log audio file from disk
run['audio'] = neptune.types.File('/path/to/audio-file.mp3')

As a result, an audio player is rendered in the All metadata section of the Run UI, where you can watch, open in full screen and download it.

Tables

When you log tabular data, such as CSV files or Pandas DataFrame, Neptune will display it as a table.

CSV files

You can log CSV files and have them displayed as a table in the Neptune UI.

# CSV file
run['test/preds'].upload('path/to/test_preds.csv')

A CSV file is displayed in the All metadata section of the Run UI.

Pandas DataFrame

You can log Pandas DataFrames and have them displayed as a table in the Neptune UI.

from neptune.new.types import File
​
# Pandas DataFrame 
run['data/sample'].upload(File.as_html(sample_df))

A Pandas DataFrame is displayed in the All metadata section of the Run UI.

What's next?