Logging run data

During machine learning experimentation, you need to keep track of many different types of meta-data. Neptune helps you do that by logging, keeping track of and visualizing it.

You can track many different types of data to the run. It can be metric, loss, image, interactive visualization, model checkpoint, pandas DataFrame and many more.

Simply check what you can log section below for a complete list.

Basics of logging

Logging runs data to Neptune is simple and straightforward.

Let’s create a minimal code snippet that logs a single value to the run: 'acc'=0.95.

import neptune.new as neptune
# Set project and create run
run = neptune.init(project='my_workspace/my_project')
# Log 'acc' value 0.95
run['acc'] = 0.95

Above snippet sets project, creates run and log one value to it. As a result you have new run in Neptune with one value in one metric (‘acc’=0.95).

Run is created when run = neptune.init(project='my_workspace/my_project') is evaluated and run is your handle.

Runs can be viewed as dictionary-like structures - namespaces - that you can define in your code. You can apply hierarchical structure to your metadata that will be reflected in the UI as well. Thanks to this you can easily organize your metadata in a way you feel is most convenient.

What objects can you log to Neptune

Neptune supports logging many different types of data. Here, you can find all of them listed and described.

Remember to set project before you start logging:

run = neptune.init(project='my_workspace/my_project')

Remember to set project before you start logging:

run = neptune.init(project='my_workspace/my_project')

Metrics

Log metric to neptune using

# logging score (single value)
run['score'] = 0.97
run['test/acc'] = 0.97
# logging metric (series of values)
for epoch in range(100):
acc = ...
loss = ...
metric = ...
run["train/accuracy"].log(acc)
run["train/loss"].log(loss)
run['metric'].log(metric)

Generic recipe being:

# single value
run['some/structure'] = val
# for series of values
run['some/structure'].log(val)

Metric can be accuracy, MSE or any numerical value. All metrics are visualized as charts in the run. You can also check grouped values in the all metadata section.

You can create as many metrics as you wish.

back to top

Parameters

Define parameters and log them one-by-one

# Define parameters
EPOCH_NR = 5
BATCH_SIZE = 32
DENSE = 512
OPTIMIZER = 'sgd'
METRICS = ['accuracy', 'mae']
ACTIVATION = 'relu'
# Pass parameters
run['parameters/epoch_nr'] = EPOCH_NR
run['parameters/batch_size'] = BATCH_SIZE
run['parameters/dense'] = DENSE
run['parameters/optimizer'] = OPTIMIZER
run['parameters/metrics'] = METRICS
run['parameters/activation'] = ACTIVATION

Check this example in Neptune.

Define parameters as Python dictionary and log them all at once

# Define parameters
PARAMS = {'epoch_nr': 5,
'batch_size': 32,
'dense': 512,
'optimizer': 'sgd',
'metrics': ['accuracy', 'binary_accuracy'],
'activation': 'relu'}
# Pass parameters
run['parameters'] = PARAMS

Check this example in Neptune.

Parameters in the run view
Parameters in the dashboard

You can use them later to analyse or compare runs. They are displayed in the parameters section of the run. Moreover every parameter can be displayed as a column on the runs dashboard.

back to top

Code

Neptune supports code versioning. There are a few ways to do that.

Track your git information

If you start a run from a directory that is a part of the git repo, Neptune will automatically find the .git directory and log some information from it.

It creates a summary in the details section with following metadata:

  • commit

    • authorEmail

    • authorName

    • commitDate

    • commitId

    • message

  • currentBranch

  • remotes

  • repositoryDirty

back to top

Code Snapshot

Neptune also automatically snapshots code when you create a run with neptune.init().

By default, it will only save the entry point file (e.g.main.py if you run python main.py), but you can pass a list of files or wildcard patterns ("glob") like: *.py to specify more files:

# Snapshot model.py and prep_data.py
run = neptune.init(source_files=["model.py", "prep_data.py"])
# Snapshot all python files and 'config.yaml' file
run = neptune.init(source_files=["*.py", "config.yaml"])

You will have all sources in the source code section of the run. Neptune also logs the entrypoint file so that you have all the information about the run sources.

When using pattern expansion, such as '*.py', make sure that your expression does not log too many files or non-source code files. For example, using '*' as a pattern will upload all files and directories from the cwd. It may result in logging files that you did not want to upload and to clutter your storage.

back to top

Notebook Code Snapshot

Neptune auto-snapshots your notebook every time you create run in that notebook.

Another option to log notebook checkpoint is by clicking a button in the Jupyter or JupyterLab UI. It is useful to log notebook with EDA or manual model analysis.

To get started, install notebook extension, then go to the Keeping track of Jupyter Notebooks guide that will explain everything.

back to top

Images

Log images to Neptune. You can log either single image or series of them. Several data formats are available:

In all cases you will have images in the all metadata section of the run (under hierarchy defined by you), where you can browse and download them.

You can log unlimited number of images either in the single log or in the multiple image logs. Simply use:

import neptune.new as neptune
from neptune.new.types import File
run = neptune.init(project='my_workspace/my_project')
# single image
run['train/bounding_boxes'].upload('bboxes.png')
# for series of images
for i in range(valid_nr):
ith_path = ...
run['valid/misclassified'].log(File(ith_path))

back to top

Image file

You can log image files directly from disk.

Log single image from disk.

run['train/bounding_boxes'].upload('bboxes.png')

Log series of images in for loop.

from neptune.new.types import File
for name in misclassified_images_names:
y_pred = ...
y_true = ...
run['misclassified_images'].log(File('misclassified_image.png'))

Notice the difference between upload() for single images and log() for series of images.

back to top

Matplotlib

Log Matplotlib figure (matplotlib.figure.Figure) as an image:

# Import matplotlib
import matplotlib.pyplot as plt
# Generate figure
fig = plt.figure(figsize=(7, 9))
...
# Log figure to run
run['matplotlib-fig'].upload(fig)

You can also log series of Matplotlib figures:

for epoch in range(params['iterations']):
plt_fig = get_histogram()
run['train/distribution'].log(plt_fig)

You will have Matplotlib figure in the all metadata section of the run, where you can browse and download them.

back to top

PIL

Log PIL image right from the memory:

image1 = Image.open('Representation-learning.jpg')
image2 = Image.open('RL-agents-1.jpg')
# Log image to run
run['representation_learning'].upload(image1)
run['reinforcement_learning'].upload(image2)

You can also log series of PIL images:

from neptune.new.types import File
for epoch in range(images_nr):
pil_image = Image.open('path_to_image')
run['train/distribution'].log(File(pil_image))

You will have images in the all metadata section of the run, where you can browse and download them.

back to top

NumPy

Log NumPy array (2d or 3d) right from the memory, and have it visualized as image:

import neptune.new as neptune
import numpy as np
from neptune.new.types import File
im_array = np.random.rand(100, 100, 3) * 255
run['np_image'].upload(File.as_image(im_array))

You can also log series of NumPy arrays:

import neptune.new as neptune
from neptune.new.types import File
for epoch in range(data):
im_array = ...
run['train/distribution'].log(File.as_image(im_array))

You will have NumPy images in the all metadata section of the run, where you can browse and download them.

back to top

Interactive chart

You can log interactive charts and they will be rendered interactively in the "all metadata" section under base_namespace of your choice. Common visualization libraries are supported:

  • Matplotlib -> we turn it interactive automatically

  • Altair

  • Bokeh

  • Plotly

For a full screen view, you can open visualization in the new browser tab, by clicking on the “arrow-pointing-top-right” icon, located right above your visualization:

back to top

Matplotlib

Log Matplotlib figure (matplotlib.figure.Figure) as an interactive chart, by using neptune.types.File.as_html().

This option is tested with matplotlib==3.2.0 and plotly==4.12.0. Make sure that you have correct versions installed. See: plotly installation guide.

import neptune.new as neptune
from neptune.new.types import File
run = neptune.init(project='my_workspace/my_project')
fig = ...
run['visuals/matplotlib-fig'] = File.as_html(fig)

Interactive chart will appear in the all metadata section, under base_namespace of your choice (in the snippet above visuals/matplotlib-fig) where you can explore, open in full screen and download it.

Check images logging to see how to log matplotlib figure as an image.

back to top

Altair

Log Altair chart as an interactive chart, by using neptune.types.File.as_html().

import neptune.new as neptune
from neptune.new.types import File
run = neptune.init(project='my_workspace/my_project')
fig = ...
run['visuals/altair-fig'] = File.as_html(fig)

Interactive chart will appear in the all metadata section, under base_namespace of your choice (in the snippet above visuals/altair-fig) where you can explore, open in full screen and download it.

You need to install plotly to log Altair as interactive chart. See: plotly installation guide.

back to top

Bokeh

Log Bokeh chart as an interactive chart, by using neptune.types.File.as_html().

import neptune.new as neptune
from neptune.new.types import File
run = neptune.init(project='my_workspace/my_project')
fig = ...
run['visuals/bokeh-fig'] = File.as_html(fig)

Interactive chart will appear in the all metadata section, under base_namespace of your choice (in the snippet above visuals/bokeh-fig) where you can explore, open in full screen and download it.

You need to install plotly to log Bokeh as interactive chart. See: plotly installation guide.

back to top

Plotly

Log plotly chart as an interactive chart, by using neptune.types.File.as_html().

import neptune.new as neptune
from neptune.new.types import File
run = neptune.init(project='my_workspace/my_project')
fig = ...
run['visuals/plotly-fig'] = File.as_html(fig)

Interactive chart will appear in the all metadata section, under base_namespace of your choice (in the snippet above visuals/plotly-fig) where you can explore, open in full screen and download it.

You need to install plotly to enable this feature. See: plotly installation guide

back to top

Text

Log textual information to the run by using:

# Log single String
run['aux/text'] = 'text I keep track of, like query or tokenized word'
# Log series of String to one log
for epoch in range(epochs_nr):
token = str(...)
run['train/tokens'].log(token)

You will have it in the all metadata section of the run, where you can browse and download it.

Single line of text log is limited to 512 characters. At the same time number of lines is not limited.

back to top

Hardware consumption

Automatically monitor hardware utilization for your runs:

  • CPU (average of all cores),

  • memory,

  • for each GPU unit - memory usage and utilization.

All that information is visualized in the monitoring section. You can turn off this feature when you set project and create run:

# Turn off hardware monitoring
run = neptune.init(capture_hardware_metrics=False)

As a result hardware consumption is not being tracked.

You can assign custom namespace for the hardware monitoring logs. Default is "monitoring":

# Assign custom name to the monitoring namespace
run = neptune.init(monitoring_namespace='worker-7-monitoring')

To enable this feature you need to install psutil. Check our installation guide for more info.

If you see the Info (NVML): NVML Error: NVML Shared Library Not Found - GPU usage metrics may not be reported. your GPU consumption is not being logged.

It means that either:

  • there are no GPUs on your machine

  • your NVIDIA NVML library is not installed or configured properly. See how to install and configure NVML.

Logging GPU on Windows

On Windows, Neptune searches for the nvml.dll file in the standard locations:

  • C:\Program Files\NVIDIA Corporation\NVSMI\nvml.dll

  • C:\Windows\System32\nvml.dll

If you are having trouble logging GPU metrics on Windows, please check that your NVML installation is correct and that you have the nvml.dll file in either of those locations.

Alternatively, you can set a custom location of nvml.dll on Windows by setting the NVML_DLL_PATH environment variable.

back to top

Run information

To better describe a run you can use name, description and tags.

back to top

Run name

You can add name to the run when you set project and create it. Try to keep it short and descriptive.

run = neptune.init(name='Mask R-CNN with data-v2')

Run name appears in the all metadata, sys section and can be displayed as a column on the runs dashboard.

You can also create/edit name when the run is online:

run["sys/name"] = 'mask RCNN'

You can edit name directly on the run dashboard.

You can search for a run by name. Here is how: Searching and filtering runs.

back to top

Run description

You can add longer note to the run when you set project.

run = neptune.init(
description='neural net trained on Fashion-MNIST with high LR and low dropout'
)

Run description appears in the all metadata, sys section and can be displayed as a column on the runs dashboard.

You can also create/edit description when the run is online:

run['sys/description'] = 'neural net trained on Fashion-MNIST with high LR'

You can edit description directly on the run dashboard.

You can use info in the description to later search for a run in the UI. Here is how: Searching and filtering runs.

back to top

Run tags

You can add tags to the run when you set project.

run = neptune.init(tags=['maskRCNN', 'finetune', 'prod_v2.0.1'])

Tags are a convenient way to organize or group runs. They appear in the all metadata, sys section and can be displayed as a column on the runs dashboard.

You can also append tags when the run is online:

run["sys/tags"].add(['prod_v1.9.0', 'keras'])

You can edit tags directly on the run dashboard.

You can quickly filter by tag by clicking on it in the runs dashboard. Check Searching and filtering runs guide for more options.

back to top

Data versions

Log data version or dataset hash to Neptune.

# Prepare dataset
(train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.fashion_mnist.load_data()
train_images = train_images / 255.0
test_images = test_images / 255.0
# Log data version as run property
run['train/images_version'] = hashlib.md5(train_images).hexdigest()
run['test/images_version'] = hashlib.md5(test_images).hexdigest()

In this way you can keep track on what data given model was trained. Data version will appear in the all metadata section (under the hierarchy set by you) and can be displayed as a column on the run dashboard.

back to top

Files

You can log any file or directory you want by using for a specified path .upload(file_path) to log single files and .upload_files(file_path) to log multiple files. This includes model_checkpoint, csv, binaries, or anything else.

# Log file
run['aux/data'].upload('auxiliary-data.zip')
# Save multiple files under a path
run["config_files"].upload_files([path_to_config_1, path_to_config_2])
# You can also use wildcard patterns ("glob") with .upload_files()
run["preprocessing_scripts"].upload_files("./preprocessing/*.py")

You can browse and download files in the all metadata section of the run.

Make sure that you define the correct path to files that you want to upload when using wildcards as it may result in the unintended logging of a large amount of data and cluttering your storage.

You can delete files from run:

run.pop('aux/data')

back to top

Model checkpoints

Log model checkpoints to run.

# Log PyTorch model weights
my_model = ...
torch.save(my_model, 'my_model.pt')
run['model_checkpoints/my_model'].upload('model_checkpoints/my_model.pt')

This technique lets you save model from any deep learning framework. Model checkpoint will appear in the all metadata section, for example in the ‘model_checkpoints’ namespace: example checkpoints.

back to top

HTML objects

Log HTML files , and have them rendered in the UI

run['html'] = neptune.types.File('button_example.html')

HTML will appear in the all metadata (namespace defined by you) section.

back to top

Video

Log video files and watch them right in the UI.

# Log video file from disk
run['video'] = neptune.types.File('/path/to/video-file.mp4')

As a result, video player is rendered in the UI under video name, where you can watch, open in full screen and download it.

back to top

Audio

Log audio files and listen to them right in the UI.

# Log audio file from disk
run['audio'] = neptune.types.File('/path/to/audio-file.mp3')

As a result, audio player is rendered in the UI under audio name, where you can listen to it and download it.

back to top

Tables

When you log tabular data, such as csv or DataFrame, Neptune will display it as table.

back to top

csv

Log csv file and have it visualized as table.

# Pandas DataFrame
run['test/preds'].upload('path/to/test_preds.csv')

csv file is displayed in the all metadata section under namespace specified by the user.

back to top

Logging with integrations

Besides logging using Neptune Python library, you can also use integrations that let you log relevant data with almost no code changes. Have a look at Integrations page for more information or find your favorite library:

Advanced

Pass Run object around to log from multiple Python files

When you create a run object:

run = neptune.init(project='my_workspace/my_project')

You can pass it around and use it to populate functions’ parameters and perform logging from multiple Python files.

Let’s create a recipe for that:

main.py

# Import from file shown below
from utils import log_images_epoch
# Create run in project
run = neptune.init(project='my_workspace/my_project')
# Log metrics in the same file
run['train/acc'] = 0.95
run['train/f1'] = 0.65
# Log by using imported function, pass 'run'
log_images_epoch(run=run)

utils.py

# 'run' is Neptune run
def log_images_epoch(run):
image1 = ...
image2 = ...
run['images/img_1'].upload(image1)
run['images/img_2'].upload(image2)

In this way you can work with larger codebase and use logging from multiple Python files.

Logging to multiple runs in one script

You can freely create multiple runs in the single script and log to them separately. A general recipe is very straightforward, as you simply create multiple run objects.

Create three runs and log metric to each separately:

import neptune.new as neptune
# Start 1st run
my_run1 = neptune.init(project='common/showroom',
name='logging-to-multipe-runs',
tags=['do-not-remove', 'multiple'])
my_run1['params'] = {'lr': 0.007}
# Start 2nd run
my_run2 = neptune.init(project='common/showroom',
name='logging-to-multipe-runs',
tags=['do-not-remove', 'multiple'])
my_run2['params'] = {'lr': 0.0034}
# Start 3rd run
my_run3 = neptune.init(project='common/showroom',
name='logging-to-multipe-runs',
tags=['do-not-remove', 'multiple'])
my_run3['params'] = {'lr': 0.005}
# Log metric to my_run1
for j in range(1, 101):
my_run1['train/loss'].log(0.7/j**0.5)
# Log metric to my_run2
for j in range(1, 101):
my_run2['train/loss'].log(0.8/j**0.5)
# Log metric to my_run3
for j in range(1, 101):
my_run3['train/loss'].log(0.9/j**0.5)

We log loss, by using the my_run1, my_run2 and my_run3. In this way you can log to many runs from the same Python script.