Querying and downloading metadata

Almost all ML metadata that is logged to Runs in the Project can be downloaded to the local machine. You may want to do it for a variety of reasons:
Build custom analysis or visualizations using Runs data
Use saved model checkpoint elsewhere
Get the source code of a run and execute it again
Build a report that uses data across projects
Archive old project
There are two ways to download ML metadata from Neptune:
1.​Programmatically, by using neptune-client: for example downloading runs dashboard as pandas DataFrame. 
2.​Directly from the UI: for example, downloading notebook checkpoint or runs dashboard as CSV. 
Querying ML metadata via Neptune API
Download the Runs table as a Pandas DataFrame
To download runs table as pandas DataFrame use .fetch_runs_table().
1
import neptune.new as neptune
2
​
3
project = neptune.get_project(
4
    name='<YOUR_WORKSPACE/YOUR_PROJECT>', 
5
    api_token='<YOUR_API_TOKEN>')
6
​
7
# Get dashboard with runs contributed by 'sophia' tagged 'cycleLR'
8
run_table_df = project.fetch_runs_table(
9
    owner='sophia',
10
    tag='cycleLR').to_pandas()
11
​
12
run_table_df.head()
13
​
Copied!
run_table_df is a Pandas DataFrame, where each row is a Run and columns represent logged metadata such as metrics, text logs, and parameters. 
To download only runs that you want, you can filter them by: id, state, owner, tag
Example data frame will look like this:
.fetch_runs_table()returns shallow information about the Runs. You can think of it as a data frame version of the Runs Table that you see in the web application. 
If you want to access all values of a series or download files logged to a Run read section Downloading single run data.​
Downloading single Run data
You can fetch the Run metadata in a similar way you logged it. 

First, instantiate an existing run:
1
import neptune.new as neptune
2
​
3
run = neptune.init(
4
    project='<YOUR_WORKSPACE/YOUR_PROJECT>',
5
    api_token='<YOUR_API_TOKEN>',
6
    run='<RUN_ID>' # for example 'SAN-123'
7
    )
8
    
9
# query a field
10
epoch_nr = run['epoch_nr'].fetch() 
Copied!
If you are only fetching the Run metadata you may instantiate the existing run in Read-only mode. This way no accidental modification of the metadata will be possible.
Querying single value Fields
You can access the Run ID and most of the single value Fields by using .fetch():
1
# Numerical fields
2
batch_size = run['parameters/batch_size'].fetch() 
3
​
4
# Strings
5
username = run['sys/owner'].fetch()
6
​
7
# Datetime
8
last_updated = run['sys/modification_time'].fetch()
9
​
10
# Run ID
11
my_run_id = run["sys/id"].fetch()
Copied!
Tags Fields can be accessed by using .fetch()method that returns a set of str objects:
1
run_tags = run['sys/tags'].fetch()
2
​
3
if 'exploration' in run_tags:
4
    print_analysis()
Copied!
Querying Series Fields
You can access numerical and text Series by using .fetch_last():
1
# Accessing FloatSeries
2
final_loss = run['train/loss'].fetch_last()
3
​
4
# Accessing StringSeries
5
last_stderr_line = run['sys/stderr'].fetch_last()
Copied!
You can also retrieve all the values and their indexes as a Pandas DataFrame by using .fetch_values():
1
# Accessing all values and their indexes as a Pandas DataFrame
2
loss_df = run['train/loss'].fetch_values()
3
​
4
# Don't include timestampe
5
loss_df = run['train/loss'].fetch_values(include_timestamp=False)
Copied!
Querying Artifact hash
You can get a hash of an Artifact field by using .fetch_hash():
1
# Query Artifact Hash (md5) 
2
md5 = run['dataset/train'].fetch_hash()
Copied!
To use artifacts you need at least 0.10.10 version of neptune client.    
1
pip install neptune-client>=0.10.10
Copied!
Downloading from single File Fields
Download files stored in File fields by using .download():
1
# Download example_image to the current directory
2
run["data/example_image"].download()
3
​
4
# Download model to the specified directory
5
run["trained_model"].download(destination_path)
Copied!
Downloading from FileSeries Fields
Similarly, for FileSeries fields you can download the last file to the disk with .download_last() or download all files with .download():
1
# Download last file in the FileSeries
2
run['train/epoch/histogram'].download_last()
3
​
4
# Download all files in the FileSeries
5
run['train/epoch/histogram'].download()
Copied!
You can query a Run namespace the same way you would access any other Field by specifying its path:
1
# Instead of using full path
2
batch_size = run['parameters/batch_size'].fetch()
3
​
4
# You can access a namespace and then a specific field within that namespace
5
parameters = run['parameters']
6
batch_size = parameters['batch_size'].fetch()
Copied!
Downloading Artifacts
You can download files from artifact fields by using the .download() method:
1
# Download all files from the artifact
2
run['datasets/train'].download(destination="./datasets")
Copied!
You can only download all files that were saved as a Neptune artifact.

To use artifacts you need at least 0.10.10 version of neptune client.    
1
pip install neptune-client>=0.10.10
Copied!
Field types
Downloading ML metadata from Neptune UI
When in Neptune, simply look for the download icon. It’s located in multiple places in the application and allows you to download various parts of the runs or notebooks. Below are some examples:
Model checkpoints
Export Runs table as CSV
Notebook checkpoint
Run source code
Images in full size
What's next?
Logging metadata
How-to guides
Examples
Integrations and Supported Tools
Field types

You should know - Previous

Organizing and filtering Runs

Next - You should know

Connection modes

Last modified 1mo ago

Export as PDF

Copy link

Contents

Querying ML metadata via Neptune API

Download the Runs table as a Pandas DataFrame

Downloading single Run data

Downloading ML metadata from Neptune UI

What's next?