Downloading runs data

Almost all data that is logged to the project (runs and notebooks) can be downloaded to the local machine. You may want to do it for a variety of reasons:

  • Build custom analysis or visualizations using runs data.

  • Use saved model checkpoint elsewhere.

  • Get source code of a run and execute it again.

  • Build report that uses data across projects.

  • Archive old project.

There are three ways to download data from Neptune:

  1. Programmatically, by using neptune-client: for example downloading runs dashboard as pandas DataFrame. Check download programmatically below.

  2. Directly from the UI: for example downloading notebook checkpoint or runs dashboard as csv. Check downloading from Neptune UI below.

  3. From the JupyterLab interface: Check downloading checkpoint documentation.

Downloading programmatically

You can download run's data programmatically.

All the data fetching calls are synchronous (blocking) and cause calls to the Neptune server.

Download runs dashboard as DataFrame

To download runs table as pandas DataFrame use fetch_runs_table.

import neptune.new as neptune
my_project = neptune.get_project("my_workspace/my_project")
# Get dashboard with runs contributed by 'sophia'
sophia_df = my_project.fetch_runs_table(owner='sophia').to_pandas()
# Get another dashboard with runs tagged 'cycleLR'
cycleLR_df = my_project.fetch_runs_table(tag='cycleLR').to_pandas()
cycleLR_df.head()

sophia_df and cycleLR_df are pandas DataFrames, where each row is a run and columns represent tracked metadata such as metrics, text logs and parameters. For metrics, the latest value is returned.

Example dataframe will look like this:

To download only runs that you want, you can filter them by:

  • id

  • state

  • owner

  • tag

Downloading from the run

All the data fetching calls are synchronous (blocking) and cause calls to the Neptune server.

You can fetch the run's data in a similar way you track the data. Look at the following example:

run["parameters/batch_size"].get() # returns single value e.g. 5
run["train/accuracy"].get_last() # return the last accuracy value
run["trained_model"].download() # downloads model file to the current directory

Namespaces

Run Fields can be grouped into namespaces to organize them by type, use case, phase etc. You can access a namespace the same way you would access any other field by specifying its path:

# Instead of using full path
batch_size = run["parameters/batch_size"].get()
# You can access a namespace and then a specific field within that namespace
parameters = run["parameters"]
batch_size = parameters["batch_size"].get()

Remember that namespaces can be nested e.g. run["parameters/preprocessing/resampling_size"]

Accessing Atom fields

You can access most of the atom fields by using .get():

# Numerical fields
batch_size = run["parameters/batch_size"].get()
# Strings
username = run["sys/owner"].get()
# Datetime
last_updated = run["sys/modification_time"].get()

Download files stored in File fields by using .download():

# Download example_image to the current directory
run["data/example_image.png"].download()
# Download model to the specified directory
run["trained_model.h5"].download(destination_path)

Accessing Series fields

You can access numerical and text Series by using .get_last():

# Accessing FloatSeries
final_loss = run["train/loss"].get_last()
# Accessing StringSeries
last_stderr_line = run["sys/stderr"].get_last()

Accessing tags

Tags fields can be accessed by using .get() method that returns set of str objects:

run_tags = run["sys/tags"].get()
if "exploration" in run_tags:
print_analysis()

Accessing Files fields

You can download all files stored in a Files field by using .download() method:

# Download files to the current directory
run["misclasified_samples"].download()
# You can also download to a specific path
run["source_code/files"].download(path_to_dir)

Downloading from Neptune UI

You can download runs data directly from the UI to your local machine. Check downloading from the UI documentation page for details.

Downloading from Jupyter Notebook

You can download notebook checkpoint directly from Neptune to the Jupyter or JupyterLab interface. Check downloading checkpoint documentation for details.