Skip to main content

Export data from Neptune 2.x

Use Neptune Exporter to migrate your data from Neptune 3.x to other platforms, such as MLflow or Weights & Biases.

Verify Neptune version​

To check which version of the Neptune app you are using:

  1. Log in to your Neptune workspace.
  2. Go to Help & Support → About Neptune.

Neptune app version 2.x

The version number should be 2.x.x.

Example workflows​

For detailed instructions, see the Neptune Exporter README file on GitHub.

Export an entire project​

To export all runs from the specified projects to your local storage, run:

uv run neptune-exporter export \
  -p "my-workspace/my-project-1" \
  -p "my-workspace/my-project-2" \
  --exporter neptune2 \
  --data-path ./exports/data \
  --files-path ./exports/files

This command saves the Parquet data to the ./exports/data directory and the files to ./exports/files.

note

To improve the export speed, you can:

  • Split the export by project or run across different processes
  • Export only a relevant subset of projects, runs, or attributes
  • Exclude files from the export

If you're exporting a large amount of data, we recommend exporting not just across different processes, but different machines altogether. This way, disk I/O is not a bottleneck.

Export specific runs​

To export only runs that match the specified pattern, use the -r flag:

uv run neptune-exporter export \
  -p "my-workspace/my-project" \
  --exporter neptune2 \
  -r "RUN-.*" \

where RUN-.* is a regular expression that matches all run IDs starting with RUN-.

The run ID refers to the auto-generated string stored in the sys/id field of a run.

Export specific attributes​

To export only fields that match the specified patterns, use the -a flag:

uv run neptune-exporter export \
  -p "my-workspace/my-project" \
  --exporter neptune2 \
  -a "metrics/accuracy" -a "metrics/loss" -a "config/.*"

Validate your export​

See what was exported by running the following command:

uv run neptune-exporter summary --data-path ./exports/data

This command returns the number of exported projects or runs, attribute types, and the basic step statistics.

Improve export performance​

Exporting large amounts of data can take a significant amount of time, especially for projects with many runs or large files. A recommended approach is to split the export into parallel jobs instead of exporting everything at once.

Common ways to split exports by:

  • Run creation date (time ranges)
  • Tags
  • Other Neptune Query Language (NQL) filters

Example: split exports by run creation date​

uv run neptune-exporter export \
  --exporter neptune2 \
  -p "workspace/proj" \
  --runs-query '`sys/creation_time`:datetime > "2024-02-06T05:00:00Z"'

To parallelize, you can run multiple commands at the same time, each covering a different date range, for example:

# Process 1
--runs-query '`sys/creation_time`:datetime >= "2024-01-01T00:00:00Z" AND `sys/creation_time`:datetime < "2024-02-01T00:00:00Z"'

# Process 2
--runs-query '`sys/creation_time`:datetime >= "2024-02-01T00:00:00Z" AND `sys/creation_time`:datetime < "2024-03-01T00:00:00Z"'

More filtering options​

See Neptune Query Language for more ways you can split your exports.

info

For detailed instructions, see the Neptune Exporter README file on GitHub.

Migrate to another tool​

You can migrate your exported data to another tool. See the following guides: