Skip to content

Upgrade to self-hosted Neptune 2.4#

Before you start#

Before upgrading, create a snapshot of any persistent volume or database that you provide for Neptune.

Effect on end users#

A typical upgrade takes from a few minutes minutes to an hour, depending on your internet connection speed.

Some upgrades require an upgrade of the client libraries (neptune-client). To learn more, see Compatibility with Neptune client packages.

Upgrades are often imperceptible to the end user (the data scientist running experiments). However, there's always a risk of Neptune being unavailable for some time.

This risk is mitigated on the Neptune client library side: If the Neptune server is unavailable, started runs switch to offline mode and the logged metadata is stored on the local disk. The data can be later manually uploaded using the neptune sync CLI command.

Upgrading to 2.4#

If you're upgrading from 2.3, contact support at support@neptune.ai for a migration script.

Neptune 2.4 introduces some technical changes:

  • Elasticsearch 8.x is supported in addition to 7.11+.
  • For object storage (to store data logged with Neptune), we introduce Azure Blob Storage as an experimental option.
  • Ansible is no longer required for fresh deployments. Instead, we offer a simplified installation procedure that involves deploying a K3s cluster and installing Neptune with a preset configuration.

Upgrading to 2.3.x#

Compared to version 2.2.x, version 2.3.x of Neptune does not involve significant technical changes.

  • Neptune 2.3 drops support of Elasticsearch 6.x and only supports 7.10+.
    • If you have specified your own Elasticsearch cluster in the configuration.yml file, make sure to upgrade to 7.10+ before upgrading Neptune to 2.3.
    • If you have not specified your own Elasticsearch cluster, you do not need to take any action.
  • We are gradually phasing out the wiki feature. If you were using the wiki and want to keep it enabled, you need to edit the configuration.yml file and manually set the wiki_enabled parameter to true.
    • If you upgrade with the wiki_enabled parameter set to false (default), your existing wiki data is not lost. It will be restored if you set the parameter to true.
  • Neptune 2.3 adds support for the 1.x version of the Neptune Python client library. For details, see Compatibility with client packages.

Upgrading to Elasticsearch 7.10+#

This is only applicable if you're using your own Elasticsearch service and not the one bundled within the Neptune installer. (The Neptune-managed cluster is upgraded automatically.)

  1. Upgrade the Elasticsearch cluster to version 7.10+.
  2. Proceed with upgrading Neptune to version 2.3.
I'm getting errors after upgrading

You may experience errors after the upgrade, such as the experiments table not showing.

Solution: Remove the entire index from Elasticsearch. Neptune will then rebuild it from the database.

It may take a few minutes up to a few hours, depending on the size of the project.

Upgrading Neptune#

A typical upgrade procedure is similar to a first-time installation.

  1. Perform a backup. This step is optional, but strongly recommended.

    In particular, this means creating a snapshot of any persistent volume or database that you provide for Neptune.

  2. Download the installer (neptune_installation_{version}.tgz) with the new version to a host where you will run the installer and unpack the archive.

  3. Execute the installation command. (See Installation.)

Note

The configuration file used previously contains your license, credentials for downloading Neptune images, your Neptune administrator credentials, among other things.

Do not change them without consulting with the neptune.ai team.

Upgrading to 2.2.x#

Version 2.2.x of Neptune comes with several important changes. Most notably, we're introducing S3 support for storing most of your data.

If you'd like to use S3 or S3-compatible object storage (strongly recommended), see the migration overview below. If you decide to go for it, contact us – we'd love to help!

We're also changing the requirements on the version of Elasticsearch. Neptune 2.2.x works with Elasticsearch 6.7+ and 7.10+. The next version of Neptune (2.3) drops support of Elasticsearch 6.x completely and supports 7.10+ exclusively.

Migrating to S3-compatible object storage#

Until version 2.1, most of the Neptune users' data was stored either on a persistent volume provided to Neptune via the storage_pvc_name option, or on an internal POSIX file system. If you wish to continue using the storage you're using, you can skip this section. For improved scalability and performance, however, we recommend switching to S3-compatible object storage, especially in cloud environments.

The data that needs migrating can be roughly categorized into two kinds: series data (numeric and text series) and files.

The migration happens in two phases:

  1. Migrating series data to S3. This phase requires downtime. The length of this downtime depends on how much series data there is and on the connection speed.
  2. Migrating files. This works in the background during normal operations of Neptune 2.2. The duration depends on the volume of data, but as it's a background process, it won't affect Neptune users.

Upgrading to Elasticsearch 7.10+#

This is only applicable if you're using your own Elasticsearch service (and not the one bundled within the Neptune installer).

Neptune 2.1 requires an Elasticsearch cluster in version 6.7+, while Neptune 2.2 can work with either Elasticsearch 6.7+ or 7.10+.

Upgrade procedure:

  1. Upgrade the Elasticsearch cluster to version 6.8.23.
  2. Upgrade Neptune to version 2.2.
  3. Upgrade the Elasticsearch cluster to version 7.10+.
I'm getting errors after upgrading

You may experience errors after the upgrade, such as the experiments table not showing.

Solution: Remove the entire index from Elasticsearch. Neptune will then rebuild it from the database.

It may take a few minutes up to a few hours, depending on the size of the project.

Upgrading Neptune#

A typical upgrade procedure is similar to a first-time installation.

  1. Perform a backup. This step is optional, but strongly recommended.

    In particular, this means creating a snapshot of any persistent volume or database that you provide for Neptune.

  2. Download the installer (neptune_installation_{version}.tgz) with the new version to a host where you will run the installer and unpack the archive.

  3. Execute the installation command. (See Installation.)

Note

The configuration file used previously contains your license, credentials for downloading Neptune images, your Neptune administrator credentials, among other things.

Do not change them without consulting with the neptune.ai team.