Troubleshooting self-hosted Neptune#

This page lists problem situations related to self-hosted Neptune and how to work around or solve them.

Wrong username#

If you delete your Neptune account, you can't use the same email to create a new account.

To change the username associated with your account, contact support@neptune.ai.

Caution

Don't delete your account, unless you no longer use the email associated with the account.

New runs not showing up in web app#

Issue: New experiments are not showing up in the experiments table, but can be accessed via direct URL.

Solution: If you're running low on disk space, Neptune may not be able to index new runs.

Note

Make sure to check the usage of the Neptune-dedicated disk, not other disks (such as the system disk).

Long loading times and `502` errors#

Issue: When using Google Load Balancer to expose Neptune, the web interface slows down and there are frequent 502 errors.

Nginx closes TCP connections in Kubernetes after one minute, whereas Google Load Balancer expects them to be alive for 10 minutes. This leads to connection timeouts and 502 errors.

To confirm that this is the problem, in a monitoring services that can collect logs (such as BigQuery), inspect a 502 log and check the fields for a status that suggest issues with the backend.

Solution: To keep connections stable, increase the Nginx keep-alive timeout.

Assuming a single-VM installation:

Edit the ingress controller with the following command:

kubectl edit cm -n neptune nginx-ingress-controller

In the data section, set the keep-alive option to a value of more than 600:

apiVersion: v1
data:
disable-ipv6: "true"
keep-alive: "650"
hsts: "false"
large-client-header-buffers: 4 32k
...

For help, reach out to us at support@neptune.ai.

`NeptuneConnectionLostException`#

This error indicates that the Neptune client lost connection to the server. This can happen for various reasons, such as network issues or server downtime. Most often, it's temporary and resolves itself.

If the issue does not resolve itself, follow the steps below to troubleshoot the problem:

To allow the client to connect to the server with a self-signed certificate, set the NEPTUNE_ALLOW_SELF_SIGNED_CERTIFICATE environment variable to true.
If the issue is still not resolved, execute the following command to check the connection between the Neptune client and server:

curl NEPTUNE_SERVER_URL

where NEPTUNE_SERVER_URL is the URL to your self-hosted Neptune instance.

If the output is an HTML string, it indicates that the connection is fine and there might be some other issue. Reach out to Neptune support at support@neptune.ai with the output.

Example output

<!doctype html>
<html>
<head>
  <!-- neptune-frontend version: cc5aae46 -->
  <meta charset="utf-8">
  <title>neptune.ai</title>
  ...
</html>

Other results, such as curl: (6) Could not resolve host: app.neptune.ai, indicate a connection issue unrelated to Neptune. In this case, check your network settings and the server's status. Your IT/DevOps team can help you with this.

How to send data to Neptune#

Your currently used domain is encoded into the token itself. The place where you run your Neptune client library just needs to know how to resolve this domain.

To find and copy your API token:

Log in to the web interface of your self-hosted installation of Neptune.
In the bottom-left, open the user menu → Get your API token.
Use the token with the Python client library. For help, see Set your API token.

Note

We strongly recommend using HTTPS (SSL certificates) when exposing Neptune. If you want to use HTTP or you have a self-signed certificate, exporting some environmental variables might be required for the client to work properly.

Getting help