Skip to content

Troubleshooting self-hosted Neptune#

This page lists problem situations related to self-hosted Neptune and how to work around or solve them.

Got issues or questions?

To get dedicated support or leave feedback, email us at support@neptune.ai.

You can also check out the self-hosting FAQ and other ways to get help.

New runs not showing up in web app#

Issue: New experiments are not showing up in the experiments table, but can be accessed via direct URL.

Solution: If you're running low on disk space, Neptune may not be able to index new runs.

Important

Make sure to check the usage of the Neptune-dedicated disk, not other disks (such as the system disk).

Long loading times and 502 errors#

Issue: When using Google Load Balancer to expose Neptune, the web interface slows down and there are frequent 502 errors.

Nginx closes TCP connections in Kubernetes after one minute, whereas Google Load Balancer expects them to be alive for 10 minutes. This leads to connection timeouts and 502 errors.

To confirm that this is the problem, in a monitoring services that can collect logs (such as BigQuery), inspect a 502 log and check the fields for a status that suggest issues with the backend.

Solution: To keep connections stable, increase the Nginx keep-alive timeout.

Assuming a single-VM installation:

  1. Edit the ingress controller with the following command:

    kubectl edit cm -n neptune nginx-ingress-controller

  2. In the data section, set the keep-alive option to a value of more than 600:

    apiVersion: v1
data:
disable-ipv6: "true"
keep-alive: "650"
hsts: "false"
large-client-header-buffers: 4 32k
...

For help, reach out to us at support@neptune.ai.

How to send data to Neptune#

Your currently used domain is encoded into the token itself. The place where you run your Neptune client library just needs to know how to resolve this domain.

To find and copy your API token:

  1. Log in to the web interface of your self-hosted installation of Neptune.
  2. In the bottom-left, open the user menu → Get your API token.
  3. Use the token with the Python client library. For help, see Set your API token.

Note

We strongly recommend using HTTPS (SSL certificates) when exposing Neptune. If you want to use HTTP or you have a self-signed certificate, exporting some environmental variables might be required for the client to work properly.

Getting help