Neptune-AWS EC2 Integration

You can run Neptune and track experiments that you run on AWS cloud.

AWS neptune.ai integration

To set this up, perform the following steps:

  1. Register to AWS.

    Follow the instructions to create your AWS account.

  2. Start an EC2 instance.

    Select ubuntu as your instance type and choose the worker type you need. You can use t2.micro just to test it out.

  3. SSH to your instance.

    From the terminal, run:

    ssh -i /path_to_key/my_key.pem ubuntu@public_dns_name
    

    Make sure that you specify the correct key and public_dns_name.

  4. Install Docker.

    1. Create a new file install_docker.sh:

    nano install_docker.sh
    
    1. Copy the following commands to it:

    sudo apt-get update
    sudo apt-get install \
        apt-transport-https \
        ca-certificates \
        curl \
        gnupg-agent \
        software-properties-common
    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
    sudo apt-key fingerprint 0EBFCD88
    sudo add-apt-repository \
    "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
    $(lsb_release -cs) \
    stable"
    sudo apt-get update
    sudo apt-get install docker-ce docker-ce-cli containerd.io
    
    1. Run the installation script:

    source install_docker.sh
    
  5. Define your secrets.

    1. Go to Neptune, get your NEPTUNE_API_TOKEN and copy it to the clipboard.

    2. Create a password for your JupyterLab server.

    3. Set the following two secrets to your environment variables, NEPTUNE_API_TOKEN and JUPYTERLAB_PASSWORD:

      export NEPTUNE_API_TOKEN='your_api_token=='
      export JUPYTERLAB_PASSWORD='difficult_password'
      
  6. Build the Docker image.

    1. Create a new file Dockerfile:

    nano Dockerfile
    
    1. Copy insights of the following Dockerfile to your newly created file. (If you want to run on a GPU, make sure to change your Dockerfile to start from Nvidia Docker images.)

    # Use a miniconda3 as base image
    FROM continuumio/miniconda3
    
    # Installation of jupyterlab and extensions
    RUN pip install jupyterlab==1.2.6  && \
        pip install jupyterlab-server==1.0.6  && \
        conda install -c conda-forge nodejs
    
    # Installation of Neptune and enabling neptune extension
    RUN pip install neptune-client  && \
        pip install neptune-notebooks  && \
        jupyter labextension install neptune-notebooks
    
    # Setting up Neptune API token as env variable
    ARG NEPTUNE_API_TOKEN
    ENV NEPTUNE_API_TOKEN=$NEPTUNE_API_TOKEN
    
    # Adding current directory to container
    ADD . /mnt/workdir
    WORKDIR /mnt/workdir
    
    1. Run the following command to build your Docker image:

    sudo docker build -t jupyterlab --build-arg NEPTUNE_API_TOKEN=$NEPTUNE_API_TOKEN .
    
  7. Spin up a JupyterLab server with Docker:

    sudo docker run --rm -v `pwd`:/work/output -p 8888:8888 jupyterlab:latest \
    /opt/conda/bin/jupyter lab --allow-root --ip=0.0.0.0 --port=8888 --NotebookApp.token=$JUPYTERLAB_PASSWORD
    
  8. Forward ports via SSH tunnel. Open a new terminal on your local machine and run:

    ssh -L 8888:localhost:8888 ubuntu@public_dns_name &
    

    Make sure that you specify the correct public_dns_name.

  9. Open the JupyterLab server in your browser.

    Go to localhost:8888.

Enjoy your JupyterLab server with Neptune!

Neptune extensions are enabled and NEPTUNE_API_TOKEN is already in the environment variable so you can work with Notebooks and run experiments with no problem.