What will you get with this integration?

​Jupyter Lab and Jupyter Notebooks are popular IDEs used by data scientists for data exploration, model training, error analysis, reporting, and more.

Neptune + Jupyter integration lets you version, manage and share notebook checkpoints in your projects. Specifically, you can:

• log and display notebook checkpoints either manually or automatically during model training

• connect notebook checkpoints with model training Runs in Neptune

• organize checkpoints with names and descriptions

• browse checkpoints history across all Notebooks in the project

• compare Notebooks side-by-side, with diffs for source, markdown, output, and execution count cells

• share Notebook checkpoints or diffs with persistent links​

• download Notebook checkpoints directly from the UI or Jupyter

Technically Neptune + Jupyter integration is implemented as a neptune-notebooks Python package which includes:

• Standard Jupyter extension. The extension is written in Javascript and is visible in the web-based Jupyter UI.

• JupyterLab extension. The extension is written in Javascript and is visible in the web-based Jupyter UI.

• A very basic CLI (command line interface) for uploading notebooks.

Installation

Jupyter Notebook

Prerequisites

The following libraries must be installed on your system before you begin installing neptune-notebooks extension for Jupyter:

• Python 3.6 +

• ​pip or conda​

• ​Jupyter​

Step 1: Install neptune-notebooks extension

pip install neptune-notebooks

conda install -c conda-forge neptune-notebooks

Step 2: Enable the extension for Jupyter

jupyter nbextension enable --py neptune-notebooks

Step 3: Install neptune-client

pip install neptune-client

conda install -c conda-forge neptune-client

JupyterLab

Prerequisites

The following libraries must be installed on your system before you begin installing neptune-notebooks extension for JupyterLab:

• Python 3.6 +

• ​pip or conda​

• ​Node.js​

• ​npm​

• ​JupyterLab​

Install through Command Line

This is the most complete installation of neptune-notebooks which includes:

• Extensions for both standard Jupyter and JupyterLab

• CLI for uploading Notebook files

• neptune-client package is a dependency

Step 1: Go to the console and run

pip install neptune-notebooks

Step 2: Start JupyterLab by running

jupyter lab

Step 3: In the “Build Recommended” notification that is displayed, click Build.

The build process takes up to several minutes.

Step 4: In the “Build Complete” notification that is displayed, click Reload.

Install through JupyterLab Extension Manager

Step 1: In JupyterLab, enable Extension Manager, by clicking Settings > Enable Extension Manager.

Step 2: Find neptune-notebooks and click Install.

Confirm successful installation

To see if neptune-notebooks extension for Jupyter or JupyterLab was successfully installed:

• open up a notebook

• look at the top navigation of your notebook and look for Neptune buttons: Neptune Icon, Upload, Download, and Activate

Configuration

After you have successfully installed the Jupyter extension for Neptune, you connect it to your Neptune account.

Step 1: In Jupyter, click Neptune Icon.

The Configure your connection to Neptune dialog appears.

Step 2: Paste your Neptune API token.

See how to get your Neptune API token from the Neptune UI.

Step 3: Click Connect

A confirmation message is displayed. It contains a link through which you can go directly to this Notebook checkpoint in Neptune.

Step 4: Click Activate

To conclude, to see Runs that you will create associated with this Notebook, you need to click Activate.

Uploading notebook checkpoints

Upload notebook with a button

You can upload Notebook checkpoints from Jupyter to Neptune.

To upload the current Notebook as a checkpoint:

• Click Upload.

• In the dialog that is displayed, select a project from the list.

• (Optional) Type in a checkpoint name and description.

• Click Upload checkpoint.

A confirmation message is displayed. You can click the link in the message to open the Notebook in Neptune.

Auto-saving with neptune.init()

You can save notebook checkpoints automatically to Neptune whenever you create a Neptune Run in a notebook cell.

Neptune connects Runs created in a notebook checkpoint with that notebook checkpoints letting you:

• find saved notebook checkpoints connected to the Neptune Run by going to the Source code > Notebook snapshot

• find Neptune Runs that were created in a Notebook by going to the Notebooks pane and then Details > Runs

To save notebook checkpoint with every new Neptune Run:

Step 1: Log the Notebook to Neptune

Before the very first time you create a Neptune Run from a notebook, you should upload that notebook to Neptune. After that, whenever you create a Run your notebook checkpoint will be logged automatically.

To upload it click Upload button in the Jupyter navigation.

Step 2: Create a Neptune Run

Create a Neptune Run in a notebook cell:

import neptune.new as neptune
​
run = neptune.init(api_token="YOUR_API_TOKEN",
                   project="YOUR_WORKSPACE/YOUR_PROJECT")

Now, and with every future execution of a cell that contains neptune.init() your notebook checkpoint gets uploaded to Neptune and connected to that Neptune Run.

Step 3: See notebook snapshot in Neptune UI

• Find Neptune Run you created in the Neptune UI.

• Go to Source code > Notebook snapshot to find the notebook checkpoint associated with this Run.

Displaying Notebooks and checkpoints

The Notebooks tab in the Neptune UI provides a table of all the Notebooks in the current project.

This view lets you see what your team members are working on, review details and checkpoints associated with a Notebook, as well as share or download a Notebook and compare two or more Notebooks.

Once you select a Notebook, you can see all its contents including code and markdown cells, outputs, and execution count.

There are two tabs on the right:

• Details: ID, size, creation date, latest checkpoint, owner, description, and associated Runs of the selected Notebook.

• Checkpoints: all the checkpoints of the Notebook. Click a checkpoint to see the details in the main pane. From this tab, you can also access the Runs that are associated with the checkpoint.

Comparing checkpoints

The Notebooks Comparison pane lets you compare Notebook checkpoints including:

• Code

• Markdown

• Cell Output: with prints, visualizations, and other HTML outputs

• Execution count: order in which cells were executed

To compare notebook checkpoints:

• select two Notebook checkpoints

• click Compare

• explore the side-by-side comparison, just like source code.

Downloading checkpoints

You can download a specific Notebook checkpoint from Neptune to Jupyter.

To download a Notebook checkpoint from Jupyter:

• Go to your Jupyter notebook

• Find Neptune buttons in the top navigation

• Click Download

• In the dialog that is displayed, select Project, Notebook and Checkpoint

• Click Download

Using Neptune CLI for notebooks

You can create Notebooks and update Notebook checkpoints in Neptune from the command line.

To do that, use notebook sync command.

neptune notebook sync \
   --project MY_WORKSPACE/MY_PROJECT 
   your_notebook.ipynb

notebook sync lets you create new notebooks in Neptune or update existing ones with new checkpoints.

Setup

To use the CLI command, you must export your NEPTUNE_API_TOKEN as an environment variable. You can do this in either of two ways:

• run this command:

export NEPTUNE_API_TOKEN='YOUR_LONG_API_TOKEN'

set NEPTUNE_API_TOKEN="YOUR_LONG_API_TOKEN"

• append the line above to your ~/.bashrc or ~/.bash_profile file.

Parameters

--project (optional)

Project to which to log Notebook or checkpoint. If the NEPTUNE_PROJECT environment variable is set, then this command overwrites the environment variable.

--new (optional)

• Takes the user’s Notebook and creates a new Notebook in Neptune.

• If the Notebook is not known to Neptune, it will be created. In such a case, you do not need to use this flag.

neptune notebook sync \
   --project MY_WORKSPACE/MY_PROJECT \
   your_notebook.ipynb \
   --new

--update (optional)

• Updates the user’s Notebook in Neptune by adding a new checkpoint to it.

• If the Notebook is known to Neptune it will be updated. In such a case, you do not need to use this flag.

neptune notebook sync \
   --project MY_WORKSPACE/MY_PROJECT \
   your_notebook.ipynb \
   --update

How to ask for help?

Please visit the Getting help page. Everything regarding support is there:

See also