Jupyter Lab and Jupyter Notebooks are popular IDEs used by data scientists for data exploration, model training, error analysis, reporting, and more.
Neptune + Jupyter integration lets you version, manage and share notebook checkpoints in your projects. Specifically, you can:
log and display notebook checkpoints either manually or automatically during model training
connect notebook checkpoints with model training Runs in Neptune
organize checkpoints with names and descriptions
browse checkpoints history across all Notebooks in the project
compare Notebooks side-by-side, with diffs for source, markdown, output, and execution count cells
share Notebook checkpoints or diffs with persistent links
download Notebook checkpoints directly from the UI or Jupyter
Technically Neptune + Jupyter integration is implemented as a neptune-notebooks Python package which includes:
A very basic CLI (command line interface) for uploading notebooks.
The following libraries must be installed on your system before you begin installing neptune-notebooks extension for Jupyter:
Python 3.6 +
Step 1: Install neptune-notebooks extension
pip install neptune-notebooks
conda install -c conda-forge neptune-notebooks
Step 2: Enable the extension for Jupyter
jupyter nbextension enable --py neptune-notebooks
Step 3: Install neptune-client
pip install neptune-client
conda install -c conda-forge neptune-client
The following libraries must be installed on your system before you begin installing neptune-notebooks extension for JupyterLab:
This is the most complete installation of neptune-notebooks which includes:
Extensions for both standard Jupyter and JupyterLab
CLI for uploading Notebook files
neptune-client package is a dependency
Step 1: Go to the console and run
pip install neptune-notebooks
Step 2: Start JupyterLab by running
Step 3: In the “Build Recommended” notification that is displayed, click Build.
The build process takes up to several minutes.
Step 4: In the “Build Complete” notification that is displayed, click Reload.
Step 1: In JupyterLab, enable Extension Manager, by clicking Settings > Enable Extension Manager.
Step 2: Find neptune-notebooks and click Install.
To see if neptune-notebooks extension for Jupyter or JupyterLab was successfully installed:
open up a notebook
look at the top navigation of your notebook and look for Neptune buttons: Neptune Icon, Upload, Download, and Activate
After you have successfully installed the Jupyter extension for Neptune, you connect it to your Neptune account.
Step 1: In Jupyter, click Neptune Icon.
The Configure your connection to Neptune dialog appears.
Step 2: Paste your Neptune API token.
Step 3: Click Connect
A confirmation message is displayed. It contains a link through which you can go directly to this Notebook checkpoint in Neptune.
Step 4: Click Activate
To conclude, to see Runs that you will create associated with this Notebook, you need to click Activate.
You can upload Notebook checkpoints from Jupyter to Neptune.
To upload the current Notebook as a checkpoint:
In the dialog that is displayed, select a project from the list.
(Optional) Type in a checkpoint name and description.
Click Upload checkpoint.
A confirmation message is displayed. You can click the link in the message to open the Notebook in Neptune.
You can save notebook checkpoints automatically to Neptune whenever you create a Neptune Run in a notebook cell.
Neptune connects Runs created in a notebook checkpoint with that notebook checkpoints letting you:
find saved notebook checkpoints connected to the Neptune Run by going to the Source code > Notebook snapshot
find Neptune Runs that were created in a Notebook by going to the Notebooks pane and then Details > Runs
To save notebook checkpoint with every new Neptune Run:
Step 1: Log the Notebook to Neptune
Before the very first time you create a Neptune Run from a notebook, you should upload that notebook to Neptune. After that, whenever you create a Run your notebook checkpoint will be logged automatically.
To upload it click Upload button in the Jupyter navigation.
Step 2: Create a Neptune Run
Create a Neptune Run in a notebook cell:
import neptune.new as neptunerun = neptune.init(api_token="YOUR_API_TOKEN",project="YOUR_WORKSPACE/YOUR_PROJECT")
Now, and with every future execution of a cell that contains
neptune.init() your notebook checkpoint gets uploaded to Neptune and connected to that Neptune Run.
Step 3: See notebook snapshot in Neptune UI
Find Neptune Run you created in the Neptune UI.
Go to Source code > Notebook snapshot to find the notebook checkpoint associated with this Run.
The Notebooks tab in the Neptune UI provides a table of all the Notebooks in the current project.
This view lets you see what your team members are working on, review details and checkpoints associated with a Notebook, as well as share or download a Notebook and compare two or more Notebooks.
Once you select a Notebook, you can see all its contents including code and markdown cells, outputs, and execution count.
There are two tabs on the right:
Details: ID, size, creation date, latest checkpoint, owner, description, and associated Runs of the selected Notebook.
Checkpoints: all the checkpoints of the Notebook. Click a checkpoint to see the details in the main pane. From this tab, you can also access the Runs that are associated with the checkpoint.
The Notebooks Comparison pane lets you compare Notebook checkpoints including:
Cell Output: with prints, visualizations, and other HTML outputs
Execution count: order in which cells were executed
To compare notebook checkpoints:
select two Notebook checkpoints
explore the side-by-side comparison, just like source code.
You can download a specific Notebook checkpoint from Neptune to Jupyter.
To download a Notebook checkpoint from Jupyter:
Go to your Jupyter notebook
Find Neptune buttons in the top navigation
In the dialog that is displayed, select Project, Notebook and Checkpoint
You can create Notebooks and update Notebook checkpoints in Neptune from the command line.
To do that, use
notebook sync command.
neptune notebook sync \--project MY_WORKSPACE/MY_PROJECTyour_notebook.ipynb
notebook sync lets you create new notebooks in Neptune or update existing ones with new checkpoints.
To use the CLI command, you must export your NEPTUNE_API_TOKEN as an environment variable. You can do this in either of two ways:
run this command:
append the line above to your
Project to which to log Notebook or checkpoint. If the NEPTUNE_PROJECT environment variable is set, then this command overwrites the environment variable.
Takes the user’s Notebook and creates a new Notebook in Neptune.
If the Notebook is not known to Neptune, it will be created. In such a case, you do not need to use this flag.
neptune notebook sync \--project MY_WORKSPACE/MY_PROJECT \your_notebook.ipynb \--new
Updates the user’s Notebook in Neptune by adding a new checkpoint to it.
If the Notebook is known to Neptune it will be updated. In such a case, you do not need to use this flag.
neptune notebook sync \--project MY_WORKSPACE/MY_PROJECT \your_notebook.ipynb \--update
Please visit the Getting help page. Everything regarding support is there: