Skip to content

Open in Colab

Tracking models end-to-end#

To track a model across all stages of its lifecycle, you can use Neptune for the following:

  • Logging model and run metadata to a central project
  • Grouping models by their stage or other criteria
  • Comparing models to select the best performing model
  • Monitoring a model once in production

Model metadata grouped by stage in the web app

To demonstrate this, we provide Jupyter notebook example.

View notebook on GitHub  See result in Neptune 

The example uses Optuna hyperparameter-optimization to simulate the training and evaluating of multiple XGBoost models, and Evidently to monitor models in production. However, you can use any library and framework of your choice.

The script can be used as a template to design an automated end-to-end pipeline that covers the entire lifecycle of a model without needing any manual intervention.

To download the notebooks and scripts of this example, visit the Neptune examples repo .

Summary of steps#

The end-to-end tracking flow consists of the following key elements and Neptune features:

  1. Create a run and log model metadata:

    Sample snippet
    run = neptune.init_run()

run["training/parameters"] = train_params_dict
...

  2. Use tags to mark the model stage:

    Sample snippet
    run["sys/group_tags"].add(["development"])

    You can also apply tags later in the web app.

  3. In the experiments tab, use Group mode to group the runs by stage. You can save the grouping as a custom view that's shared project-wide. See an example →

  4. When comparing metrics, you can display the mean of each group as a single graph. For example, if the runs are grouped by stage, you can have one graph per stage. See an example →

    To enable grouping on a chart, in the widget settings, turn on the Average grouped runs option.

To fetch runs by model stage, use the query language:

# Connect to your Neptune project
project = neptune.init_project(mode="read-only")

# Fetch runs with models in production
production_models_df = project.fetch_runs_table(
    query='`sys/group_tags`:stringSet CONTAINS "production"',
).to_pandas()

For more detailed examples of promoting and monitoring models programmatically, see the tutorial notebook.

Related