Skip to content

Tracking and organizing model-training runs#

Open in Colab

Need a more detailed walkthrough that starts from installation? The Neptune tutorial has you covered.

This example walks you through the ways you can track and organize runs in Neptune:

  • Keep track of code, data, environment, and parameters.
  • Log results, like evaluation metrics and model files.
  • Work with tags.
  • Create a custom view of the runs table and save it for later.

Before you start#

What if I don't use scikit-learn?

No worries, we're just using it for demonstration purposes. You can use any framework you like, and Neptune has intregrations with various popular frameworks. For details, see the Integrations tab.

Create a basic training script#

As an example, we'll use a script that trains a scikit-learn model on the wine dataset.

Create a file train.py and copy the script below.

train.py
from sklearn.datasets import load_wine
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import f1_score
from joblib import dump

data = load_wine()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.4, random_state=1234
)

params = {
    "n_estimators": 10,
    "max_depth": 3,
    "min_samples_leaf": 1,
    "min_samples_split": 2,
    "max_features": 3,
}

clf = RandomForestClassifier(**params)
clf.fit(X_train, y_train)
y_train_pred = clf.predict_proba(X_train)
y_test_pred = clf.predict_proba(X_test)

train_f1 = f1_score(y_train, y_train_pred.argmax(axis=1), average="macro")
test_f1 = f1_score(y_test, y_test_pred.argmax(axis=1), average="macro")
print(f"Train f1:{train_f1} | Test f1:{test_f1}")

dump(clf, "model.pkl")

In your terminal program, run the script to ensure that it works properly.

python train.py

Connect Neptune to your code#

At the top of your script, add the following:

import neptune.new as neptune

run = neptune.init_run()  # (1)
  1. We recommend saving your API token and project name as environment variables. If needed, you can pass them as arguments when initializing Neptune: neptune.init_run(project="workspace-name/project-name", api_token="Your Neptune API token here")
Haven't registered yet?

No problem. You can try Neptune anonymously by logging to a public project with a shared API token:

run = neptune.init_run(api_token=neptune.ANONYMOUS_API_TOKEN, project="common/quickstarts")

This creates a new run in Neptune, to which you can log various types of metadata.

Let's track the parameters, code, and environment by assigning them to the run:

run["parameters"] = params

Neptune captures the contents of the entry-point script by default, but you can specify additional files to snapshot when you initialize the run. It can be helpful if you forget to commit your code changes with Git.

To specify what source code to track, pass a list of files or a regular expression to the source_files argument:

run = neptune.init_run(source_files=["*.py", "requirements.txt"])  # (1)
  1. When using pattern expansion, such as **/*.py, make sure that your expression does not capture files you don't intend to upload. For example, using * as a pattern will upload all files and directories from the current working directory (cwd).

If you start the run from a directory that is a part of a Git repository, Neptune will automatically find the .git directory and log some information from it.

Putting it all together, your script should now look like this:

train.py
from sklearn.datasets import load_wine
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import f1_score
from joblib import dump
import neptune.new as neptune

run = neptune.init_run(source_files=["*.py", "requirements.txt"])

data = load_wine()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.4, random_state=1234
)

params = {
    "n_estimators": 10,
    "max_depth": 3,
    "min_samples_leaf": 1,
    "min_samples_split": 2,
    "max_features": 3,
}

run["parameters"] = params

clf = RandomForestClassifier(**params)
clf.fit(X_train, y_train)
y_train_pred = clf.predict_proba(X_train)
y_test_pred = clf.predict_proba(X_test)

train_f1 = f1_score(y_train, y_train_pred.argmax(axis=1), average="macro")
test_f1 = f1_score(y_test, y_test_pred.argmax(axis=1), average="macro")
print(f"Train f1:{train_f1} | Test f1:{test_f1}")

dump(clf, "model.pkl")

Add tags to organize things#

Tags can help you find runs later, especially if you try a lot of ideas. You can filter by tag when viewing the runs table and querying runs through the API.

Tags are stored in the tags field of the system namespace "sys".

To tag the run through the code, add a set of strings to the "sys/tags" field with the add() method:

run["sys/tags"].add(["run-organization", "me"])

Tip

You can also add and manage tags through the Neptune app.

Log train and evaluation metrics#

Log the scores you want to track by assigning them to fields in the run:

run["train/f1"] = train_f1
run["test/f1"] = test_f1

You can log a series of values to the same field with the log() method:

acc = ...
run["train/loss"].log(acc)

In the code above, each log() call appends a value to the series stored in train/loss. You can view the resulting series as a chart in the Neptune app.

Upload files#

You can upload any file, such as a model file, with the upload() method:

run["model"].upload("my_model.pkl")

Track a few runs with different parameters#

Let's execute some runs with different model configurations.

  1. Change some parameters in the params dictionary:

    params = {
        "n_estimators": 10,
        "max_depth": 3,
        "min_samples_leaf": 1,
        "min_samples_split": 2,
        "max_features": 3,
    }
    
  2. To stop the connection to Neptune and sync all data, call the stop() method:

    run.stop()
    
    Warning

    Always call stop() in interactive environments, such as a Python interpreter or Jupyter notebook. The connection to Neptune is not stopped when the cell has finished executing, but rather when the entire notebook stops.

    If you're running a script, the connection is stopped automatically when the script finishes executing. However, it's a best practice to call stop() when the connection is no longer needed.

  3. Execute the script

    python train.py
    
If Neptune can't find your project name or API token

As a best practice, you should save your Neptune API token and project name as environment variables.

You can, however, also pass them as arguments when initializing Neptune:

run = neptune.init_run(
    api_token="h0dHBzOi8aHR0cHM6Lkc78ghs74kl0jvYh3Kb8",  # your token here
    project="ml-team/classification",  # your full project name here
)
  • Find and copy your API token by clicking your avatar and selecting Get my API token.
  • Find and copy your project name in the project SettingsProperties.

If you haven't registered, you can also log anonymously to a public project (make sure not to publish sensitive data through your code!):

run = neptune.init_run(
    api_token=neptune.ANONYMOUS_API_TOKEN,
    project="common/quickstarts",
)

Organize results in the Neptune app#

To access a run directly, click the run link that appears in the console output.

If you already have the project open in the app, click on a run in the runs table to open it.

See that everything was logged#

To check that everything was logged correctly, navigate to the following sections:

  • All metadata: Review all logged metadata
  • monitoring namespace: See hardware utilization charts.
  • parameters namespace: See your parameters.
  • "sys" namespace: View metadata about your run.
    • To edit the name, description, or tags, click Details in the left pane.

Filter runs by tag#

Navigate back to the runs table.

In the input box above the table, filter the runs by the run-organization tag with the following query: "Tags" + "one of" + "run-organization".

Choose parameter and metric columns to display#

Neptune generally suggests columns if the fields have values that differ between the selected runs.

To add a column manually:

  1. Click Add column above the table.
  2. Select a field from the list, or start typing to match available fields. For example, f1.

Customize column appearance#

To customize the runs table view even further, you can click the settings icon () of any column to set a custom name and color.

Save the custom view#

To save the current view of the runs table for later, click Save as new above the query input box. The saved view retains column customization and row filtering.

To share the view with collaborators, you can copy the link, send it via email, or export the view as CSV.

Tip

Create and save multiple views of the runs table for different use cases or run groups.

See in Neptune  Code examples