Log model building metadata

Before you start
Make sure you meet the following prerequisites before starting:
Have Python 3.6+ installed
Have scikit-learn and joblib installed
​Have Neptune installed​
​Create a project​
​Configure Neptune API token on your system​
Step 1: Connect Neptune to your script
At the top of your script add
import neptune.new as neptune
​
run = neptune.init_run(
    project="common/quickstarts",
    api_token="ANONYMOUS",
)
This creates a new run in Neptune to which you can log metadata.
You need to tell Neptune who you are and where you want to log things. To do that you specify:
project=my_workspace/my_project: your workspace name and project name,
api_token=YOUR_API_TOKEN : your Neptune API token.
If you configured your Neptune API token correctly, as described here, you can skip the api_tokenargument.
Runs can be viewed as dictionary-like structures - namespaces - that you can define in your code. You can apply hierarchical structure to your metadata that will be reflected in the UI as well. Thanks to this you can easily organize your metadata in a way you feel is most convenient.
Step 2: Log parameters
PARAMS = {"lr": 0.1, "epoch_nr": 10, "batch_size": 32}
run["parameters"] = PARAMS
It logs your PARAMS dictionary with all the parameters that you want to keep track of.
See also:
How to Track Hyperparameters of Machine Learning Models? - neptune.ai
neptune.ai
Step 3: Log dataset version
Log dataset as a versioned artifact with the .track_files() method:
run["dataset/train"].track_files("data/train/images")
run["dataset/test"].track_files("data/test/images")
Step 4: Add logging for metrics and losses
To log a metric or loss during training you should use:
loss = ...
run["train/loss"].log(loss)
Few explanations here:
"train/loss" is a name of the log with a hierarchical structure.
"train/loss" is a series of values - you can log multiple values to this log.
You can have one or multiple log names like "train/acc", "val/f1_score", "train/log-loss", "test/acc").
The argument of the log() method is the actual value you want to log.
Typically during training, there will be some sort of a loop where those losses are logged. You can simply call run["train/loss"].log(loss) multiple times at each step.
for i in range(epochs):
    ...
    run["train/loss"].log(loss)
    run["train/acc"].log(accuracy)
Step 5: Add logging of test score
run["test/acc"] = 0.76
In this way, you will log a single value, not a series of values. Again, you can make use of the hierarchical structure of the run to organize scores - "test/acc".
You can also update runs after the script is done running. Read about updating existing runs.​
Step 6: Add logging of performance charts
# confusion_matrix is matplotlib figure object
confusion_matrix = ...
run["diagnostic_charts"].upload(confusion_matrix)
​
# log image predictions
from neptune.new.types import File
img_pred = File("path/to/image")
run["predictions"].log(img_pred)
Step 7: Add logging of model files
run["model"].track_files("my_model.pkl")
Log your model as a versioned artifact with .track_files() method. Just pass the path to the file you want to log to Neptune.
There are many other objects that you can log to Neptune:
What can you log and display?
Step 7: Stop logging
Once you are done logging, you should stop tracking the run using the stop() method. This is needed only while logging from a notebook environment. While logging through a script, Neptune automatically stops tracking once the script has completed execution.
run.stop()

Organize ML experiments

Next - How-to guides

Model registry

Last modified 15d ago

Export as PDF

Copy link

Outline

Before you start

Step 1: Connect Neptune to your script

Step 2: Log parameters

Step 3: Log dataset version

Step 4: Add logging for metrics and losses

Step 5: Add logging of test score

Step 6: Add logging of performance charts

Step 7: Add logging of model files

Step 7: Stop logging