Neptune legacy API#
The legacy client is deprecated
With the 1.0 release of the Neptune client library, we're ending active maintenance and support for the Neptune legacy API.
From 1.0 onward, you must import the legacy API with the following statement:
We strongly encourage you to migrate to the new client at your earliest convenience.
Legacy docs#
Neptune legacy client documentation
Migrating to the new API#
This guide walks you through migrating your legacy code to the new Python API.
Initialization#
import neptune.legacy as neptune
neptune.init(project_qualified_name="my-workspace/my-project")
neptune.create_experiment(tags=["resnet"])
Note
The names of the environment variables have not changed.
Instead of specifying the project name or API token in the code, you can provide them by setting the NEPTUNE_PROJECT and NEPTUNE_API_TOKEN environment variables.
Hierarchical structure#
Runs can be viewed as nested dictionary-like structures that you can define in your code. This lets you organize your metadata in a way that is most convenient for you.
The hierarchical structure that you apply to your metadata is reflected when you view the metadata in the app.
The structure of a run consists of fields that are organized into namespaces.
The path of a field is a combination of the namespaces and its name: if you store a value (such as a float, 0.8) in a field named momentum under a namespace params, the full path of the field is params/momentum.
You can organize any type of metadata this way – such as images, parameters, metrics, scores, model checkpoints, and CSV files.
Let's look at the following code:
import neptune
run = neptune.init_run(project="my_workspace/my_project")
run["about/JIRA"] = "NPT-952"
run["parameters/batch_size"] = 5
run["parameters/algorithm"] = "ConvNet"
for epoch in range(100):
acc_value = ...
loss_value = ...
run["train/accuracy"].append(acc_value)
run["train/loss"].append(loss_value)
exp["trained_model"].upload("model.pt")
The resulting structure of the run will be:
"about":
"JIRA": String
"parameters":
"batch_size": Float
"algorithm": String
"train":
"accuracy": FloatSeries
"loss": FloatSeries
"trained_model": File
Batch assign#
You can assign values to multiple fields in batch by using a dictionary.
With the legacy API, you had to pass parameters when creating an experiment and it was not possible to change them afterward. In addition, nested dictionaries were not fully supported.
import neptune.legacy as neptune
PARAMS = {"epoch_nr": 100, "lr": 0.005, "use_nesterov": True}
neptune.init()
neptune.create_experiment(params=PARAMS)
With the new neptune API, it's up to you when and where you want to specify parameters. You can also update them later:
import neptune
PARAMS = {"epoch_nr": 100, "lr": 0.005, "use_nesterov": True}
run = neptune.init_run()
run["my_params"] = PARAMS
# You can also specify parameters one by one
run["my_params/batch_size"] = 64
# Update lr value
run["my_params/lr"] = 0.007
The artificial distinction between parameters and properties is also gone. You can log and access them in a single, unified way.
Interacting with files (artifacts)#
You are no longer bound to store files only in the artifacts folder.
Whether it's a model checkpoint, custom interactive visualization, or audio file, you can track it in the same hierarchical structure with the rest of the metadata:
| Legacy | New |
|---|---|
neptune.log_artifact("model_viz.png") |
run[model/viz"].upload("model_viz.png") |
neptune.log_artifact("model.pt") |
run[trained_model"].upload("model.pt") |
neptune.download_artifact("model.pt") |
run["trained_model"].download() |
Note
In the legacy API, artifacts mimicked a file system. What you uploaded was saved under the same name, with the extension, etc.
The idea behind the new Python API is more database-like: We have a field (with a path) and under it, we store some content - Float, String, series of String, or File. In this model, the extension is part of the content.
Example: If under the path "model/last" you upload a .pt file, the file will be displayed as "last.pt" in the app.
When it's unambiguous, we implicitly convert an object to type File and there is no need for explicit conversion.
Example: For Matplotlib charts, you can do
instead of
neptune-contrib#
We've integrated neptune-contrib file-related functionalities into the core library.
The conversion methods are available as File factory methods:
Interactive charts (Altair, Bokeh, Plotly, Matplotlib)#
| Legacy | New |
|---|---|
from neptunecontrib.api import log_chart |
from neptune.types import File |
log_chart("int_chart", chart) |
run["int_chart"].upload(File.as_html(chart)) |
pandas DataFrame#
| Legacy | New |
|---|---|
from neptunecontrib.api import log_table |
from neptune.types import File |
log_table("pred_df", df) |
run["pred_df"].upload(File.as_html(df)) |
Audio and video#
We've expanded the range of files that are natively supported in the Neptune app, so for audio and video files you no longer need to use conversion methods:
| Legacy | New |
|---|---|
|
run["sample"].upload("sample.mp3") |
|
run["sample"].upload("sample.mp4") |
Pickled objects#
| Legacy | New |
|---|---|
from neptunecontrib.api import log_pickle |
from neptune.types import File |
log_pickle("model.pkl", model) |
run["model"].upload(File.as_pickle(model)) |
HTML strings#
| Legacy | New |
|---|---|
from neptunecontrib.api import log_html |
from neptune.types import File |
log_html("custom_viz", html_string) |
run["custom_viz"].upload(File.from_content(model), extension="html") |
Scores and metrics#
Logging metrics is quite similar, except that you can now organize them in a hierarchical structure:
| Legacy API | New API |
|---|---|
neptune.log_metric("acc", 0.97) |
run["acc"].append(0.97) |
neptune.log_metric("train_acc", 0.97) |
run["train/acc"].append(0.97) |
neptune.log_metric("loss", 0.8) |
run["key"].append() |
To log scores you don't need to use Series fields anymore as you can track single values anytime, anywhere:
| Legacy API | New API |
|---|---|
neptune.log_metric("final_accuracy", 0.8) |
run["final_accuracy"] = 0.8 |
Text and image series#
Similar changes need to be applied for text and image series:
| Legacy API | New API |
|---|---|
neptune.log_text("train_log", custom_log_msg) |
run["train/log"].append(custom_log_msg) |
neptune.log_image("misclassified", filepath) |
run["misclassified"].append(File(filepath)) |
neptune.log_image("pred_dist", hist_chart) |
run["pred_dist"].append(hist_chart) |
To add a single image that you want to view with the rest of the metrics you no longer need to use Series fields. As you control whether they are grouped in the same namespace you can upload it as a single File field.
| Legacy API | New API |
|---|---|
neptune.log_image("train_hist", hist_chart) |
run["train/hist"].upload(hist_chart) |
Integrations#
We've redesigned how our integrations work:
- They're more tightly integrated with our base library and the API is more unified.
- You can update the version of each integration separately in case of dependency conflicts.
- There's a smaller number of dependencies if you're not using all integrations.
There is no longer a neptune-contrib library for the new Python API – instead, each integration has two parts:
-
Boilerplate code for ease of use in the main library:
import neptune.integrations.framework_name -
The actual integration that can be installed separately:
pip install neptune-framework-nameor as an extra together with neptune:
pip install "neptune[framework-name]"
Existing integrations from neptune-contrib are still fully functional. You can use them both with projects using the previous structure and the new structure. However, integrations from neptune-contrib use the legacy Python API, while the new integrations are rewritten to fully leverage the new Python API for better metadata organization.
Related
For detailed integration guides, see the Integrations tab.
Let's look at how this works like in the case of the Keras integration:
Installation
| Legacy | New |
|---|---|
pip install neptune-contrib |
or
|
Usage
import neptune.legacy as neptune
neptune.init()
from neptunecontrib.monitoring.keras import NeptuneMonitor
model.fit(
x_train,
y_train,
epochs=5,
batch_size=64,
callbacks=[NeptuneMonitor()],
)
import neptune
run = neptune.init_run()
from neptune.integrations.tensorflow_keras import NeptuneCallback
model.fit(
x_train,
y_train,
epochs=5,
batch_size=64,
callbacks=[NeptuneCallback(run=run)],
)
Tags#
Interaction with tags is quite similar: tags are stored as a StringSet field under the sys/tags field.
import neptune.legacy as neptune
neptune.init()
neptune.create_experiment(params=PARAMS, tags=["maskRCNN"])
neptune.append_tag("prod_v2.0.1")
neptune.append_tags("finetune", "keras")
import neptune
run = neptune.init_run(tags=["maskRCNN"])
run["sys/tags"].add("prod_v2.0.1")
run["sys/tags"].add(["finetune", "keras"])
Logging to a project with legacy runs#
If you have a project with a set of runs (previously Experiments) that were created using the Legacy API and you would like to add runs using the new API, there are a few things to remember.
Metadata structure#
Each type of metadata from the previous version (parameters, properties, logs, artifacts, and monitoring) are mapped into a separate fixed namespace. If you want to be able to compare those, you need to make sure that the field path is backward compatible:
| Legacy | New |
|---|---|
neptune.create_experiment(params=params_dict) |
run["parameters"] = params_dict |
neptune.set_property("SEED", 2458) |
run["properties/SEED"] = str(2458) |
neptune.log_metric("acc", 0.98) |
run["logs/acc"].append(0.98) |
neptune.log_artifact("model.pt", "model.pt") |
run["artifacts/model"].upload("model.pt") |
Data types#
Neptune is type-sensitive. For example, a column in the experiments table represents a field with a specific path and type. In particular, FloatSeries and Float are two different types and Neptune by default doesn't know how to compare them.
With the Legacy API, it wasn't possible to set Float fields outside of the create_experiment() function, and a common practice was to store those as FloatSeries with one entry through the use of log_metric() function. To achieve backward compatibility, you need to create a FloatSeries with the new API as well, by using the append() method.
Properties set through Legacy API are always String fields. To achieve backward compatibility, use assignment with explicit casting to str to ensure that the resulting field will be a string.