Log to handlers#
You can organize your metadata into folder-like structures with the help of namespaces and handler objects.
If you assign a run namespace path to a variable, as on the highlighted line below, you get a namespace handler object.
import neptune
# You can upload the value through an absolute path
run["params/max_epochs"] = 20
# Or with a namespace handler and a relative path
params_namespace = run["params"]
params_namespace["learning_rate"] = 0.005
In this case, the params
namespace is created when we assign the nested max_epochs
field to the run. You can then access the params_namespace
object and use it to nest any logged metadata under the params
namespace.
In the pipeline logging example below, we show how to use a handler to organize metadata from a particular step of a run into a given namespace.
Using a generic Handler
object#
If you assign a run path that doesn't exist yet, you get a Handler
object that can become anything (including a namespace, if you then nest other things under it).
Example: Handler becomes field#
Because no namespaces or fields exist in the run yet, accessing the path run["train/acc"]
returns a Handler
object:
It's a reference that points to some location in the run – we just don't know what's there yet.
Let's assign something to the handler:
The handler now represents a FloatSeries
.
Once a field exists under the path which the handler points to, it only accepts the corresponding field methods:
handler_object.append(0.72) # Works
handler_object.upload("image.png") # Error: Can't upload an image to a FloatSeries
Example: Handler becomes namespace#
In this example, we'll have an overall run structure that is identical to the example above, but our Handler
object will reference a different location in the run.
If we try accessing the path run["train"]
, a Handler
is returned, as that field does not exist yet and no value has been assigned to it:
We can now assign something to handler_object
. Let's assign the value 0.82
to the field acc
and assign that to the handler:
Since the handler object for run["train"]
has something nested under it – in this case, a FloatSeries
field – the handler now represents a namespace.
Pipeline logging example#
Let's say we have a data preprocessing step in our pipeline, and we would like to ensure that all of the metadata generated during this part is organized under a folder named "preprocessing".
We would start a Neptune run early in the script:
import neptune
from sklearn.datasets import fetch_lfw_people
from utils import *
dataset = fetch_lfw_people(min_faces_per_person=70, resize=0.4)
run = neptune.init_run()
dataset_config = {...}
To collect the metadata from this step into one place, we set up a preprocessing
namespace inside the run. This will be the base namespace where all the preprocessing metadata is logged.
We define the path run["preprocessing"]
and assign it to a handler object, which we'll then use to log metadata as if it were a run.
preprocessing_handler = run["preprocessing"]
# Log dataset details
preprocessing_handler["dataset/config"] = dataset_config
# Preprocess dataset
dataset_transform = Preprocessing(...)
...
# Log scaler and features files
preprocessing_handler["dataset/scaler"].upload(path_to_scaler)
This way, whenever we log something to preprocessing_handler
, it'll end up under the path run["preprocessing/..."]
.
See full example script on GitHub 
Tip
The handler exposes similar methods as other Neptune objects.
For details, see API reference Field types: Namespace handler.