Log to handlers#
You can organize your metadata into folder-like structures with the help of namespaces and handler objects.
If you assign a run namespace path to a variable, as on the highlighted line below, you get a namespace handler object.
In this case, the
params namespace is created when we assign the nested
max_epochs field to the run. You can then access the
params_namespace object and use it to nest any logged metadata under the
In the pipeline logging example below, we show how to use a handler to organize metadata from a particular step of a run into a given namespace.
Using a generic
If you assign a run path that doesn't exist yet, you get a
Handler object that can become anything (including a namespace, if you then nest other things under it).
Example: Handler becomes field#
Because no namespaces or fields exist in the run yet, accessing the path
run["train/acc"] returns a
It's a reference that points to some location in the run – we just don't know what's there yet.
Let's assign something to the handler:
The handler now represents a
Once a field exists under the path which the handler points to, it only accepts the corresponding field methods:
Example: Handler becomes namespace#
In this example, we'll have an overall run structure that is identical to the example above, but our
Handler object will reference a different location in the run.
If we try accessing the path
Handler is returned, as that field does not exist yet and no value has been assigned to it:
We can now assign something to
handler_object. Let's assign the value
0.82 to the field
acc and assign that to the handler:
Since the handler object for
run["train"] has something nested under it – in this case, a
FloatSeries field – the handler now represents a namespace.
Pipeline logging example#
Let's say we have a data preprocessing step in our pipeline, and we would like to ensure that all of the metadata generated during this part is organized under a folder named "preprocessing".
We would start a Neptune run early in the script:
To collect the metadata from this step into one place, we set up a
preprocessing namespace inside the run. This will be the base namespace where all the preprocessing metadata is logged.
We define the path
run["preprocessing"] and assign it to a handler object, which we'll then use to log metadata as if it were a run.
This way, whenever we log something to
preprocessing_handler, it'll end up under the path
The handler exposes similar methods as other Neptune objects.
For details, see API reference Field types: Namespace handler.