Setting a base namespace

You can organize your metadata into folder-like structures with the help of namespaces and handler objects.

If you assign a run namespace path to a variable, you get a namespace handler object. The namespace handler exposes similar methods as other Neptune objects – however, all field paths are relative to the namespace.

Start by creating a Handler object and turn it into a namespace by organizing metadata inside it:

Example

params_ns = run["params"]  # Create a namespace handler

params_ns["max_epochs"] = 20  # Log directly to the namespace instead of to the run
params_ns["batch_size"] = 32

Result

run
|--> params
     |--> max_epochs: 20
     |--> learning_rate: 0.005

In the pipeline logging example below, we show how to use a handler to organize metadata from a particular step of a run into a given namespace.

Pipeline logging example

Let's say we have a data preprocessing step in our pipeline, and we would like to ensure that all of the metadata generated during this part is organized under a folder named "preprocessing".

We would start a Neptune run early in the script:

import neptune
from sklearn.datasets import fetch_lfw_people
from utils import *

dataset = fetch_lfw_people(min_faces_per_person=70, resize=0.4)
run = neptune.init_run()
dataset_config = {...}

To collect the metadata from this step into one place, we set up a preprocessing namespace inside the run. This will be the base namespace where all the preprocessing metadata is logged.

We define the path run["preprocessing"] and assign it to a handler object, which we'll then use to log metadata as if it were a run.

Create namespace handler

preprocessing_handler = run["preprocessing"]

Log dataset details

preprocessing_handler["dataset/config"] = dataset_config

Preprocess dataset, then log scaler and features files

dataset_transform = Preprocessing(...)
preprocessing_handler["dataset/scaler"].upload(path_to_scaler)

This way, whenever we log something to preprocessing_handler, it'll end up under the path run["preprocessing/..."].

See full example script on GitHub 

Setting base namespace in integrations

Most integrations automatically organize the metadata logged by the integration framework into a dedicated namespace.

You can usually customize the name of this namespace with the help of a parameter.

Customize base namespace name with parameter

import neptune
from neptune.integrations.tensorflow_keras import NeptuneCallback

neptune_run = neptune.init_run()

model.fit(
     x_train,
     y_train,
     ...,
     callbacks=[NeptuneCallback(
          run=neptune_run,
          base_namespace="train",
     )],
)

Even if there is no automatic namespace implementation, whenever you can pass a Neptune Run object to a callback, you can instead pass a namespace handler.

Set base namespace with handler

neptune_run = neptune.init_run()

train_namespace = run["train"]

model.some_method(
     ...,
     callbacks=[NeptuneLogger(run=train_namespace)],
)

For the exact implementations, check the specific Integrations and their API references.