Skip to content

Namespaces and fields#

In a Neptune object, such as a Run, fields and namespaces provide a folder-like structure for the metadata you log.

  • A field represents a location for storing a piece of metadata.
  • Fields can be organized under namespaces.

A field can be flat (stored at the root of the object) or nested (stored under one or more namespaces).

object
|-- namespace
    |-- field
        |-- Float
    |-- field
        |-- FileSeries
|-- namespace
    |-- namespace
        |-- field
            |-- String
        |-- field
            |-- StringSeries

Example:

Python code
# Flat
run["f1_score"] = 0.66
run["model"].upload("model.pkl")

# Nested
run["params/batch_size"] = 64

# Nested twice
run[
    "params/optimization/learning_rate"
] = 0.001
run[
    "params/optimization/algorithm"
] = "Adam"
Resulting metadata structure
run
|-- "f1_score"
    |-- 0.66
|-- "model"
    |-- model.pkl
|-- "params"
    |-- "batch_size"
        |-- 64
    |-- "optimization"
        |-- "learning_rate"
            |-- 0.001
        |-- "algorithm"
            |-- "Adam"

Field types and limitations#

Fields can contain values of various types: strings, dictionaries, floats, files, or artifacts. A field can also contain a series of values.

Namespaces (folders) can contain multiple fields of various types, but a field is always fixed to a type.

Good to know

When you assign some metadata to a new path in the Neptune object, that creates a field of a certain type.

If you reassign a value of the same type, it'll overwrite the previous value stored in the field. Attempting to assign data of a different type to an existing field will result in an error.

Example
# Create new Float field
run["parameters/momentum"] = 0.9

# Update the value of the field
run["parameters/momentum"] = 0.8

# It's no longer possible to store a File under a Float field
run["parameters/momentum"].upload("sample.csv")  # Error

In the above example, in order to upload a file, you'd need to change the parameters/momentum path to something else. You could keep the "parameters" namespace, but at minimum, "momentum" should be changed to something different.

There are several options for storing metadata, such as simple assignment (=) for single values, upload() for uploading files, and track_files() for tracking artifact versions, to name a few. Each yields a different field type.

The most suitable logging method depends on:

  • The data type of metadata you're logging.

  • The options you want for visualizing, storing, and interacting with the data in Neptune.

    Example: For large files, you may want to simply track their metadata rather than upload them in full.

Example: How to work with fields#

How to log#

import neptune

run = neptune.init_run()
run["params/activation"] = "ReLU"

Result

  • Namespace: params
  • Field: activation
  • Field value: "ReLU"

When viewing the run in the app:

Text field in run metadata view

How to query#

To fetch the value of the field through the Neptune API:

>>> run = neptune.init_run(with_id="HELLO-23", mode="read-only") # (1)!
>>> activation = run["params/activation"].fetch()
>>> print(activation)
'ReLU'
  1. The Neptune ID of the run. It's stored in the sys/id field. You can also use a custom run ID.

How to overwrite#

To assign a new value to an existing field, just log a different value of the same type.

>>> run = neptune.init_run(with_id="HELLO-23")
>>> run["params/activation"] = "ReLU"
>>> run["params/activation"] = "ELU"  # Updates the field with the new value

How to delete#

Use del on any namespace or field to delete the metadata stored in it.

>>> run = neptune.init_run(with_id="HELLO-23")
>>> del run["params/activation"]

Field limits#

You can create at most 9000 fields in any one Neptune object that can contain metadata, such as a Run or Model object.

Note that the field limit is not about metadata limits per se.

For example, rather than assigning 100 values to 100 fields (run["field1"] = 0, run["field2"] = 1, ..., ) you could construct a series of those 100 values and log them under a single field:

for i in range(100):
    run["field"].append(i)

You can log numerical values, strings, and files as a series. For details, see Logging series of values or files.

Using namespace handlers#

To make metadata organization easier, you can create a namespace handler object. This way, you can operate on the object like a run, but all the metadata is logged under the specified namespace.

To create a namespace handler, assign a run namespace path to a variable:

import neptune

run = neptune.init_run()

params_namespace = run["params"]

params_namespace["learning_rate"] = 0.005

To learn more, see Setting a base namespace.

Related