Skip to content

Namespaces and fields#

In a Neptune Run object, fields and namespaces provide a folder-like structure for the metadata you log.

  • A field represents a location for storing a piece of metadata.
  • Fields can be organized under namespaces.

A field can be flat (stored at the root of the object) or nested (stored under one or more namespaces).

run
|-- namespace
    |-- field
        |-- Float
    |-- field
        |-- FileSeries
|-- namespace
    |-- namespace
        |-- field
            |-- String
        |-- field
            |-- StringSeries

Example:

Python code
# Flat
run["f1_score"] = 0.66
run["model"].upload("model.pkl")

# Nested
run["params/batch_size"] = 64

# Nested twice
run[
    "params/optimization/learning_rate"
] = 0.001
run[
    "params/optimization/algorithm"
] = "Adam"
Resulting metadata structure
run
|-- "f1_score"
    |-- 0.66
|-- "model"
    |-- model.pkl
|-- "params"
    |-- "batch_size"
        |-- 64
    |-- "optimization"
        |-- "learning_rate"
            |-- 0.001
        |-- "algorithm"
            |-- "Adam"

Field types and limitations#

Fields can contain values of various types: strings, dictionaries, floats, files, or artifacts. A field can also contain a series of values.

Namespaces (folders) can contain multiple fields of various types, but a field is always fixed to a type.

There are several options for storing metadata, such as simple assignment (=) for single values, upload() for uploading files, and track_files() for tracking artifact versions, to name a few. Each yields a different field type.

The most suitable logging method depends on:

  • The data type of metadata you're logging.

  • The options you want for visualizing, storing, and interacting with the data in Neptune.

    Example: For large files, you may want to simply track their metadata rather than upload them in full.

Examples: How to work with fields#

How to log#

  • What you log:

    import neptune

run = neptune.init_run()
run["params/activation"] = "ReLU"

  • What you see:

    Text field in run metadata view

How to query#

To fetch the value of the field through the Neptune API:

>>> run = neptune.init_run(with_id="HELLO-23", mode="read-only") # (1)!
>>> activation = run["params/activation"].fetch()
>>> print(activation)
'ReLU'
  1. The Neptune ID of the run. It's stored in the sys/id field. You can also use a custom run ID.

How to overwrite#

To assign a new value to an existing field, log a different value of the same type:

>>> run = neptune.init_run(with_id="HELLO-23")
>>> run["params/activation"] = "ReLU"
>>> run["params/activation"] = "ELU"  # Updates the field with the new value

Good to know

Attempting to assign data of a different type to an existing field will result in an error.

Example
# Create new Float field
run["parameters/momentum"] = 0.9

# It's no longer possible to store a File under a Float field
run["parameters/momentum"].upload("sample.csv")  # Error

In the above example, in order to upload a file, you must change the parameters/momentum path to something different. You can keep the "parameters" namespace, but at minimum, you must change "momentum".

How to delete#

Use del on any namespace or field to delete the metadata stored in it.

>>> run = neptune.init_run(with_id="HELLO-23")
>>> del run["params/activation"]

Field limits#

You can create at most 9000 fields in one Neptune run.

Note that the field limit is not about metadata limits per se. For example, rather than assigning 100 values to 100 fields (run["field1"] = 0, run["field2"] = 1, ..., ) you could construct a series of those 100 values and log them under a single field:

for i in range(100):
    run["field"].append(i)

You can log numerical values, strings, and files as a series. For details, see Logging series of values or files.

Using namespace handlers#

To make metadata organization easier, you can create a namespace handler object. This way, you can operate on the object like a run, but all the metadata is logged under the specified namespace.

To create a namespace handler, assign a run namespace path to a variable:

import neptune

run = neptune.init_run()

params_namespace = run["params"]

params_namespace["learning_rate"] = 0.005

To learn more, see Setting a base namespace.

Comparing field values#

You can view, track, and compare the logged values in the Experiments tab in the Neptune app.

Note that in the runs table, Neptune compares values of fields that have the same data type. For example, if a field named batch_size stores an integer in RUN-1 but a float in RUN-2, Neptune treats them as two different fields that aren't comparable. As a result, the runs table will have two columns named batch_size, one for each data type.

Related