Namespaces and fields

In a Neptune object, such as a run, fields and namespaces provide a folder-like structure for the metadata you log.

• A field represents a location for storing a piece of metadata.
• Fields can be organized under namespaces.

A field can be flat (stored at the root of the object) or nested (stored under one or more namespaces).

object
|-- namespace
    |-- field
        |-- Float
    |-- field
        |-- FileSeries
|-- namespace
    |-- namespace
        |-- field
            |-- String
        |-- field
            |-- StringSeries

Example:

Python code

# Flat
run["f1_score"] = 0.66
run["model"].upload("model.pkl")

# Nested
run["params/batch_size"] = 64

# Nested twice
run[
    "params/optimization/learning_rate"
] = 0.001
run[
    "params/optimization/algorithm"
] = "Adam"

Resulting metadata structure

run
|-- "f1_score"
    |-- 0.66
|-- "model"
    |-- model.pkl
|-- "params"
    |-- "batch_size"
        |-- 64
    |-- "optimization"
        |-- "learning_rate"
            |-- 0.001
        |-- "algorithm"
            |-- "Adam"

Field types and limitations

Fields can contain values of various types: strings, dictionaries, floats, files, or artifacts. A field can also contain a series of values.

Namespaces (folders) can contain multiple fields of various types, but a field is always fixed to a type.

Good to know

When you assign some metadata to a new path in the Neptune object, that creates a field of a certain type.

If you reassign a value of the same type, it'll override the previous value stored in the field. Attempting to assign data of a different type to an existing field will result in an error.

Example

# Create new Float field
run["parameters/momentum"] = 0.9

# Update the value of the field
run["parameters/momentum"] = 0.8

# It's no longer possible to store a File under a Float field
run["parameters/momentum"].upload("sample.csv")  # Error

In the above example, in order to upload a file, you'd need to change the "parameters/momentum" path to something else. You could keep the "parameters" namespace, but at minimum, "momentum" should be changed to something different.

There are several options for storing metadata, such as simple assignment (=) for single values, upload() for uploading files, and track_files() for tracking file metadata, to name a few. Each yields a different field type.

The most suitable logging method depends on:

• The data type of metadata you're logging.

• The options you want for visualizing, storing, and interacting with the data in Neptune.

  Example: For large files, you may want to simply track their metadata rather than upload them in full.

Learn more:

• Essential logging methods
• Field types reference
• What you can log and display

Field limit per object

You can create at most 9000 fields in any one Neptune object that can contain metadata, such as a Run or Model object.

Note that the field limit is not about metadata limits per se.

For example, rather than assigning 100 values to 100 fields (run["field1"] = 0, run["field2"] = 1, ..., ) you could construct a series of those 100 values and log them under a single field:

for i in range(100):
    run["field"].append(i)

You can log numerical values, strings, and files as a series. For details, see the Field types reference.