Namespaces and fields#
In a Neptune object, such as a
Run, fields and namespaces provide a folder-like structure for the metadata you log.
- A field represents a location for storing a piece of metadata.
- Fields can be organized under namespaces.
A field can be flat (stored at the root of the object) or nested (stored under one or more namespaces).
Field types and limitations#
Fields can contain values of various types: strings, dictionaries, floats, files, or artifacts. A field can also contain a series of values.
Namespaces (folders) can contain multiple fields of various types, but a field is always fixed to a type.
Good to know
When you assign some metadata to a new path in the Neptune object, that creates a field of a certain type.
If you reassign a value of the same type, it'll overwrite the previous value stored in the field. Attempting to assign data of a different type to an existing field will result in an error.
In the above example, in order to upload a file, you'd need to change the
parameters/momentum path to something else. You could keep the "parameters" namespace, but at minimum, "momentum" should be changed to something different.
There are several options for storing metadata, such as simple assignment (
=) for single values,
upload() for uploading files, and
track_files() for tracking artifact versions, to name a few. Each yields a different field type.
The most suitable logging method depends on:
- The data type of metadata you're logging.
The options you want for visualizing, storing, and interacting with the data in Neptune.
Example: For large files, you may want to simply track their metadata rather than upload them in full.
Example: How to work with fields#
How to log#
How to query#
To fetch the value of the field through the Neptune API:
- The Neptune ID of the run. It's stored in the
sys/idfield. You can also use a custom run ID.
How to overwrite#
To assign a new value to an existing field, just log a different value of the same type.
How to delete#
del on any namespace or field to delete the metadata stored in it.
You can create at most 9000 fields in any one Neptune object that can contain metadata, such as a
Note that the field limit is not about metadata limits per se.
For example, rather than assigning 100 values to 100 fields (
run["field1"] = 0,
run["field2"] = 1, ..., ) you could construct a series of those 100 values and log them under a single field:
You can log numerical values, strings, and files as a series. For details, see Logging series of values or files.
Using namespace handlers#
To make metadata organization easier, you can create a namespace handler object. This way, you can operate on the object like a run, but all the metadata is logged under the specified namespace.
To create a namespace handler, assign a run namespace path to a variable:
To learn more, see Setting a base namespace.