Skip to content

Too many fields in object#

Error

NeptuneFieldCountLimitExceedException: There are too many fields (more than 9000) in the <Neptune object>.

Issue: You're creating too many individual fields in a single Neptune object (run, model, etc.).

For example, when logging run["params/lr"] = 0.01, then "lr" is the field.

Workarounds

Note

The field limit is not about metadata limits per se.

For example, rather than assigning 100 values to 100 fields (run["field1"] = 0, run["field2"] = 1, ..., run["field100"] = 99) you could construct a series of those 100 values and log them under a single field:

for i in range(100):
    run["field"].log(i)

You can also do this with strings and images. For more, see Essential logging methods: Creating a series

If you exceed the field limit while logging, Neptune stops the synchronization to the server and stores the data locally. To continue uploading the metadata, you need to delete some excess fields from the object in question, then sync the data manually.

Counting fields#

If you want to count the fields of a Neptune object, you can use the get_structure() method and some scripting:

>>> import neptune.new as neptune
>>> def count_fields(d):
...     return sum([count_fields(v) if isinstance(v, dict)
...     else 1 for v in d.values()])
...
>>> run = neptune.init_run(
...     project="ml-team/classification",
...     with_id="CLS-34",
...     mode="read-only",
... )
https://app.neptune.ai/ml-team/classification/e/CLS-34
Remember to stop your run once youve finished logging your metadata...
>>> count_fields(run.get_structure())
571

Deleting fields#

You can delete fields or namespaces with the del command. For example, to delete the "training/checkpoints" namespace from a run with identifier "ACC-47":

import neptune.new as neptune

# Resume an existing run
run = neptune.init_run(with_id="CLS-47")

# Delete data
del run["training/checkpoints"]

Syncing data manually#

Once you're done deleting fields, synchronize the data manually:

  1. Navigate to the folder from which you executed the script. It should contain a .neptune folder.
  2. In a terminal, enter the following command:

    neptune sync
    

For further details, see the API reference:

Getting help