Skip to content

Field count limit exceeded#

Error

NeptuneFieldCountLimitExceedException: There are too many fields (more than 9000) in the <Neptune object>.

Issue: You're creating too many individual fields in a single Neptune object (run, model, etc.). The limit for a single run or other Neptune object is 9000 unique fields.

For example, when logging run["params/lr"] = 0.01, then "lr" is the field.

Workarounds

Note

The field limit is not about metadata limits per se.

For example, rather than assigning 100 values to 100 fields (run["field1"] = 0, run["field2"] = 1, ..., run["field100"] = 99) you could construct a series of those 100 values and log them under a single field:

for i in range(100):
    run["field"].append(i)

You can also do this with strings and images. For more, see Essential logging methods: Creating a series

If you exceed the field limit while logging, Neptune stops the synchronization to the server and stores the data locally. To continue uploading the metadata, you need to delete some excess fields from the object in question, then sync the data manually.

Counting fields#

If you want to count the fields of a Neptune object, you can use the get_structure() method and some scripting:

>>> import neptune
>>> def count_fields(d):
...     return sum([count_fields(v) if isinstance(v, dict)
...     else 1 for v in d.values()])
...
>>> run = neptune.init_run(
...     project="ml-team/classification",
...     with_id="CLS-34",
...     mode="read-only",
... )
https://app.neptune.ai/ml-team/classification/e/CLS-34/metadata
...
>>> count_fields(run.get_structure())
571

Deleting fields#

You can delete fields or namespaces with the del command. For example, to delete the "training/checkpoints" namespace from a run with identifier "ACC-47":

import neptune

# Resume an existing run
run = neptune.init_run(with_id="CLS-47")

# Delete data
del run["training/checkpoints"]
How do I find the ID?

The Neptune ID is a unique identifier for the run. In the table view, it's displayed in the leftmost column.

The ID is stored in the system namespace (sys/id).

If the run is active, you can obtain its ID with run["sys/id"].fetch(). For example:

>>> run = neptune.init_run(project="ml-team/classification")
>>> run["sys/id"].fetch()
'CLS-26'

Syncing data manually#

Once you're done deleting fields, synchronize the data manually:

  1. Navigate to the folder from which you executed the script. It should contain a .neptune folder.
  2. In a terminal, enter the following command:

    neptune sync
    

For further details, see the API reference:

Getting help