Too many fields in object#
NeptuneFieldCountLimitExceedException: There are too many fields (more than 9000) in the <Neptune object>.
Issue: You're creating too many individual fields in a single Neptune object (run, model, etc.).
For example, when logging
run["params/lr"] = 0.01, then "lr" is the field.
The field limit is not about metadata limits per se.
For example, rather than assigning 100 values to 100 fields (
run["field1"] = 0,
run["field2"] = 1, ...,
run["field100"] = 99) you could construct a series of those 100 values and log them under a single field:
You can also do this with strings and images. For more, see Essential logging methods: Creating a series
If you exceed the field limit while logging, Neptune stops the synchronization to the server and stores the data locally. To continue uploading the metadata, you need to delete some excess fields from the object in question, then sync the data manually.
If you want to count the fields of a Neptune object, you can use the
get_structure() method and some scripting:
>>> import neptune.new as neptune >>> def count_fields(d): ... return sum([count_fields(v) if isinstance(v, dict) ... else 1 for v in d.values()]) ... >>> run = neptune.init_run( ... project="ml-team/classification", ... with_id="CLS-34", ... mode="read-only", ... ) https://app.neptune.ai/ml-team/classification/e/CLS-34 Remember to stop your run once you’ve finished logging your metadata... >>> count_fields(run.get_structure()) 571
You can delete fields or namespaces with the
del command. For example, to delete the "training/checkpoints" namespace from a run with identifier "ACC-47":
Syncing data manually#
Once you're done deleting fields, synchronize the data manually:
- Navigate to the folder from which you executed the script. It should contain a
In a terminal, enter the following command:
For further details, see the API reference: