X-coordinates (step) must be strictly increasing#
Error occurred during asynchronous operation processing: X-coordinates (step) must be strictly increasing for series attribute. <field_name>. Invalid point: <invalid step value>
This error occurs when you're creating a series field with a step value that is less than or equal to the value of the previous logged step. For the chart to be constructed correctly, step values must be strictly increasing.
You might run into this in the following scenarios:
- You're resuming an existing run and appending fresh values to a series field with steps already logged. Remember that
append()is iterative, so each call adds a value to the existing set of values.
- You have distributed systems logging to the same monitoring namespace. For example, two GPUs logging utilization to the same field.
Starting a new series#
If you want to resume a run and override an existing series field, you need to either:
- delete the field, or
- change the name of the field so that effectively a new field is created.
Now the step will be appended correctly.
Tips for distributed systems#
If logging of system metrics is the cause for the error, check the below resources for setup tips:
- How to use Neptune in distributed computing
- Use cases ≫ Track distributed training jobs with Neptune
- Learn how the monitoring namespace works: Log system metrics