X-coordinates (step) must be strictly increasing#
Error occurred during asynchronous operation processing: X-coordinates (step) must be strictly increasing for series attribute. <field_name>. Invalid point: <invalid step value>
This error occurs when you're creating a series field with a step value that is less than or equal to the value of the previous logged step. For the chart to be constructed correctly, step values must be strictly increasing.
You might run into this in the following scenarios:
- You're resuming an existing run and appending fresh values to a series field with steps already logged. Remember that
append()is iterative, so each call adds a value to the existing set of values.
- You have distributed systems logging to the same monitoring namespace. For example, two GPUs logging utilization to the same field.
Starting a new series#
If you want to resume a run and overwrite an existing series field, you need to either:
- delete the field, or
- change the name of the field so that effectively a new field is created.
Now the step will be appended correctly.
Tips for distributed systems#
If logging of system metrics is the cause for the error, check the below resources for tips:
- Best practices
- Tracking distributed training jobs with Neptune
- Learn how the monitoring namespace works: Logging system metrics