Timestamp must be non-decreasing

Error

Error occurred during asynchronous operation processing: Timestamp must be non-decreasing for series attribute

Issue

This error occurs when you're creating a series field with a timestamp value that is less than or equal to the value of the previous logged value.

You might run into this if your logging is distributed over multiple GPUs.

Solution

This issue should generally not cause problems and can be ignored.

If you suspect hardware monitoring is the cause for the error, check the below resources for tips:

• Best practices
• Tracking distributed training jobs with Neptune
• Learn how the monitoring namespace works: Logging system metrics

Related

• Learn more about the step argument: Log custom x values for graph
• API references for series fields:
  • FileSeries
  • FloatSeries
  • StringSeries
• Trash and delete data

Getting help