Skip to content

Timestamp must be non-decreasing#

Error

Error occurred during asynchronous operation processing: Timestamp must be non-decreasing for series attribute

Issue#

This error occurs when you're creating a series field with a timestamp value that is less than or equal to the value of the previous logged value.

You might run into this if your logging is distributed over multiple GPUs.

Solution#

This issue should generally not cause problems and can be ignored.

If you suspect hardware monitoring is the cause for the error, check the below resources for tips:

Filtering out the error#

Instead of turning off logging completely, you can filter out this error by adding the below snippet to your scripts:

import logging


class _FilterCallback(logging.Filterer):
    def filter(self, record: logging.LogRecord):
        return not (
            record.name == "neptune"
            and record.getMessage().startswith(
                "Error occurred during asynchronous operation processing:"
            )
        )


neptune.internal.operation_processors.async_operation_processor.logger.addFilter(
    _FilterCallback()
)

The above filters out both of the following messages:

  • Error occurred during asynchronous operation processing: Timestamp must be non-decreasing for series attribute
  • Error occurred during asynchronous operation processing: X-coordinates (step) must be strictly increasing for series attribute

To filter out either one of the above, modify the example startswith() definition accordingly.

Related

Getting help