Remote stop and abort#
You can stop or abort an active run in the runs table.
- When you stop a run, it's considered to be finished. You can use this option, for example, to stop a monitoring script or to signal that the training is sufficient.
- Aborting a run is similar to stopping it, but it will be marked as a failure (the state is set to
failed). You can use this option to manually mark a run as unsuccessful.
To send a remote stop or abort request:
- In the runs table, click the status icon next to the run ID.
- Select Remote stop or Remote abort.
Neptune sends the signal to all processes that have an active connection to the run. The exact behavior depends on whether the psutil library is installed on the machine running the process. (By default, the psutil library is installed with the Neptune client.)
- If psutil is installed, the current process and its children are aborted by sending
SIGTERM. If the processes are not terminated after the grace period,
- If psutil is not installed, we make a best effort to stop the process – for example, by invoking
os.exit()– but it's not guaranteed to work in every environment.