You can update runs even after they finished running. This lets you add new data or visualizations to the previously closed run and makes multi-stage training convenient.
Updating existing run can come in handy in several situations:
You want to add metrics or visualizations to the closed run.
You finished model training and closed the run earlier, but now you want to continue training from that moment. Actually, you can even make multiple iterations of the procedure:
resume run -> log more data. Have a look at the simple example below for details.
run parameter set to the existing project name and run's ID. Now you can continue working with the run's metadata as if you just started a brand new run.
Look at the following snippet of code, that illustrates the idea:
import neptune# SUN-123 is the run you want to resumerun = neptune.init(project="my_workspace/my_project", run="SUN-123")# download snapshot of model weightsmodel = run["train/model_weights"].download()# 450 is the epoch from where you want to resume training processcheckpoint = 450# continue training as usualfor epoch in range(checkpoint, 1000):run["train/accuracy"].log(0.75)# ...
Each time you resume a run, Python file from which run was started is uploaded: see example.