You can update runs even after they finished running. This lets you add new data or visualizations to the previously closed run and makes multi-stage training convenient.
Updating existing run can come in handy in several situations:
You want to add metrics or visualizations to the closed run.
You finished model training and closed the run earlier, but now you want to continue training from that moment. Actually, you can even make multiple iterations of the procedure:
resume run -> log more data. Have a look at the simple example below for details.
run parameter set to the existing project name and run's ID. Now you can continue working with the run's metadata as if you just started a brand new run.
Look at the following snippet of code, that illustrates the idea:
import neptune.new as neptune# SUN-123 is the run you want to resumerun = neptune.init(project="my_workspace/my_project", run="SUN-123")# download snapshot of model weightsmodel = run["train/model_weights"].download()# 450 is the epoch from where you want to resume training processcheckpoint = 450# continue training as usualfor epoch in range(checkpoint, 1000):run["train/accuracy"].log(0.75)# ...
When you resume a run, the Python file from which the run was resumed is not uploaded by default, but you can upload it by providing the
source_files parameter in the .init() function.