Updating existing experiment

You can update experiments even after they finished running. This lets you add new data or visualizations to the previously closed experiment and makes multi-stage training convenient.

Why you may want to update an existing experiment?

Updating existing experiment can come in handy in several situations:

  • You want to add metrics or visualizations to the closed experiment.

  • You finished model training and closed the experiment earlier, but now you want to continue training from that moment. Actually, you can even make multiple iterations of the procedure: resume experiment -> log more data. Have a look at the simple example below for details.

How to update existing experiment?

To update the experiment you need to get the project where this experiment is. Then you need to get the Experiment object of the experiment you want to update.

import neptune

# Get project
project = neptune.init('my_workspace/my_project')

# Get experiment object for appropriate experiment, here 'SHOW-2066'
my_exp = project.get_experiments(id='SHOW-2066')[0]

Experiment with id='SHOW-2066' is now ready to be updated. Use my_exp to continue logging to it.

Note that with get_experiments() you can get experiments by id, state, owner, tag and min_running_time.

from neptunecontrib.api.table import log_chart

# Log metrics, images, text
my_exp.log_metric(...)
my_exp.log_image(...)
my_exp.log_text(...)

# Append tag
my_exp.append_tag('updated')

# Log new chart
log_chart('matplotlib-interactive', fig, my_exp)

Technique is the same as described in section about logging by using experiment object.

Note

You can retrieve an experiment and log more data to it multiple times.

Example below shows updated experiment with more data-points logged to the 'mse' metric and 'pretty-random-metric' added.

Charts in original experiment

Charts in original experiment

Charts in updated experiment

Charts in updated experiment

What you can/cannot update

You can freely use all Experiment methods. These include methods for logging more data:

Moreover, you can use all logging methods from neptunecontrib, that is:

Note

Learn more about logging options to see why and how to use each method.

However, updating experiment comes with some limitations. Specifically:

  • you cannot update parameters and source code, but you can upload sources as artifact, using log_artifact().

  • hardware consumption for the update will not be tracked.

  • stdout and stderr are not logged during update.

  • experiment status (failed/succeeded/aborted) will not be updated.