Sacred

Sacred is a tool to configure, organize, log, and reproduce computational experiments. It is designed to introduce only minimal overhead, while encouraging modularity and configurability of experiments.

What you will get with this integration?

With Neptune + Sacred integration the following metadata will be logged automatically for you:

  • Hyper-parameters

  • Losses & metrics

  • Training code(Python scripts or Jupyter notebooks) and git information

  • Dataset version

  • Model Configuration

You can also log other metadata types like interactive charts, video, audio, and more. See What can you log and display?

Step 0: Before you start

You need to have Python 3.6+ and install the following libraries installed:

  • neptune-client[sacred]

  • sacred

  • torch

  • torchvision

To install them just go to you console and run:

pip install neptune-client[sacred] sacred torch torchvision

This code example was tested using:

neptune-client[sacred]==0.9.19 sacred==0.8.2 torch==1.9.0 torchvision==0.10.0

Step 1: Initialize a Neptune Run

Place this code snippet at the beginning of your script or notebook cell

import neptune.new as neptune
run = neptune.init(project = '<YOUR_WORKSPACE/YOUR_PROJECT>',
api_token = '<YOURR_API_TOKEN>')

This opens a new Run in Neptune that allows you to log various objects.

You need to authenticate yourself and open an existing project. Here is how:

You can use the api_token='ANONYMOUS' and project='common/sacred-integration' to explore without having to create a Neptune account.

Step 2: Add NeptuneObserver() to your sacred experiment's observers

# Create sacred experiment
ex = Experiment('image_classification', interactive=True)
# Add NeptuneObserver
ex.observers.append(NeptuneObserver(run=neptune_run))

Using NeptuneObsever the following is automatically logged to Neptune UI for you:

  • Hyperparameters

  • Loss

  • Metrics

Step 3: Run your experiment and explore metadata in Neptune UI

After you define your @ex.config(hyperparameters and configuration) and @ex.main(training loop) you can run your experiment as you normally would.

ex.run()

After running your script or notebook cell you will get a link similar to: https://app.neptune.ai/o/common/org/sacred-integration/e/SAC-11/ with common/sacred-integration replaced by your project, and SAC-11 replaced by your run.

Click on the link to open the Run in Neptune to watch your model training live.

Initially, it may be empty but keep the tab with the Run open to see your experiment metadata update in real-time.

Step 4: Log artifacts

When sacred.Experiment.add_artifact() is called with a filename and optionally a name this will trigger an event in the Neptune0bserver to upload the file to Neptune.

ex.add_artifact(filename='model_weights', name=f'./model.pth')

The same goes for sacred resources.

Step 5: Stop Neptune Run

run.stop()

Once you are done logging, you should stop tracking the Run using the stop() method. This is only needed when logging from a notebook or some other interactive environment.

How to ask for help?

Please visit the Getting help page. Everything regarding support is there

What’s next