Skip to content

Working with Sacred#

Open in Colab

Custom dashboard displaying metadata logged with Sacred

Sacred is a tool to configure, organize, log, and reproduce computational experiments.

With the Neptune–Sacred integration, you can log the following metadata automatically:

  • Hyperparameters
  • Metrics and losses
  • Training code and Git information
  • Dataset version
  • Model configuration

See example in Neptune  Code examples 

Related

Before you start#

Tip

If you'd rather follow the guide without any setup, you can run the example in Colab .

Installing the Neptune–Sacred integration#

On the command line or in a terminal app, such as Command Prompt, enter the following:

pip install -U neptune-sacred

Sacred logging example#

Track your metadata with Neptune by adding a NeptuneObserver to the observers of your Sacred experiment.

  1. Create a run:

    import neptune.new as neptune

run = neptune.init_run()  # (1)!
    1. If you haven't set up your credentials, you can log anonymously: neptune.init_run(api_token=neptune.ANONYMOUS_API_TOKEN, project="common/sacred-integration")

  2. Create a Sacred experiment:

    from sacred import Experiment

ex = Experiment("image_classification")  # (1)!
    1. If you're in an interactive environment such as Jupyter Notebook, you need to add the argument interactive=True to the Experiment constructor. For details about this safeguard, see the Sacred documentation .

  3. Add a NeptuneObserver instance to the observers of the experiment and pass the created run:

    from neptune.new.integrations.sacred import NeptuneObserver

ex.observers.append(NeptuneObserver(run=run))

  4. Define your @ex.config (hyperparameters and configuration) and @ex.main (training loop).

  5. Run your experiment as you normally would.

    To open the run, click the Neptune link that appears in the console output.

    Example link: https://app.neptune.ai/o/common/org/sacred-integration/e/SAC-1341

Select the Charts section to view the model training metrics live, or create a custom dashboard.

Stop the run when done

Once you are done logging, you should stop the Neptune run. You need to do this manually when logging from a Jupyter notebook or other interactive environment:

run.stop()

If you're running a script, the connection is stopped automatically when the script finishes executing. In notebooks, however, the connection to Neptune is not stopped when the cell has finished executing, but rather when the entire notebook stops.

More options#

Logging artifacts#

When you call sacred.Experiment.add_artifact() with a filename and optionally a name, this triggers an event in the NeptuneObserver to upload the file to Neptune.

ex.add_artifact(filename=f"./model.pth", name="model_weights")

The same applies to Sacred resources. For details, see the Sacred documentation .

Manually logging metadata#

If you have other types of metadata that are not covered in this guide, you can still log them using the Neptune client library.

When you initialize the run, you get a run object, to which you can assign different types of metadata in a structure of your own choosing.

import neptune

# Create a new Neptune run
run = neptune.init_run()

# Log metrics or other values inside loops
for epoch in range(n_epochs):
    ...  # Your training loop

    run["train/epoch/loss"].append(loss)  # Each append() appends a value
    run["train/epoch/accuracy"].append(acc)

# Upload files
run["test/preds"].upload("path/to/test_preds.csv")

# Track and version artifacts
run["train/images"].track_files("./datasets/images")

# Record numbers or text
run["tokenizer"] = "regexp_tokenize"

Related