Skip to content

Catalyst integration guide#

Open in Colab

Custom dashboard displaying metadata logged with Catalyst

Neptune compatibility note

This integration has not yet been updated for neptune 1.x and requires using neptune-client <1.0.0.

Catalyst is a PyTorch framework for Deep Learning R&D. Neptune support is implemented as a logger in Catalyst. With the Neptune logger, you can automatically track:

  • Metrics
  • Hyperparameters
  • Images
  • Artifacts (such as videos, audio, model checkpoints, and files)
  • Hardware consumption metrics
  • stdout and stderr streams
  • Training code and Git information

See in Neptune  Code examples 

Quickstart#

Tip

This section is for Catalyst users who are familiar with loggers, like CSV logger or TensorBoard logger.

NeptuneLogger is part of the Catalyst library. To start logging, create a Neptune logger and pass it to the runner:

  1. Create the logger:

    import neptune  # (1)!
from catalyst import dl

neptune_logger = dl.NeptuneLogger(
    api_token=neptune.ANONYMOUS_API_TOKEN,  # (2)!
    project="common/catalyst-integration",  # (3)!
    tags=["pretraining", "retina"],  # optional
)

    1. Only needed to access the anonymous API token. If you're logging to your own project, you can omit the import.

    2. The api_token argument is included to enable anonymous logging.

      Once you register, you should leave the token out of your script and instead save it as an environment variable.

    3. Projects in the common workspace are public and can be used for testing. To log to your own workspace, pass the full name of your Neptune project: workspace-name/project-name. For example, "ml-team/classification".

      To copy the name, click the menu in the top-right corner and select Edit project details.

    There are further ways to customize the behavior of the logger. For details, see the Catalyst API reference .

  2. Pass the logger to the runner:

    Pass to SupervisedRunner
    my_runner = dl.SupervisedRunner()

my_runner.train(
    loggers={"neptune": neptune_logger},
    ...
)
    Pass to custom Runner
    class CustomRunner(dl.IRunner):
    ...
    def get_loggers(self):
        return {
            "console": dl.ConsoleLogger(),
            "neptune": neptune_logger
        }

runner = CustomRunner().run()

The Neptune logger setup is complete and you can run your scripts without additional changes.

Your metadata will be logged in the Neptune project for further analysis, comparison, and collaboration.

Full walkthrough#

This guide walks you through connecting NeptuneLogger to your machine-learning scripts and using it in your experimentation.

Before you start#

  • Sign up at neptune.ai/register.
  • Create a project for storing your metadata.
  • Have Catalyst installed.

  • Have version 0.16.18 of neptune-client installed.

    pip install -U "neptune-client<1.0.0"

    conda install -c conda-forge neptune-client=0.16.18
    Passing your Neptune credentials

    Once you've registered and created a project, set your Neptune API token and full project name to the NEPTUNE_API_TOKEN and NEPTUNE_PROJECT environment variables, respectively.

    export NEPTUNE_API_TOKEN="h0dHBzOi8aHR0cHM.4kl0jvYh3Kb8...6Lc"

    To find your API token: In the bottom-left corner of the Neptune app, expand the user menu and select Get my API token.

    export NEPTUNE_PROJECT="ml-team/classification"

    To find your project: Your full project name has the form workspace-name/project-name. To copy the name, click the menu in the top-right corner and select Edit project details.

    While it's not recommended especially for the API token, you can also pass your credentials in the code when initializing Neptune.

    run = neptune.init_run(
    project="ml-team/classification",  # your full project name here
    api_token="h0dHBzOi8aHR0cHM6Lkc78ghs74kl0jvYh...3Kb8",  # your API token here
)

    For more help, see Set Neptune credentials.

If you'd rather follow the guide without any setup, you can run the example in Colab .

Adding NeptuneLogger to the Catalyst script#

Catalyst has a unified way of logging metadata, by using loggers.

You can learn more about Catalyst Loggers in the Catalyst docs .

To start logging, create a Neptune logger and pass it to the runner:

  1. Create a NeptuneLogger instance.

    from catalyst import dl
import neptune.new as neptune

neptune_logger = dl.NeptuneLogger()  # (1)!

    1. If you haven't set up your credentials, you can log anonymously:

      neptune_logger = dl.NeptuneLogger(
    api_token=neptune.ANONYMOUS_API_TOKEN,
    project="common/catalyst-integration",
)

    There are further ways to customize the behavior of the logger. For details, see the Catalyst API reference .

  2. Pass neptune_logger to the runner.

    The example below uses SupervisedRunner as an example:

    # Create runner
my_runner = dl.SupervisedRunner()
my_runner.train(loggers={"neptune": neptune_logger}, ...)

    The Neptune logger is now ready.

  3. Once you're done logging, stop tracking the run.

    neptune_logger.run.stop()

  4. Run your script:

    python main.py
If Neptune can't find your project name or API token

As a best practice, you should save your Neptune API token and project name as environment variables. However, you can also pass them as arguments when you're using a function that takes api_token and project as parameters:

  • api_token="Your Neptune API token here"
    • Find and copy your API token by expanding your user menu and selecting Get my API token.
  • project="workspace-name/project-name""
    • Find and copy your project name in the top-right menu: → Edit project details.

For example:

Neptune client library
model_version = neptune.init_model_version(
    api_token="h0dHBzOi8aHR0cHM6Lkc78ghs74kl0jvYh3Kb8",
    project="ml-team/named-entity-recognition",
    model= ...,
)
Neptune integration
neptune_logger = dl.NeptuneLogger(
    api_token="h0dHBzOi8aHR0cHM6Lkc78ghs74kl0jvYh3Kb8",
    project="ml-team/named-entity-recognition",
)

Analyzing the logged metadata in Neptune#

Your metadata will be logged in the given Neptune project for analysis, comparison, and collaboration.

To open the run, click the Neptune link that appears in the console output.

Example link: https://app.neptune.ai/o/common/org/catalyst-integration/e/CATALYST-38823/metadata

You can also open the project and look for your run in the Runs section.

Logging best model#

After training, use the log_artifact() method to log a model checkpoint.

my_runner.log_artifact(
    path_to_artifact="./checkpoints/model.best.pth",
    tag="best_model",
    scope="experiment",
)

Manually logging metadata#

If you have other types of metadata that are not covered in this guide, you can still log them using the Neptune client library.

When you initialize the run, you get a run object, to which you can assign different types of metadata in a structure of your own choosing.

import neptune

# Create a new Neptune run
run = neptune.init_run()

# Log metrics or other values inside loops
for epoch in range(n_epochs):
    ...  # Your training loop

    run["train/epoch/loss"].append(loss)  # Each append() appends a value
    run["train/epoch/accuracy"].append(acc)

# Upload files
run["test/preds"].upload("path/to/test_preds.csv")

# Track and version artifacts
run["train/images"].track_files("./datasets/images")

# Record numbers or text
run["tokenizer"] = "regexp_tokenize"

Related