Kedro

You can use Kedro-Neptune plugin to log metadata about the Kedro pipeline execution to Neptune.

You can find detailed information on how to install and use the integration in the user guide.

kedro neptune

Command line interface (CLI) command for initializing Kedro-Neptune plugin.

Kedro-Neptune plugin lets you log metadata related to Kedro pipelines to Neptune so that you can monitor, visualize, and compare your pipelines and node outputs in the Neptune UI.

You can only use kedro neptune command in the Kedro project folder.

kedro neptune [OPTIONS] COMMAND [ARGS]...

Options

--help

Shows CLI usage - options, commands and their arguments

kedro neptune init

Initialize the plugin, and configure Neptune credentials and logging options.

After initializing it, whenever you run $ kedro run, you will log:

  • parameters

  • pipeline execution configuration (run_params)

  • metadata about Kedro DataSet

  • hardware consumption and node execution time

  • configuration files from the conf/base directory

  • full Kedro run command

  • any additional metadata like metrics, charts, or images that you logged from inside of your node functions.

Options

api-token

(str , optional) Neptune API token or the environment variable name where it is stored. Default is '$NEPTUNE_API_TOKEN'. See How to find your API token.

project

(str, optional) Neptune project name or the environment variable name where it is stored. Default is '$NEPTUNE_PROJECT'. See How to find your Neptune project name.

base-namespace

(str, optional) Namespace in Neptune where all the Kedro-related metadata is logged. Default is 'kedro'.

config

(str, optional) Name of the Subdirectory inside of the Kedro 'conf' directory for configuration and catalog files. Default is 'base'.

Examples

kedro neptune init
kedro neptune init --api-token $NEPTUNE_API_TOKEN --project common/kedro-integration

NeptuneFileDataset

NeptuneFileDataSet is a Kedro Data Set that lets you log files to Neptune. It can be any file on the POSIX compatible filesystem. To log it, you need to define the NeptuneFileDataSet in any Kedro catalog, including catalog.yml.

Parameters

filepath

(str) Filepath in POSIX format to a text file prefixed with a protocol like s3://. Same as for Kedro TextDataset.

credentials

(dict, optional) Credentials required to get access to the underlying filesystem. Same as for Kedro TextDataset.

fs_args

(dict, optional) Extra arguments to pass into underlying filesystem class constructor. Same as for Kedro TextDataset.

Examples

catalog.yml
predictions:
type: kedro.extras.datasets.json.JSONDataSet
filepath: data/07_model_output/predictions.json
type: kedro_neptune.NeptuneFileDataSet
filepath: data/07_model_output/predictions.json