API reference: Kedro integration#

You can use the Kedro-Neptune plugin to log metadata about the Kedro pipeline execution to Neptune.

Configuration#

Logging configuration#

You can configure where and how Kedro pipeline metadata is logged by editing the parameters in the conf/base/neptune.yml file.

Options	Parameter	Default	Description
`neptune`	`project`	`$NEPTUNE_PROJECT`	Name of the Neptune project where metadata is stored. If you haven't set the `$NEPTUNE_PROJECT` environment variable, you can replace it with the full name of your Neptune project (`workspace-name/project-name`).
	`base_namespace`	`kedro`	Base namespace (folder) where your metadata will be logged.
	`dependencies`		Tracks environment requirements. If you pass `infer` to this argument, Neptune logs dependencies installed in the current environment. You can also pass a path to your dependency file directly. If left empty, no dependency file is uploaded.
	`enabled`	`true`	If Neptune logging is enabled.
	`upload_source_files`	`- '*/.py'` `- conf/base/*.yml`	List of files you want to upload to Neptune.

Example

conf/base/neptune.yml

neptune:
#GLOBAL CONFIG
  project: $NEPTUNE_PROJECT
  base_namespace: kedro
  dependencies: infer
  enabled: true

#LOGGING
  upload_source_files:
  - '**/*.py'
  - conf/base/*.yml

Configuring Neptune API token#

You can configure where the Kedro-Neptune plugin will look for your Neptune API token in the conf/local/credentials_neptune.yml file:

conf/local/credentials_neptune.yml

neptune:
  api_token: $NEPTUNE_API_TOKEN

You can:

leave it empty, in which case Neptune will look for your token in the $NEPTUNE_API_TOKEN environment variable.

Important

There must be a dollar sign ($) before the variable name.
pass a different environment variable, such as $MY_SPECIAL_NEPTUNE_API_TOKEN_VARIABLE.
pass your token as a string, such as eyJhcGlfYWRk...Imh0dHBzOi8v (not recommended).

How do I find my API token?

In the bottom-left corner of the Neptune app, open the user menu and select Get your API token.

You can copy your token from the dialog that opens. It's very long – make sure to copy and paste it in full!

`kedro neptune`#

CLI command for initializing the Kedro-Neptune plugin.

You can monitor, visualize, and compare your pipelines and node outputs in the Neptune app.

Note

You can only use the kedro neptune command in the Kedro project folder.

Command syntax: kedro neptune [OPTIONS] COMMAND [ARGS...]

Options	Description
`--help`	Shows CLI usage: options, commands and their arguments.

`kedro neptune init`#

Initializes the plugin and configures Neptune credentials and logging options.

After executing this command, whenever you execute kedro run, you will log:

Parameters
Pipeline execution configuration (run_params)
Metadata about Kedro Dataset
Hardware consumption and node execution time
Configuration files from the conf/base directory
Full Kedro run command
Any additional metadata like metrics, charts, or images that you logged from inside of your node functions.

Parameters

Name	Type	Default	Description
`api-token`	`str`, optional	`$NEPTUNE_API_TOKEN`	User's Neptune API token, or the name of the environment variable where it is stored. You can find your API token in the user menu, in the bottom-left corner of the app.
`project`	`str`, optional	`$NEPTUNE_PROJECT`	Neptune project name, or the name of the environment variable where it is stored. You can find and copy your project name in the app, in the top-right corner: → Edit project details.
`base-namespace`	`str`, optional	`experiment`	Namespace in Neptune where all the Kedro-related metadata is logged.
`config`	`str`, optional	`base`	Name of the subdirectory inside of the Kedro `conf` directory for configuration and catalog files.
`dependencies`	`str`, optional	`None`	Tracks environment requirements. If you pass `infer` to this argument, Neptune logs dependencies installed in the current environment. You can also pass a path to your dependency file directly. If left empty, no dependency file is uploaded.

Examples

Initialize with defaults

kedro neptune init

Initialize with more options

kedro neptune init \
    --api-token $NEPTUNE_API_TOKEN \
    --project workspace-name/project-name \
    --base-namespace my-namespace \
    --config my-config-location \
    --dependencies infer

`NeptuneFileDataset`#

NeptuneFileDataset is a Kedro Data Set that lets you log files to Neptune.

It can be any file on the POSIX compatible filesystem.

To log it, you need to define the NeptuneFileDataset in any Kedro catalog, including catalog.yml.

Parameters

Name	Type	Description
`filepath`	`str`	In POSIX format, filepath to a text file prefixed with a protocol, such as `s3://`. Same as for Kedro TextDataset.
`credentials`	`dict`, optional	Credentials required to access the underlying filesystem. Same as for Kedro TextDataset.
`fs_args`	`dict`, optional	Extra arguments to pass to underlying filesystem class constructor. Same as for Kedro TextDataset.

Examples

catalog.yml

predictions:
  type: kedro.extras.datasets.json.JSONDataset
  filepath: data/07_model_output/predictions.json

predictions@neptune:
  type: kedro_neptune.NeptuneFileDataset
  filepath: data/07_model_output/predictions.json