API reference: Kedro integration#
You can use the Kedro-Neptune plugin to log metadata about the Kedro pipeline execution to Neptune.
- For setup and usage, see Integrations ≫ Working with Kedro
You can configure where and how Kedro pipeline metadata is logged by editing the parameters in the
||Name of the Neptune project where metadata is stored.
If you haven't set the
||Base namespace (folder) where your metadata will be logged.|
||-||List of files you want to upload to Neptune.|
neptune: #GLOBAL CONFIG project: $NEPTUNE_PROJECT base_namespace: kedro enabled: true #LOGGING upload_source_files: - '**/*.py' - 'conf/base/*.yml'
Configuring Neptune API token#
You can configure where the Kedro-Neptune plugin will look for your Neptune API token in the
leave it empty, in which case Neptune will look for your token in the
There must be a dollar sign (
$) before the variable name.
pass a different environment variable, such as
- pass your token as a string, such as
How do I find my API token?
In the top-right corner of the Neptune app, click your avatar and select Get your API token.
You can copy your token from the dialog that opens. It's very long – make sure to copy and paste it in full!
CLI command for initializing the Kedro-Neptune plugin.
You can monitor, visualize, and compare your pipelines and node outputs in the Neptune app.
You can only use the
kedro neptune command in the Kedro project folder.
kedro neptune [OPTIONS] COMMAND [ARGS...]
||Shows CLI usage: options, commands and their arguments.|
kedro neptune init#
Initializes the plugin and configures Neptune credentials and logging options.
After executing this command, whenever you execute
kedro run, you will log:
- Pipeline execution configuration (
- Metadata about Kedro DataSet
- Hardware consumption and node execution time
- Configuration files from the conf/base directory
- Full Kedro run command
- Any additional metadata like metrics, charts, or images that you logged from inside of your node functions.
||User's Neptune API token, or the name of the environment variable where it is stored. You can find and copy your API token by clicking your avatar in the Neptune app.|
||Neptune project name, or the name of the environment variable where it is stored. You can find and copy your project name in the app, by going to Settings → Properties.|
||Namespace in Neptune where all the Kedro-related metadata is logged.|
||Name of the subdirectory inside of the Kedro
NeptuneFileDataSet is a Kedro Data Set that lets you log files to Neptune.
It can be any file on the POSIX compatible filesystem.
To log it, you need to define the
NeptuneFileDataSet in any Kedro catalog, including
||In POSIX format, filepath to a text file prefixed with a protocol, such as
||Credentials required to access the underlying filesystem. Same as for Kedro TextDataset.|
||Extra arguments to pass to underlying filesystem class constructor. Same as for Kedro TextDataset.|