Kedro

​​
​​
​​See code examples on GitHub​
​​
​​
​​See runs logged to Neptune​
What will you get with this integration?
​Kedro is a popular open-source project that helps standardize ML workflows. It gives you a clean and powerful pipeline abstraction where you put all your ML code logic.
​Kedro-Neptune plugin lets you have all the benefits of a nicely organized kedro pipeline with a powerful user interface built for ML metadata management that lets you:
browse, filter, and sort your model training runs
compare nodes and pipelines on metrics, visual node outputs, and more
display all pipeline metadata including learning curves for metrics, plots, and images, rich media like video and audio or interactive visualizations from Plotly, Altair, or Bokeh
and do whatever else you would expect from a modern ML metadata store
Kedro-Neptune plugin supports distributed pipeline execution and works in Kedro setups that use orchestrators like Airflow or Kubeflow.
Installation
Before you start, make sure that:
You have Python 3.7+ in your system,
You are already a registered user so that you can log metadata to your private projects.
You have your Neptune API token set to the NEPTUNE_API_TOKEN environment variable.
Install neptune-client, kedro, and kedro-neptune
Depending on your operating system open a terminal or CMD and run this command. All required libraries are available via pip and conda:
pip
conda
1
pip install neptune-client kedro kedro-neptune
Copied!
1
conda install -c conda-forge neptune-client kedro kedro-neptune
Copied!
For more, see installing neptune-client.
This integration is tested with kedro==0.17.4, kedro-neptune==0.0.5, and neptune-client==0.10.10
Quickstart
​​
​​
​​See code examples on GitHub​
​​
​​
​​See runs logged to Neptune​
This quickstart will show you how to:
Connect Neptune to your Kedro project
Log pipeline and dataset metadata to Neptune
Add explicit metadata logging to a node in your pipeline  
Explore logged metadata in the Neptune UI.
Kedro pipeline metadata in custom dashboard in the Neptune UI
Before you start
​Have Kedro installed​
​Have neptune-client and kedro-neptune plugin installed​
Step 1: Create a Kedro project from "pandas-iris" starter
Go to your console and create a Kedro starter project "pandas-iris"​
1
kedro new --starter=pandas-iris
Copied!
Follow instructions and choose a name for your Kedro project. For example,  "Great-Kedro-Project"
Go to your new Kedro project directory
If everything was set up correctly you should see the following directory structure: 
1
Great-Kedro-Project # Parent directory of the template
2
├── conf            # Project configuration files
3
├── data            # Local project data (not committed to version control)
4
├── docs            # Project documentation
5
├── logs            # Project output logs (not committed to version control)
6
├── notebooks       # Project related Jupyter notebooks (can be used for experimental code before moving the code to src)
7
├── README.md       # Project README
8
├── setup.cfg       # Configuration options for `pytest` when doing `kedro test` and for the `isort` utility when doing `kedro lint`
9
├── src             # Project source code
10
    ├── pipelines   
11
        ├── data_science
12
            ├── nodes.py
13
            ├── pipelines.py
14
            └── ...
Copied!
You will use nodes.py and pipelines.py files in this quickstart.
Step 2: Initialize kedro-neptune plugin
Go to your Kedro project directory and run
1
kedro neptune init
Copied!
The command line will ask for your Neptune API token  
Input your Neptune API token:
Press enter if it was set to the NEPTUNE_API_TOKEN environment variable
Pass a different environment variable to which you set your Neptune API token. For example   MY_SPECIAL_NEPTUNE_TOKEN_VARIABLE 
Pass your Neptune API token as a string 
The command line will ask for your Neptune project name  
Input your Neptune project name:
Press enter if it was set to the NEPTUNE_PROJECT environment variable
Pass a different environment variable to which you set your Neptune project name. For example   MY_SPECIAL_NEPTUNE_PROJECT_VARIABLE 
Pass your project name as a string in a format WORKSPACE/PROJECT
If everything was set up correctly you should:
see the message: "kedro-neptune plugin successfully configured"
see three new files in your kedro project:
Credentials file:YOUR_KEDRO_PROJECT/conf/local/credentials_neptune.yml
Config file:YOUR_KEDRO_PROJECT/conf/base/neptune.yml
Catalog file:YOUR_KEDRO_PROJECT/conf/base/neptune_catalog.yml
You can always go to those files and change the initial configuration. 
Step 3: Add Neptune logging to a Kedro node
Go to a pipeline node src/KEDRO_PROJECT/pipelines/data_science/nodes.py
Import Neptune client toward the top of the nodes.py
snippet
full nodes.py
1
import neptune.new as neptune
Copied!
pipelines/data_science/nodes.py
1
# Copyright 2021 QuantumBlack Visual Analytics Limited
2
#
3
# Licensed under the Apache License, Version 2.0 (the "License");
4
# you may not use this file except in compliance with the License.
5
# You may obtain a copy of the License at
6
#
7
#     http://www.apache.org/licenses/LICENSE-2.0
8
#
9
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
10
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
11
# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND
12
# NONINFRINGEMENT. IN NO EVENT WILL THE LICENSOR OR OTHER CONTRIBUTORS
13
# BE LIABLE FOR ANY CLAIM, DAMAGES, OR OTHER LIABILITY, WHETHER IN AN
14
# ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF, OR IN
15
# CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
16
#
17
# The QuantumBlack Visual Analytics Limited ("QuantumBlack") name and logo
18
# (either separately or in combination, "QuantumBlack Trademarks") are
19
# trademarks of QuantumBlack. The License does not grant you any right or
20
# license to the QuantumBlack Trademarks. You may not use the QuantumBlack
21
# Trademarks or any confusingly similar mark as a trademark for your product,
22
# or use the QuantumBlack Trademarks in any other manner that might cause
23
# confusion in the marketplace, including but not limited to in advertising,
24
# on websites, or on software.
25
#
26
# See the License for the specific language governing permissions and
27
# limitations under the License.
28
​
29
"""Example code for the nodes in the example pipeline. This code is meant
30
just for illustrating basic Kedro features.
31
​
32
Delete this when you start working on your own Kedro project.
33
"""
34
# pylint: disable=invalid-name
35
​
36
import logging
37
import matplotlib.pyplot as plt
38
import neptune.new as neptune
39
import numpy as np
40
import pandas as pd
41
from scikitplot.metrics import plot_confusion_matrix
42
from typing import Any, Dict
43
​
44
​
45
def train_model(
46
        train_x: pd.DataFrame, train_y: pd.DataFrame, parameters: Dict[str, Any]
47
) -> np.ndarray:
48
    """Node for training a simple multi-class logistic regression model. The
49
    number of training iterations as well as the learning rate are taken from
50
    conf/project/parameters.yml. All of the data as well as the parameters
51
    will be provided to this function at the time of execution.
52
    """
53
    num_iter = parameters["example_num_train_iter"]
54
    lr = parameters["example_learning_rate"]
55
    X = train_x.to_numpy()
56
    Y = train_y.to_numpy()
57
​
58
    # Add bias to the features
59
    bias = np.ones((X.shape[0], 1))
60
    X = np.concatenate((bias, X), axis=1)
61
​
62
    weights = []
63
    # Train one model for each class in Y
64
    for k in range(Y.shape[1]):
65
        # Initialise weights
66
        theta = np.zeros(X.shape[1])
67
        y = Y[:, k]
68
        for _ in range(num_iter):
69
            z = np.dot(X, theta)
70
            h = _sigmoid(z)
71
            gradient = np.dot(X.T, (h - y)) / y.size
72
            theta -= lr * gradient
73
        # Save the weights for each model
74
        weights.append(theta)
75
​
76
    # Return a joint multi-class model with weights for all classes
77
    return np.vstack(weights).transpose()
78
​
79
​
80
def predict(model: np.ndarray, test_x: pd.DataFrame) -> np.ndarray:
81
    """Node for making predictions given a pre-trained model and a test set."""
82
    X = test_x.to_numpy()
83
​
84
    # Add bias to the features
85
    bias = np.ones((X.shape[0], 1))
86
    X = np.concatenate((bias, X), axis=1)
87
​
88
    # Predict "probabilities" for each class
89
    result = _sigmoid(np.dot(X, model))
90
​
91
    # Return the index of the class with max probability for all samples
92
    return np.argmax(result, axis=1)
93
​
94
​
95
def report_accuracy(predictions: np.ndarray, test_y: pd.DataFrame,
96
                    neptune_run: neptune.run.Handler) -> None:
97
    """Node for reporting the accuracy of the predictions performed by the
98
    previous node. Notice that this function has no outputs, except logging.
99
    """
100
    # Get true class index
101
    target = np.argmax(test_y.to_numpy(), axis=1)
102
    # Calculate accuracy of predictions
103
    accuracy = np.sum(predictions == target) / target.shape[0]
104
    # Log the accuracy of the model
105
    log = logging.getLogger(__name__)
106
    log.info("Model accuracy on test set: %0.2f%%", accuracy * 100)
107
​
108
    # Log accuracy to Neptune
109
    neptune_run['nodes/report/accuracy'] = accuracy * 100
110
​
111
    # Log confusion matrix to Neptune
112
    fig, ax = plt.subplots()
113
    plot_confusion_matrix(target, predictions, ax=ax)
114
    neptune_run['nodes/report/confusion_matrix'].upload(fig)
115
​
116
​
117
def _sigmoid(z):
118
    """A helper sigmoid function used by the training and the scoring nodes."""
119
    return 1 / (1 + np.exp(-z))
120
​
Copied!
Add neptune_run argument of type neptune.run.Handler to the report_accuracy function 
snippet
full nodes.py
1
def report_accuracy(predictions: np.ndarray, test_y: pd.DataFrame, 
2
                    neptune_run: neptune.run.Handler) -> None:
3
...
Copied!
pipelines/data_science/nodes.py
1
# Copyright 2021 QuantumBlack Visual Analytics Limited
2
#
3
# Licensed under the Apache License, Version 2.0 (the "License");
4
# you may not use this file except in compliance with the License.
5
# You may obtain a copy of the License at
6
#
7
#     http://www.apache.org/licenses/LICENSE-2.0
8
#
9
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
10
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
11
# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND
12
# NONINFRINGEMENT. IN NO EVENT WILL THE LICENSOR OR OTHER CONTRIBUTORS
13
# BE LIABLE FOR ANY CLAIM, DAMAGES, OR OTHER LIABILITY, WHETHER IN AN
14
# ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF, OR IN
15
# CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
16
#
17
# The QuantumBlack Visual Analytics Limited ("QuantumBlack") name and logo
18
# (either separately or in combination, "QuantumBlack Trademarks") are
19
# trademarks of QuantumBlack. The License does not grant you any right or
20
# license to the QuantumBlack Trademarks. You may not use the QuantumBlack
21
# Trademarks or any confusingly similar mark as a trademark for your product,
22
# or use the QuantumBlack Trademarks in any other manner that might cause
23
# confusion in the marketplace, including but not limited to in advertising,
24
# on websites, or on software.
25
#
26
# See the License for the specific language governing permissions and
27
# limitations under the License.
28
​
29
"""Example code for the nodes in the example pipeline. This code is meant
30
just for illustrating basic Kedro features.
31
​
32
Delete this when you start working on your own Kedro project.
33
"""
34
# pylint: disable=invalid-name
35
​
36
import logging
37
import matplotlib.pyplot as plt
38
import neptune.new as neptune
39
import numpy as np
40
import pandas as pd
41
from scikitplot.metrics import plot_confusion_matrix
42
from typing import Any, Dict
43
​
44
​
45
def train_model(
46
        train_x: pd.DataFrame, train_y: pd.DataFrame, parameters: Dict[str, Any]
47
) -> np.ndarray:
48
    """Node for training a simple multi-class logistic regression model. The
49
    number of training iterations as well as the learning rate are taken from
50
    conf/project/parameters.yml. All of the data as well as the parameters
51
    will be provided to this function at the time of execution.
52
    """
53
    num_iter = parameters["example_num_train_iter"]
54
    lr = parameters["example_learning_rate"]
55
    X = train_x.to_numpy()
56
    Y = train_y.to_numpy()
57
​
58
    # Add bias to the features
59
    bias = np.ones((X.shape[0], 1))
60
    X = np.concatenate((bias, X), axis=1)
61
​
62
    weights = []
63
    # Train one model for each class in Y
64
    for k in range(Y.shape[1]):
65
        # Initialise weights
66
        theta = np.zeros(X.shape[1])
67
        y = Y[:, k]
68
        for _ in range(num_iter):
69
            z = np.dot(X, theta)
70
            h = _sigmoid(z)
71
            gradient = np.dot(X.T, (h - y)) / y.size
72
            theta -= lr * gradient
73
        # Save the weights for each model
74
        weights.append(theta)
75
​
76
    # Return a joint multi-class model with weights for all classes
77
    return np.vstack(weights).transpose()
78
​
79
​
80
def predict(model: np.ndarray, test_x: pd.DataFrame) -> np.ndarray:
81
    """Node for making predictions given a pre-trained model and a test set."""
82
    X = test_x.to_numpy()
83
​
84
    # Add bias to the features
85
    bias = np.ones((X.shape[0], 1))
86
    X = np.concatenate((bias, X), axis=1)
87
​
88
    # Predict "probabilities" for each class
89
    result = _sigmoid(np.dot(X, model))
90
​
91
    # Return the index of the class with max probability for all samples
92
    return np.argmax(result, axis=1)
93
​
94
​
95
def report_accuracy(predictions: np.ndarray, test_y: pd.DataFrame,
96
                    neptune_run: neptune.run.Handler) -> None:
97
    """Node for reporting the accuracy of the predictions performed by the
98
    previous node. Notice that this function has no outputs, except logging.
99
    """
100
    # Get true class index
101
    target = np.argmax(test_y.to_numpy(), axis=1)
102
    # Calculate accuracy of predictions
103
    accuracy = np.sum(predictions == target) / target.shape[0]
104
    # Log the accuracy of the model
105
    log = logging.getLogger(__name__)
106
    log.info("Model accuracy on test set: %0.2f%%", accuracy * 100)
107
​
108
    # Log accuracy to Neptune
109
    neptune_run['nodes/report/accuracy'] = accuracy * 100
110
​
111
    # Log confusion matrix to Neptune
112
    fig, ax = plt.subplots()
113
    plot_confusion_matrix(target, predictions, ax=ax)
114
    neptune_run['nodes/report/confusion_matrix'].upload(fig)
115
​
116
​
117
def _sigmoid(z):
118
    """A helper sigmoid function used by the training and the scoring nodes."""
119
    return 1 / (1 + np.exp(-z))
120
​
Copied!
You can treat neptune_run like a normal Neptune Run and log any ML metadata to it. 
Important
You have to use a special string "neptune_run" to use the Neptune Run handler in Kedro pipelines.
Log metrics like accuracy to neptune_run  
snippet
full nodes.py
1
def report_accuracy(predictions: np.ndarray, test_y: pd.DataFrame, 
2
                    neptune_run: neptune.run.Handler) -> None:
3
    target = np.argmax(test_y.to_numpy(), axis=1)
4
    accuracy = np.sum(predictions == target) / target.shape[0]
5
    
6
    neptune_run['nodes/report/accuracy'] = accuracy * 100
Copied!
pipelines/data_science/nodes.py
1
# Copyright 2021 QuantumBlack Visual Analytics Limited
2
#
3
# Licensed under the Apache License, Version 2.0 (the "License");
4
# you may not use this file except in compliance with the License.
5
# You may obtain a copy of the License at
6
#
7
#     http://www.apache.org/licenses/LICENSE-2.0
8
#
9
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
10
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
11
# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND
12
# NONINFRINGEMENT. IN NO EVENT WILL THE LICENSOR OR OTHER CONTRIBUTORS
13
# BE LIABLE FOR ANY CLAIM, DAMAGES, OR OTHER LIABILITY, WHETHER IN AN
14
# ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF, OR IN
15
# CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
16
#
17
# The QuantumBlack Visual Analytics Limited ("QuantumBlack") name and logo
18
# (either separately or in combination, "QuantumBlack Trademarks") are
19
# trademarks of QuantumBlack. The License does not grant you any right or
20
# license to the QuantumBlack Trademarks. You may not use the QuantumBlack
21
# Trademarks or any confusingly similar mark as a trademark for your product,
22
# or use the QuantumBlack Trademarks in any other manner that might cause
23
# confusion in the marketplace, including but not limited to in advertising,
24
# on websites, or on software.
25
#
26
# See the License for the specific language governing permissions and
27
# limitations under the License.
28
​
29
"""Example code for the nodes in the example pipeline. This code is meant
30
just for illustrating basic Kedro features.
31
​
32
Delete this when you start working on your own Kedro project.
33
"""
34
# pylint: disable=invalid-name
35
​
36
import logging
37
import matplotlib.pyplot as plt
38
import neptune.new as neptune
39
import numpy as np
40
import pandas as pd
41
from scikitplot.metrics import plot_confusion_matrix
42
from typing import Any, Dict
43
​
44
​
45
def train_model(
46
        train_x: pd.DataFrame, train_y: pd.DataFrame, parameters: Dict[str, Any]
47
) -> np.ndarray:
48
    """Node for training a simple multi-class logistic regression model. The
49
    number of training iterations as well as the learning rate are taken from
50
    conf/project/parameters.yml. All of the data as well as the parameters
51
    will be provided to this function at the time of execution.
52
    """
53
    num_iter = parameters["example_num_train_iter"]
54
    lr = parameters["example_learning_rate"]
55
    X = train_x.to_numpy()
56
    Y = train_y.to_numpy()
57
​
58
    # Add bias to the features
59
    bias = np.ones((X.shape[0], 1))
60
    X = np.concatenate((bias, X), axis=1)
61
​
62
    weights = []
63
    # Train one model for each class in Y
64
    for k in range(Y.shape[1]):
65
        # Initialise weights
66
        theta = np.zeros(X.shape[1])
67
        y = Y[:, k]
68
        for _ in range(num_iter):
69
            z = np.dot(X, theta)
70
            h = _sigmoid(z)
71
            gradient = np.dot(X.T, (h - y)) / y.size
72
            theta -= lr * gradient
73
        # Save the weights for each model
74
        weights.append(theta)
75
​
76
    # Return a joint multi-class model with weights for all classes
77
    return np.vstack(weights).transpose()
78
​
79
​
80
def predict(model: np.ndarray, test_x: pd.DataFrame) -> np.ndarray:
81
    """Node for making predictions given a pre-trained model and a test set."""
82
    X = test_x.to_numpy()
83
​
84
    # Add bias to the features
85
    bias = np.ones((X.shape[0], 1))
86
    X = np.concatenate((bias, X), axis=1)
87
​
88
    # Predict "probabilities" for each class
89
    result = _sigmoid(np.dot(X, model))
90
​
91
    # Return the index of the class with max probability for all samples
92
    return np.argmax(result, axis=1)
93
​
94
​
95
def report_accuracy(predictions: np.ndarray, test_y: pd.DataFrame,
96
                    neptune_run: neptune.run.Handler) -> None:
97
    """Node for reporting the accuracy of the predictions performed by the
98
    previous node. Notice that this function has no outputs, except logging.
99
    """
100
    # Get true class index
101
    target = np.argmax(test_y.to_numpy(), axis=1)
102
    # Calculate accuracy of predictions
103
    accuracy = np.sum(predictions == target) / target.shape[0]
104
    # Log the accuracy of the model
105
    log = logging.getLogger(__name__)
106
    log.info("Model accuracy on test set: %0.2f%%", accuracy * 100)
107
​
108
    # Log accuracy to Neptune
109
    neptune_run['nodes/report/accuracy'] = accuracy * 100
110
​
111
    # Log confusion matrix to Neptune
112
    fig, ax = plt.subplots()
113
    plot_confusion_matrix(target, predictions, ax=ax)
114
    neptune_run['nodes/report/confusion_matrix'].upload(fig)
115
​
116
​
117
def _sigmoid(z):
118
    """A helper sigmoid function used by the training and the scoring nodes."""
119
    return 1 / (1 + np.exp(-z))
120
​
Copied!
You can log metadata from any node to any Neptune namespace you want. 
Log images like a confusion matrix to neptune_run
snippet
full nodes.py
1
def report_accuracy(predictions: np.ndarray, test_y: pd.DataFrame, 
2
                    neptune_run: neptune.run.Handler) -> None:
3
    target = np.argmax(test_y.to_numpy(), axis=1)
4
    accuracy = np.sum(predictions == target) / target.shape[0]
5
    
6
    fig, ax = plt.subplots()
7
    plot_confusion_matrix(target, predictions, ax=ax)
8
    neptune_run['nodes/report/confusion_matrix'].upload(fig)
Copied!
pipelines/data_science/nodes.py
1
# Copyright 2021 QuantumBlack Visual Analytics Limited
2
#
3
# Licensed under the Apache License, Version 2.0 (the "License");
4
# you may not use this file except in compliance with the License.
5
# You may obtain a copy of the License at
6
#
7
#     http://www.apache.org/licenses/LICENSE-2.0
8
#
9
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
10
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
11
# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND
12
# NONINFRINGEMENT. IN NO EVENT WILL THE LICENSOR OR OTHER CONTRIBUTORS
13
# BE LIABLE FOR ANY CLAIM, DAMAGES, OR OTHER LIABILITY, WHETHER IN AN
14
# ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF, OR IN
15
# CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
16
#
17
# The QuantumBlack Visual Analytics Limited ("QuantumBlack") name and logo
18
# (either separately or in combination, "QuantumBlack Trademarks") are
19
# trademarks of QuantumBlack. The License does not grant you any right or
20
# license to the QuantumBlack Trademarks. You may not use the QuantumBlack
21
# Trademarks or any confusingly similar mark as a trademark for your product,
22
# or use the QuantumBlack Trademarks in any other manner that might cause
23
# confusion in the marketplace, including but not limited to in advertising,
24
# on websites, or on software.
25
#
26
# See the License for the specific language governing permissions and
27
# limitations under the License.
28
​
29
"""Example code for the nodes in the example pipeline. This code is meant
30
just for illustrating basic Kedro features.
31
​
32
Delete this when you start working on your own Kedro project.
33
"""
34
# pylint: disable=invalid-name
35
​
36
import logging
37
import matplotlib.pyplot as plt
38
import neptune.new as neptune
39
import numpy as np
40
import pandas as pd
41
from scikitplot.metrics import plot_confusion_matrix
42
from typing import Any, Dict
43
​
44
​
45
def train_model(
46
        train_x: pd.DataFrame, train_y: pd.DataFrame, parameters: Dict[str, Any]
47
) -> np.ndarray:
48
    """Node for training a simple multi-class logistic regression model. The
49
    number of training iterations as well as the learning rate are taken from
50
    conf/project/parameters.yml. All of the data as well as the parameters
51
    will be provided to this function at the time of execution.
52
    """
53
    num_iter = parameters["example_num_train_iter"]
54
    lr = parameters["example_learning_rate"]
55
    X = train_x.to_numpy()
56
    Y = train_y.to_numpy()
57
​
58
    # Add bias to the features
59
    bias = np.ones((X.shape[0], 1))
60
    X = np.concatenate((bias, X), axis=1)
61
​
62
    weights = []
63
    # Train one model for each class in Y
64
    for k in range(Y.shape[1]):
65
        # Initialise weights
66
        theta = np.zeros(X.shape[1])
67
        y = Y[:, k]
68
        for _ in range(num_iter):
69
            z = np.dot(X, theta)
70
            h = _sigmoid(z)
71
            gradient = np.dot(X.T, (h - y)) / y.size
72
            theta -= lr * gradient
73
        # Save the weights for each model
74
        weights.append(theta)
75
​
76
    # Return a joint multi-class model with weights for all classes
77
    return np.vstack(weights).transpose()
78
​
79
​
80
def predict(model: np.ndarray, test_x: pd.DataFrame) -> np.ndarray:
81
    """Node for making predictions given a pre-trained model and a test set."""
82
    X = test_x.to_numpy()
83
​
84
    # Add bias to the features
85
    bias = np.ones((X.shape[0], 1))
86
    X = np.concatenate((bias, X), axis=1)
87
​
88
    # Predict "probabilities" for each class
89
    result = _sigmoid(np.dot(X, model))
90
​
91
    # Return the index of the class with max probability for all samples
92
    return np.argmax(result, axis=1)
93
​
94
​
95
def report_accuracy(predictions: np.ndarray, test_y: pd.DataFrame,
96
                    neptune_run: neptune.run.Handler) -> None:
97
    """Node for reporting the accuracy of the predictions performed by the
98
    previous node. Notice that this function has no outputs, except logging.
99
    """
100
    # Get true class index
101
    target = np.argmax(test_y.to_numpy(), axis=1)
102
    # Calculate accuracy of predictions
103
    accuracy = np.sum(predictions == target) / target.shape[0]
104
    # Log the accuracy of the model
105
    log = logging.getLogger(__name__)
106
    log.info("Model accuracy on test set: %0.2f%%", accuracy * 100)
107
​
108
    # Log accuracy to Neptune
109
    neptune_run['nodes/report/accuracy'] = accuracy * 100
110
​
111
    # Log confusion matrix to Neptune
112
    fig, ax = plt.subplots()
113
    plot_confusion_matrix(target, predictions, ax=ax)
114
    neptune_run['nodes/report/confusion_matrix'].upload(fig)
115
​
116
​
117
def _sigmoid(z):
118
    """A helper sigmoid function used by the training and the scoring nodes."""
119
    return 1 / (1 + np.exp(-z))
120
​
Copied!
Note
You can log metrics, text, images, video, interactive visualizations, and more. 
See a full list of What you can log and display in Neptune.
Step 4: Add Neptune Run handler to the Kedro pipeline
Go to a pipeline definition, src/KEDRO_PROJECT/pipelines/data_science/pipelines.py
Add neptune_run Run handler as an input to the report node
snippet
full pipelines.py
1
node(
2
    report_accuracy,
3
    ["example_predictions", "example_test_y", "neptune_run"],
4
    None,
5
    name="report"),
Copied!
pipelines/data_science/pipelines.py
1
# Copyright 2021 QuantumBlack Visual Analytics Limited
2
#
3
# Licensed under the Apache License, Version 2.0 (the "License");
4
# you may not use this file except in compliance with the License.
5
# You may obtain a copy of the License at
6
#
7
#     http://www.apache.org/licenses/LICENSE-2.0
8
#
9
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
10
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
11
# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND
12
# NONINFRINGEMENT. IN NO EVENT WILL THE LICENSOR OR OTHER CONTRIBUTORS
13
# BE LIABLE FOR ANY CLAIM, DAMAGES, OR OTHER LIABILITY, WHETHER IN AN
14
# ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF, OR IN
15
# CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
16
#
17
# The QuantumBlack Visual Analytics Limited ("QuantumBlack") name and logo
18
# (either separately or in combination, "QuantumBlack Trademarks") are
19
# trademarks of QuantumBlack. The License does not grant you any right or
20
# license to the QuantumBlack Trademarks. You may not use the QuantumBlack
21
# Trademarks or any confusingly similar mark as a trademark for your product,
22
# or use the QuantumBlack Trademarks in any other manner that might cause
23
# confusion in the marketplace, including but not limited to in advertising,
24
# on websites, or on software.
25
#
26
# See the License for the specific language governing permissions and
27
# limitations under the License.
28
​
29
"""Example code for the nodes in the example pipeline. This code is meant
30
just for illustrating basic Kedro features.
31
​
32
Delete this when you start working on your own Kedro project.
33
"""
34
​
35
from kedro.pipeline import Pipeline, node
36
​
37
from .nodes import predict, report_accuracy, train_model
38
​
39
​
40
def create_pipeline(**kwargs):
41
    return Pipeline(
42
        [
43
            node(
44
                train_model,
45
                ["example_train_x", "example_train_y", "parameters"],
46
                "example_model",
47
                name="train",
48
            ),
49
            node(
50
                predict,
51
                dict(model="example_model", test_x="example_test_x"),
52
                "example_predictions",
53
                name="predict",
54
            ),
55
            node(
56
                report_accuracy,
57
                ["example_predictions", "example_test_y","neptune_run"],
58
                None,
59
                name="report",
60
            ),
61
        ]
62
    )
63
​
Copied!
Step 5: Run Kedro pipeline
Go to your console and execute your Kedro pipeline
1
kedro run
Copied!
A link to the Neptune Run associated with the Kedro pipeline execution will be printed to the console.
Step 6: Explore results in the Neptune UI 
Click on the Neptune Run link in your console or use an example link
​https://app.neptune.ai/common/kedro-integration/e/KED-632​
Go to the kedro namespace where metadata about Kedro pipelines are logged (see how to change the default logging location)
Default Kedro namespace in Neptune UI
See pipeline and node parameters in kedro/catalog/parameters 
Pipeline parameters logged from Kedro to Neptune UI
See execution parameters in kedro/run_params
Execution parameters logged from Kedro to Neptune UI
See metadata about the datasets in kedro/catalog/datasets/example_iris_data
Dataset metadata logged from Kedro to Neptune UI
See the metrics (accuracy) you logged explicitly in the kedro/nodes/report/accuracy
Metrics logged from Kedro to Neptune UI
See charts (confusion matrix) you logged explicitly in the kedro/nodes/report/confusion_matrix
Confusion matrix logged from Kedro to Neptune UI
More options
Creating a dashboard with Kedro pipeline metadata
You can combine all the metadata in a single dashboard like this. 
​See an example dashboard in Neptune​
Kedro pipeline metadata in custom dashboard in the Neptune UI
Comparing Kedro pipeline runs 
You can compare run metadata from your Kedro pipelines in the Neptune UI. 
​See an example dashboard in Neptune​
Comparing run metadata for Kedro pipelines in the Neptune UI 
Filtering and organizing Kedro pipeline runs 
You can filter and organize run metadata from your Kedro pipelines in the Neptune UI. 
​See an example dashboard in Neptune​
Filtering and organizing run metadata for Kedro pipelines in the Neptune UI
Basic logging configuration
You can configure where and how kedro pipeline metadata is logged in the conf/base/neptune.yml file. 
conf/base/neptune.yml
1
neptune:
2
#GLOBAL CONFIG
3
  project: common/kedro-integration
4
  base_namespace: kedro
5
  enabled: true
6
​
7
#LOGGING
8
  upload_source_files:
9
  - '**/*.py'
10
  - 'conf/base/*.yml'
Copied!
Specifically:
project: choose a Neptune project to which you want to log. You can use project: $NEPTUNE_PROJECTif you want to get it from the environment variable.
base_namespace: choose a base namespace (folder) where your metadata will be logged
enabled: whether Neptune should be enabled. If false Neptune will switch to Debug mode.
upload_source_files: choose which files you want to log to Neptune
Neptune API token configuration
You can configure how kedro-neptune will look for your Neptune API token in the conf/local/credentials_neptune.yml
conf/local/credentials_neptune.yml
1
neptune:
2
  api_token: eyJhcGlfYWRk123cmVqgpije5cyI6Imh0dHBzOi8v
Copied!
You can:
pass it as a string, for example eyJhcGlfYWRk123cmVqgpije5cyI6Imh0dHBzOi8v
pass an environment variable, for example $MY_NEPTUNE_API_TOKEN_VARIABLE
leave it empty in which case Neptune will look for your API token in the $NEPTUNE_API_TOKEN environment variable
Important
If you are setting api_token to an environment variable add $ before the variable name. For example $NEPTUNE_API_TOKEN
Logging files and datasets to Neptune 
You can log files to Neptune with a special Kedro DataSet called kedro_neptune.NeptuneFileDataSet.
conf/base/catalog.yml
1
example_csv_file:
2
  type: kedro_neptune.NeptuneFileDataSet
3
  filepath: data/01_raw/iris.csv
Copied!
To do that, add a Kedro Data Set to your catalog:
type: this is always kedro_neptune.NeptuneFileDataSet
filepath: path to a file you would like to log
You can find all the logged NeptuneFileDatasets in the kedro/catalog/files namespace in the Neptune UI. Many file types, like PNG, JSON, YML, CSV, HTML will be displayed in Neptune. 
Logged Kedro NeptuneFileDataSet displayed in the Neptune UI. 
If you already have a Kedro DataSet that you would log under the same name to Neptune add @neptune to the DataSet name: 
conf/base/catalog.yml
1
example_iris_data:
2
  type: pandas.CSVDataSet
3
  filepath: data/01_raw/iris.csv
4
​
5
[email protected]:
6
  type: kedro_neptune.NeptuneFileDataSet
7
  filepath: data/01_raw/iris.csv
Copied!
Important
You should not log the whole training dataset to Neptune but rather a small informative part of it that you would like to display later.  
To log files to Neptune you can do it directly through Neptune API by using the .upload() and .upload_files() methods:
1
neptune_run['dataset/example_iris_data'].upload('data/01_raw/iris.csv')
Copied!
See also
​How to Compare Kedro pipelines​
​How to Compare results between Kedro nodes​
​How to Display Kedro node metadata and outputs​
How to Compare Runs​
​What metadata you can log and display​
How to Organize and Filer Runs ​