Arize

Let's get started on using Arize with Neptune!
​Arize and Neptune are MLOps tools that aim to improve connected, but different parts of your ML pipeline and ML workflow. Arize helps you visualize your production model performance, understand drift & data quality issues. Neptune logs, stores, displays, and compares all your MLOps metadata for better experiment tracking and model registry.
With Arize and Neptune, you will be able to train the best model, and pre-launch validate your model, and compare production performances of those models.
Steps for this Walkthrough:
1.Initialize Neptune and set-up Arize client
2.Logging training callbacks to Neptune
3.Logging training and validation records to Arize
4.Storing and versioning model weights with Neptune
5.Logging and versioning model in production with Arize
Step 1: Initialize Neptune and set-up Arize client
Set-up Neptune Project
First you will need to create a Neptune account and follow these steps
1.Sign up for an account and replace YOUR_USER_NAME with your client name
2.Copy your API_TOKEN from top right of the neptune nav bar
3.Create a new Project and name it ArizeIntegration. Here is how to create project.
1
!pip install neptune-client -q
2
!pip install neptune-tensorflow-keras -q
3
​
4
import neptune.new as neptune
5
from neptune.new.integrations.tensorflow_keras import NeptuneCallback
6
​
7
NEPTUNE_USER_NAME = 'NEPTUNE_USER_NAME'
8
NEPTUNE_API_TOKEN = 'NEPTUNE_API_TOKEN'
9
​
10
if NEPTUNE_USER_NAME == 'NEPTUNE_USER_NAME' or NEPTUNE_API_TOKEN == 'NEPTUNE_API_TOKEN': 
11
    raise ValueError("NEED TO CHANGE USERNAME AND/OR API TOKEN")
12
​
13
# set parameters for initializing Neptune
14
PROJECT_NAME = f"{NEPTUNE_USER_NAME}/ArizeIntegration"
15
run = neptune.init(api_token=NEPTUNE_API_TOKEN, project=PROJECT_NAME)
16
​
17
print('Step 1.1: Initialize Neptune run and project complete!')
Copied!
You can find more info about the NeptuneCallback in the TensorFlow / Keras integration docs.
Set-up Arize Client
To set up Arize, copy the Arize API_KEY and ORG_KEY from your admin page.
1
!pip install arize -q
2
from arize.api import Client
3
from arize.types import ModelTypes
4
​
5
ORGANIZATION_KEY = 'ORGANIZATION_KEY'
6
API_KEY = 'API_KEY'
7
arize = Client(organization_key=ORGANIZATION_KEY, api_key=API_KEY)
8
​
9
model_id = 'neptune_cancer_prediction_model'
10
model_version = 'v1'
11
model_type = ModelTypes.BINARY
12
​
13
if ORGANIZATION_KEY == 'ORGANIZATION_KEY' or API_KEY == 'API_KEY': 
14
    raise ValueError("NEED TO CHANGE ORGANIZATION AND/OR API_KEY")
15
else: 
16
    print("Step 1.2: Initialize Arize client complete!")
Copied!
Step 2: Logging training callbacks to Neptune
Neptune tracks your model training callbacks, allowing training loss curves to be logged and visualized for each different training iterations. In this example, we will be working with a tensorflow.keras model to build a model for classifying whether an individual has breast cancer or not.
Import Dataset
1
import numpy as np
2
import pandas as pd
3
import uuid
4
import os
5
import concurrent.futures as cf
6
from sklearn import datasets, preprocessing
7
from sklearn.model_selection import train_test_split
8
import datetime
9
​
10
def process_data(X, y):
11
    scaler = preprocessing.MinMaxScaler()
12
    X = np.array(X).reshape((len(X), 30))
13
    y = np.array(y)
14
    return X, y
15
​
16
# 1 Load data and split data
17
data = datasets.load_breast_cancer()
18
​
19
X, y = datasets.load_breast_cancer(return_X_y=True)
20
X, y = X.astype(np.float32), y
21
​
22
X, y = pd.DataFrame(X, columns=data['feature_names']), pd.Series(y)
23
​
24
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
25
X_train, X_val, y_train, y_val = train_test_split(X_train,
26
                                                  y_train,
27
                                                  random_state=42)
28
​
29
print('Step 2.1: Load Data Done!')
Copied!
Logging Training Callbacks
By passing run instance, a live training curve should show up on Neptune under the Charts tab.
1
import tensorflow.keras as keras
2
from keras.models import Sequential
3
from keras.layers import Dense, Dropout, Flatten, Activation
4
import tensorflow as tf
5
​
6
# Step 1: Define and compile model
7
model = Sequential()
8
model.add(Dense(10, activation='sigmoid', input_shape=((30,))))
9
model.add(Dropout(0.25))
10
model.add(Dense(20, activation='sigmoid'))
11
model.add(Dropout(0.25))
12
model.add(Dense(10, activation='sigmoid'))
13
model.add(Dropout(0.25))
14
model.add(Dense(1, activation='sigmoid'))
15
model.compile(optimizer=keras.optimizers.Adam(), 
16
              loss=keras.losses.mean_squared_logarithmic_error)
17
​
18
# Step 2: Fit model and log callbacks
19
​
20
params = {'batch_size': 30,
21
          'epochs': 50,
22
          'verbose': 0,
23
         }
24
​
25
callbacked = model.fit(X_train, y_train, 
26
                batch_size=params['batch_size'], 
27
                epochs=params['epochs'], 
28
                verbose=params['verbose'], 
29
                validation_data=(X_test, y_test),
30
                # log to Neptune using NeptuneCallback
31
                callbacks=[NeptuneCallback(run=run)]
32
                )
33
​
34
print('Step 2.2: Training callbacks successfully logged!')
Copied!
Step 3: Logging training and validation records to Arize
Arize allows you to log training and validation records to an Evaluation Store for model pre-launch validation, such as visualizing performance across different feature slices (i.e, model accuracy for lower income individuals v.s higher).
The records you send in can also serve as your model baseline, which can be compared against the features your models predict on in production to inform you when the distributions of the features have shifted. You can click here to access the documentation for Arize's Python SDK and arize.log_training_records.
Optional helper
1
# OPTIONAL: A quick helper function to validate Arize responses
2
def arize_responses_helper(responses):
3
    for response in cf.as_completed(responses):
4
        res = response.result()
5
        if res.status_code != 200:
6
            raise ValueError(f'future failed with response code {res.status_code}, {res.text}')
7
​
Copied!
Logging Training Records to Arize
1
# Use the model to generate predictions
2
y_train_pred = model.predict(X_train).T[0]
3
y_val_pred = model.predict(X_val).T[0]
4
y_test_pred = model.predict(X_test).T[0]
5
​
6
# Logging training
7
train_prediction_labels = pd.Series(y_train_pred)
8
train_actual_labels = pd.Series(y_train)
9
train_feature_df = pd.DataFrame(X_train, columns=data['feature_names'])
10
​
11
train_responses = arize.log_training_records(
12
    model_id=model_id,
13
    model_version=model_version,
14
    model_type=model_type, # this will change depending on your model type
15
    prediction_labels=train_prediction_labels,
16
    actual_labels=train_actual_labels,
17
    features=train_feature_df,
18
    )
19
​
20
arize_responses_helper(train_responses)
21
​
22
print('Step 3.1: If no errors showed up, you have sent Training Inferences!')
Copied!
Logging Validation to Arize
1
val_prediction_labels = pd.Series(y_val_pred)
2
val_actual_labels = pd.Series(y_val)
3
val_features_df = pd.DataFrame(X_val, columns=data['feature_names'])
4
​
5
val_responses = arize.log_validation_records(
6
    model_id=model_id,
7
    model_version=model_version,
8
    model_type=model_type,
9
    batch_id='batch0',
10
    prediction_labels=val_prediction_labels,
11
    actual_labels=val_actual_labels,
12
    features=val_features_df,
13
    )
14
​
15
arize_responses_helper(val_responses)
16
print('Step 3.2: If no errors showed up, you have sent Validation Inferences!')
Copied!
Step 4: Storing and Versioning Model Weights with Neptune
Neptune allows you to organize your models in a folder like structure through the run instance of each project. For each run, you can log model weights or checkpoints. You can organize different trained iterations using tag model_version you used to log training records to Arize for better integration.
You can also easily log model_id for better reference information.
Code for model storing is different for different frameworks. The following is only applicable for tf.keras.
1
import glob
2
​
3
# Storing model version 1
4
directory_name = f'keras_model_{model_version}'
5
model.save(directory_name)
6
​
7
run[f'{directory_name}/saved_model.pb'].upload(f'{directory_name}/saved_model.pb')
8
for name in glob.glob(f'{directory_name}/variables/*'):
9
    run[name].upload(name)
10
​
11
# Log 'model_id', for better reference
12
run['model_id'] = model_id
13
​
14
print('Step 4: If no errors showed up, can should now see the folders in your Neptune Project')
Copied!
Step 5: Logging and versioning model in production with Arize
During production, you can use arize.bulk_log or arize.log in the Python SDK to log any data in your model serving endpoint. In this example, we send in our test data simulating production setting. But in production, you would deploy the models saved by Neptune prior to logging to Arize!
You can find more about arize.bulk_log here.
1
import datetime
2
# Generating Predictions
3
y_test_pred = pd.Series(y_test_pred)
4
num_preds = len(y_test_pred) # num_preds == 143
5
​
6
# Generating Prediction IDs
7
ids_df = pd.DataFrame([str(uuid.uuid4()) for _ in range(num_preds)])
8
​
9
# Logging the Predictions, Features, and Actuals
10
log_predictions_responses = arize.bulk_log(
11
    # Required arguments
12
    model_id=model_id,
13
    prediction_ids=ids_df,
14
    # Optional arguments
15
    model_version=model_version,
16
    prediction_labels=y_test_pred,
17
    actual_labels=y_test,
18
    features=X_test, # we recommend logging features with predictions
19
    model_type=model_type, # we recommend using model_type on first time logging to Arize
20
    feature_names_overwrite=None,
21
    )
22
​
23
arize_responses_helper(log_predictions_responses)
24
print('Step 5: If no errors appear, you just logged {} total predictions, features, and actuals to Arize!'.format(num_preds))
Copied!
​

Integrations and Supported Tools - Previous

Model monitoring

Next - API reference

Neptune

Last modified 21d ago

Export as PDF

Copy link

Contents

Step 1: Initialize Neptune and set-up Arize client

Set-up Neptune Project

Set-up Arize Client

Step 2: Logging training callbacks to Neptune

Import Dataset

Logging Training Callbacks

Step 3: Logging training and validation records to Arize

Logging Training Records to Arize

Logging Validation to Arize

Step 4: Storing and Versioning Model Weights with Neptune

Step 5: Logging and versioning model in production with Arize