You should know
How-to guides
Integrations and Supported Tools
API reference
Administration
Continuous Integration and Delivery(CI/CD)
You can use Neptune as a model registry that lets you log, query and download all model-building metadata.
This guide shows how to query and retrieve a Neptune Run and all the model metadata to execute Continuous Integration/Continuous Delivery jobs.
These jobs can be executed:
- On scheduled time
- Every time you have new data
- Each time there is a code commit
By the end of this guide, you will get a GitHub Action set up so that you can evaluate if the staging model (challenger) is better than the production model (current best) and promote the new best model to production automatically
Keywords: CI/CD, Continuous Integration, Continuous Delivery, CI/CD machine learning
Before you start
Make sure you meet the following prerequisites before starting:
Step 1: Get the Challenger (new) and Champion (best) model metadata
In this step, you will:
- 1.Get the Neptune Project where our Runs are.
- 2.Get the Run ID’s of the champion and the challenger models
- 3.Take the Run ID’s we got and use them to resume the Run in read-only mode so the metadata is not accidentally changed.
- 4.Fetching and downloading data from Neptune
- 5.Loading model checkpoints and data
First, you have to get the Neptune project that contains the Runs that you want to evaluate. You do that by using the
.get_project() method passing the project name.1
project = neptune.get_project(api_token=os.getenv("NEPTUNE_API_TOKEN"),
2
project='<YOUR_PROJECT_NAME>')
Copied!
Note
You can use
project='common/pytorch-integration'and the api_token = 'ANONYMOUS'to explore without having to create a Neptune account.The api_token is passed as Github Action environment variable created from a Github secret. How to set it up is described in a later section.
Important
To create your own private projects, you will need a Neptune account and your personal Neptune API token.
Second, we want to filter Runs and get the champion and challenger's Run ID.
To do that:
- Extract the Run ID just like you would in a pandas dataframe.
1
# Champion
2
champion_runs_table_df = project.fetch_runs_table(tag='champion').to_pandas()
3
champion_run_id = champion_runs_table_df['sys/id'].values[0]
4
​
5
# Challenger
6
challenger_runs_table_df = project.fetch_runs_table(tag='challenger').to_pandas()
7
challenger_run_id = challenger_runs_table_df['sys/id'].values[0]
Copied!
- Take the Run ID and use it to resume the Run in
read-onlymode in order to avoid logging or changing existing metadata.
1
champion_run = neptune.init(
2
api_token=os.getenv("NEPTUNE_API_TOKEN"),
3
project='<YOUR_PROJECT_NAME>',
4
run = champion_run_id,
5
mode = 'read-only'
6
)
7
​
8
challenger_run = neptune.init(
9
api_token=os.getenv("NEPTUNE_API_TOKEN"),
10
project='<YOUR_PROJECT_NAME>',
11
run = challenger_run_id,
12
mode = 'read-only'
13
)
Copied!
Executing this snippet will give you a link like this one:
https://app.neptune.ai/common/pytorch-integration/e/PYTOR1-72
with
common/pytorch-integration replaced by your_workspace/your_project_name, and PYTOR1-72replaced by your Run ID.Query and download model metadata from Neptune
To retrieve metadata from Neptune you should:
1
# fetching non-file values
2
parameters = run['config/hyperparameters'].fetch()
3
parameters['device'] = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
4
​
5
data_dir = run['config/dataset/path'].fetch()
Copied!
- Use .download() method to retrieve files such as the model weights for both the champion and challenger.
1
# Download model weights
2
model_fname = 'model.pth'
3
run['io_files/artifacts/basemodel'].download(f'./{model_fname}')
Copied!
Load model object and data loaders from metadata
- Load the data for the evaluation
1
# loading dataset
2
data_tfms = {
3
"val": transforms.Compose(
4
[
5
transforms.ToTensor(),
6
transforms.Normalize(
7
[0.485, 0.456, 0.406],
8
[0.229, 0.224, 0.225]
9
),
10
]
11
)
12
}
13
​
14
validset = datasets.CIFAR10(
15
data_dir, train=False, transform=data_tfms["val"], download=True
16
)
17
validloader = torch.utils.data.DataLoader(
18
validset, batch_size=bs, num_workers=0
19
)
Copied!
- Load the model checkpoints for both the champion and challenger.
1
# loading model weights
2
model = BaseModel(
3
parameters["input_sz"],
4
parameters["input_sz"],
5
parameters["n_classes"]).to(parameters["device"])
6
​
7
checkpoint = torch.load(model_fname, map_location=parameters['device'])
8
model.load_state_dict(checkpoint)
9
model.eval()
Copied!
Step 2: Run inference on new data and evaluate the accuracy
In this step, you need to run inference on the evaluation set for both the champion and the challenger. If the challenger performs better than the champion you will:
- add the tag 'best' to the challenger Run
- remove the tag 'best' from champion Run.
1
# Run inference on validation set and get score
2
champion_score = get_model_score(champion_model, images,labels)
3
challenger_score = get_model_score(challenger_model, images, labels)
4
​
5
# Test challenger model score against champion model score
6
assert challenger_score >= champion_score, \
7
f'''The challenger model accuracy {round(challenger_score*100,2)}
8
lower than threshold {round(champion_score*100,2)}%'''
9
​
10
print(f'''The challenger model with Run ID {challenger_run_id}
11
has accuracy of {challenger_score*100}% that is greater than
12
the current champion is promoted to production''')
13
​
14
print("------------Evaluation test passed!!!------------")
Copied!
Step 3: Push the Challenger(new-best) to production
Now you need to push the challenger to production using tags.
- Stop the challenger run and resume the challenger and former champion Run in the default mode (Asynchronous) so we can change the metadata in the run, in this case, we want to change the tags.
1
new_champion_run = neptune.init(
2
api_token=os.getenv("NEPTUNE_API_TOKEN"),
3
project='<YOUR_PROJECT_NAME>',
4
run = challenger_run_id
5
)
6
​
7
former_champion_run = neptune.init(
8
api_token=os.getenv("NEPTUNE_API_TOKEN"),
9
project='<YOUR_PROJECT_NAME>',
10
run = champion_run_id
11
)
Copied!
1
new_champion_run['sys/tags'].add('best')
2
former_champion_run['sys/tags'].remove('best')
Copied!
Note
With this, you can create another pipeline take the Neptune Run with the tag
best and deploys it to production.Step 4: Create and run a Github action
In this step, you will create the Github Action YAML file that executes the script you created in the previous steps, based on each new code version that's pushed to your repo.
After you commit this project to your Github repository you should see the following Github Action activity:
First, create a Github secret that will help you pass your
api_token securely to your python script as an environment variable. The secret name will be set to
NEPTUNE_API_TOKEN and value to ANONYMOUS.Second, create a YAML file under the folder
.github/workflows then copy and paste the code below1
name: CI/CD
2
on: push
3
jobs:
4
model-eval-promotion:
5
runs-on: ubuntu-latest
6
env:
7
NEPTUNE_API_TOKEN: ${{ secrets.NEPTUNE_API_TOKEN }}
8
steps:
9
- uses: actions/[email protected]
10
- uses: actions/setup-[email protected]
11
with:
12
python-version: 3.8
13
- name: Install dependencies
14
run: |
15
python -m pip install --upgrade pip
16
pip install -r CI_CD/requirements.txt
17
- name: Model Promotion(CI/CD)
18
run: python CI_CD/scripts/model_promotion.py
Copied!
Finally, commit your code to your Github repo!
Summary
In this guide, you learned:
- How to fetch and download metadata from the Neptune Runs into your CI/CD jobs
- How to resume Runs in read-only mode to avoid changing the metadata.
- How to create a GitHub Action that leverages the metadata about your models that you retrieved from Neptune to run a CI/CD Job that can evaluate if the challenger is better than the champion.
See also
Last modified 6mo ago
Export as PDF
Copy link
Contents
Before you start
Step 1: Get the Challenger (new) and Champion (best) model metadata
Query and download model metadata from Neptune
Load model object and data loaders from metadata
Step 2: Run inference on new data and evaluate the accuracy
Step 3: Push the Challenger(new-best) to production
Step 4: Create and run a Github action
Summary
See also