Continuous Integration and Delivery(CI/CD)

​​
​​
​​
​​
​See code examples on GitHub​
​See Github Actions​
You can use Neptune as a model registry that lets you log, query and download all model-building metadata.
This guide shows how to query and retrieve a Neptune Run and all the model metadata to execute Continuous Integration/Continuous Delivery jobs. 
These jobs can be executed:
On scheduled time
Every time you have new data
Each time there is a code commit 
By the end of this guide, you will get a GitHub Action set up so that you can evaluate if the staging model (challenger) is better than the production model (current best) and promote the new best model to production automatically
Keywords: CI/CD, Continuous Integration, Continuous Delivery, CI/CD machine learning
Before you start
Make sure you meet the following prerequisites before starting:
​Have Python 3.6 or higher installed​
​Have Pytorch 1.8.0 or higher installed​
​Have Neptune installed​
Step 1: Get the Challenger (new) and Champion (best) model metadata
In this step, you will:
1.Get the Neptune Project where our Runs are.
2.Get the Run ID’s of the champion and the challenger models
3.Take the Run ID’s we got and use them to resume the Run in read-only mode so the metadata is not accidentally changed.
4.Fetching and downloading data from Neptune
5.Loading model checkpoints and data
First, you have to get the Neptune project that contains the Runs that you want to evaluate. You do that by using the .get_project() method passing the project name.
1
project = neptune.get_project(api_token=os.getenv("NEPTUNE_API_TOKEN"),
2
                              project='<YOUR_PROJECT_NAME>')
Copied!
Note
You can useproject='common/pytorch-integration'and the api_token = 'ANONYMOUS'to explore without having to create a Neptune account.
The api_token is passed as Github Action environment variable created from a Github secret.  How to set it up is described in a later section.
Important
To create your own private projects, you will need a Neptune account and your personal Neptune API token.
Second, we want to filter Runs and get the champion and challenger's Run ID.
To do that:
Use the .fetch_runs_table() method and tag argument.
Converted our query results into a pandas dataframe using the .to_pandas() method. 
Extract the Run ID  just like you would in a pandas dataframe.
1
# Champion
2
champion_runs_table_df = project.fetch_runs_table(tag='champion').to_pandas()
3
champion_run_id = champion_runs_table_df['sys/id'].values[0]
4
​
5
# Challenger
6
challenger_runs_table_df = project.fetch_runs_table(tag='challenger').to_pandas()
7
challenger_run_id = challenger_runs_table_df['sys/id'].values[0]
Copied!
Take the Run ID  and use it to resume the Run in read-only mode in order to avoid logging or changing existing metadata.
1
champion_run = neptune.init(
2
    api_token=os.getenv("NEPTUNE_API_TOKEN"),
3
    project='<YOUR_PROJECT_NAME>',
4
    run = champion_run_id, 
5
    mode = 'read-only'
6
)
7
​
8
challenger_run = neptune.init(
9
    api_token=os.getenv("NEPTUNE_API_TOKEN"),
10
    project='<YOUR_PROJECT_NAME>',
11
    run = challenger_run_id, 
12
    mode = 'read-only'
13
)
Copied!
Executing this snippet will give you a link like this one:
https://app.neptune.ai/common/pytorch-integration/e/PYTOR1-72
with common/pytorch-integration replaced by your_workspace/your_project_name, and PYTOR1-72replaced by your Run ID.
Query and download model metadata from Neptune
To retrieve metadata from Neptune you should:
Use .fetch() method to retrieve non-file value or values.
1
# fetching non-file values 
2
parameters = run['config/hyperparameters'].fetch()
3
parameters['device'] = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
4
​
5
data_dir = run['config/dataset/path'].fetch()
Copied!
Use .download() method to retrieve files such as the model weights for both the champion and challenger.
1
# Download model weights
2
model_fname = 'model.pth'
3
run['io_files/artifacts/basemodel'].download(f'./{model_fname}')
Copied!
Load model object and data loaders from metadata 
Load the data for the evaluation
1
# loading dataset
2
data_tfms = {
3
    "val": transforms.Compose(
4
        [
5
            transforms.ToTensor(),
6
            transforms.Normalize(
7
                [0.485, 0.456, 0.406], 
8
                [0.229, 0.224, 0.225]
9
            ),
10
        ]
11
    )
12
}
13
​
14
validset = datasets.CIFAR10(
15
    data_dir, train=False, transform=data_tfms["val"], download=True
16
)
17
validloader = torch.utils.data.DataLoader(
18
    validset, batch_size=bs, num_workers=0
19
)
Copied!
Load the model checkpoints for both the champion and challenger.
1
# loading model weights
2
model = BaseModel(
3
        parameters["input_sz"],
4
        parameters["input_sz"], 
5
        parameters["n_classes"]).to(parameters["device"])
6
​
7
checkpoint = torch.load(model_fname, map_location=parameters['device'])
8
model.load_state_dict(checkpoint)
9
model.eval()
Copied!
Step 2: Run inference on new data and evaluate the accuracy
In this step, you need to run inference on the evaluation set for both the champion and the challenger. If the challenger performs better than the champion you will:
 add the tag 'best'  to the challenger Run 
remove the tag 'best' from champion Run.
1
# Run inference on validation set and get score
2
champion_score = get_model_score(champion_model, images,labels)
3
challenger_score = get_model_score(challenger_model, images, labels)
4
​
5
# Test challenger model score against champion model score
6
assert challenger_score >= champion_score, \
7
    f'''The challenger model accuracy {round(challenger_score*100,2)}
8
     lower than threshold {round(champion_score*100,2)}%'''
9
​
10
print(f'''The challenger model with Run ID  {challenger_run_id}
11
has accuracy of {challenger_score*100}% that is greater than 
12
the current champion is promoted to production''')
13
​
14
print("------------Evaluation test passed!!!------------")
Copied!
Step 3: Push the Challenger(new-best) to production
Now you need to push the challenger to production using tags.
Stop the challenger run and resume the challenger and former champion Run in the default mode (Asynchronous) so we can change the metadata in the run, in this case, we want to change the tags.
1
new_champion_run = neptune.init(
2
    api_token=os.getenv("NEPTUNE_API_TOKEN"),
3
    project='<YOUR_PROJECT_NAME>',
4
    run = challenger_run_id
5
)
6
​
7
former_champion_run = neptune.init(
8
    api_token=os.getenv("NEPTUNE_API_TOKEN"),
9
    project='<YOUR_PROJECT_NAME>',
10
    run = champion_run_id
11
)
Copied!
​Swap the tags for challanger and champion models.
1
new_champion_run['sys/tags'].add('best')
2
former_champion_run['sys/tags'].remove('best')
Copied!
Note
With this, you can create another pipeline take the Neptune Run with the tag best and deploys it to production.
Step 4: Create and run a Github action
In this step, you will create the Github Action YAML file that executes the script you created in the previous steps, based on each new code version that's pushed to your repo.
After you commit this project to your Github repository you should see the following Github Action activity:
First, create a Github secret that will help you pass your api_token securely to your python script as an environment variable. 
The secret name will be set to NEPTUNE_API_TOKEN and value to ANONYMOUS.
Second, create a YAML file under the folder .github/workflows then copy and paste the code below
1
name: CI/CD 
2
on: push
3
jobs:
4
  model-eval-promotion:
5
    runs-on: ubuntu-latest
6
    env:
7
      NEPTUNE_API_TOKEN: ${{ secrets.NEPTUNE_API_TOKEN }} 
8
    steps:
9
      - uses: actions/[email protected]
10
      - uses: actions/setup-[email protected]
11
        with:
12
          python-version: 3.8
13
      - name: Install dependencies
14
        run: |
15
          python -m pip install --upgrade pip
16
          pip install -r CI_CD/requirements.txt
17
      - name: Model Promotion(CI/CD)
18
        run: python CI_CD/scripts/model_promotion.py
Copied!
Finally, commit your code to your Github repo! 
Summary
In this guide, you learned:
How to fetch and download metadata from the Neptune Runs into your CI/CD jobs
How to resume Runs in read-only mode to avoid changing the metadata.
How to create a GitHub Action that leverages the metadata about your models that you retrieved from Neptune to run a CI/CD Job that can evaluate if the challenger is better than the champion.
​​
​​
​​
​​
​See code examples on GitHub​
​See Github Actions​
See also
Resume run
Monitor model training live
Model Registry

Use Neptune with Docker

Next - How-to guides

Data Versioning

Last modified 1mo ago

Export as PDF

Copy link

Contents

Before you start

Step 1: Get the Challenger (new) and Champion (best) model metadata

Query and download model metadata from Neptune

Load model object and data loaders from metadata

Step 2: Run inference on new data and evaluate the accuracy

Step 3: Push the Challenger(new-best) to production

Step 4: Create and run a Github action

Summary