Use Neptune with Docker

​
 
​
 
​See code examples​
​See results in Neptune​
How to use Neptune with Docker 
You can use Neptune in any Python environment to log and retrieve ML metadata, including containerized Python scripts or applications.
In this guide, you will learn how to use the Neptune-client library inside docker to log experimentation metadata.
Keywords: Docker, Docker support, Logging from docker
Before you start
Before you begin, make sure you meet the following prerequisites:
​Have Docker installed​
​Have basic familiarity with Neptune (create run and log metadata to it)​
Step 1: Add Neptune to your Python script
Create a Python script file named training.py that logs ML model training metadata to Neptune.
Neptune snippets
Full script
training.py
1
# Step 1: Initialize Neptune and create new Neptune Run
2
run = neptune.init(
3
    project="<YOUR_WORKSPACE/YOUR_PROJECT>", # use common/showroom
4
    api_token="<YOUR_API_TOKEN>" # use ANONYMOUS
5
)
6
​
7
# Step 2: Log config & pararameters
8
run["config/dataset/path"] = data_dir
9
run["config/dataset/transforms"] = data_tfms
10
run["config/dataset/size"] = dataset_size
11
run["config/params"] = params
12
​
13
# Step 3: Log losses & metrics
14
for i, (x, y) in enumerate(trainloader, 0):
15
​
16
    # Log batch loss
17
    run["metrics/training/batch/loss"].log(loss)
18
​
19
    # Log batch accuracy
20
    run["metrics/training/batch/acc"].log(acc)
21
​
22
​
23
# Stop logging
24
run.stop()
Copied!
training.py
1
import torch
2
import torch.nn as nn
3
import torch.optim as optim
4
from torchvision import datasets, transforms
5
import neptune.new as neptune
6
import os
7
​
8
# Step 1: Initialize Neptune and create new Neptune Run
9
run = neptune.init(
10
    project="<YOUR_WORKSPACE/YOUR_PROJECT>", #common/showroom
11
    api_token="<YOUR_API_TOKEN>" #ANONYMOUS
12
)
13
​
14
data_dir = "data/CIFAR10"
15
compressed_ds = "./data/CIFAR10/cifar-10-python.tar.gz"
16
data_tfms = {
17
    "train": transforms.Compose(
18
        [
19
            transforms.RandomHorizontalFlip(),
20
            transforms.ToTensor(),
21
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
22
        ]
23
    )
24
}
25
​
26
 
27
params = {
28
    "lr": 1e-2,
29
    "batch_size": 128,
30
    "input_size": 32 * 32 * 3,
31
    "n_classes": 10,
32
    "model_filename": "basemodel",
33
}
34
​
35
​
36
trainset = datasets.CIFAR10(data_dir, transform=data_tfms["train"], download=True)
37
trainloader = torch.utils.data.DataLoader(trainset, batch_size=params["batch_size"], shuffle=True)
38
dataset_size = {"train": len(trainset)}
39
​
40
model = BaseModel(params["input_size"], params["input_size"], params["n_classes"])
41
criterion = nn.CrossEntropyLoss()
42
optimizer = optim.SGD(model.parameters(), lr=params["lr"])
43
​
44
# Step 2: Log config & pararameters
45
run["config/dataset/path"] = data_dir
46
run["config/dataset/transforms"] = data_tfms
47
run["config/dataset/size"] = dataset_size
48
run["config/params"] = params
49
​
50
​
51
# Step 3: Log losses & metrics
52
for i, (x, y) in enumerate(trainloader, 0):
53
​
54
    optimizer.zero_grad()
55
    outputs = model.forward(x)
56
    _, preds = torch.max(outputs, 1)
57
    loss = criterion(outputs, y)
58
    acc = (torch.sum(preds == y.data)) / len(x)
59
​
60
    # Log batch loss
61
    run["metrics/training/batch/loss"].log(loss)
62
​
63
    # Log batch accuracy
64
    run["metrics/training/batch/acc"].log(acc)
65
​
66
    loss.backward()
67
    optimizer.step()
68
​
69
# Stop logging
70
run.stop()
Copied!
If you pass your NEPTUNE_API_TOKEN as a docker environment variable, either through docker run command options (described here) or through Dockerfile (described here), you can skip the api_token argument.
Optionally, you can also pass NEPTUNE_API_TOKEN as a docker secret.
Step 2: Create Dockerfile that installs dependencies
Create a Dockerfile and a requirements.txt with all the dependencies needed for this guide, especially Neptune-client.
First, create a file requirements.txt that is going to contain all the libraries you will need for this guide.
requirements.txt
1
neptune-client
2
torch==1.9.0
3
torchvision==0.10.0
Copied!
Now create a Dockerfile that will :
Specify the base container image from which we will build ours from
Install dependencies in the requirements.txt 
Copy our training script to the container image and define the command to execute it. 
Dockerfile
1
# syntax=docker/dockerfile:1
2
FROM python:3.8-slim-buster
3
​
4
RUN apt-get update
5
RUN apt-get -y install gcc
6
​
7
COPY requirements.txt requirements.txt
8
RUN pip3 install -r requirements.txt
9
​
10
# Copy all files in the current dir to the main dir of the container
11
COPY . .
12
CMD [ "python3", "-W ignore" ,"training.py" ]
Copied!
Step 3: Build and run Docker container passing API token
In this step, you are going to do the following:
Build a container image using the Dockerfile we created in the previous step.
Run the container image passing the NEPTUNE_API_TOKEN as a docker environment variable.
Build a docker image from the Dockerfile created in the last step by running the following command:
1
docker build --tag <image-name> . # image-name: neptune-docker
Copied!
Run the docker container image created by passing NEPTUNE_API_TOKEN as a docker environment variable using the following command:
1
# NEPTUNE_API_TOKEN: ANONYMOUS
2
# image-name: neptune-docker 
3
docker run -e NEPTUNE_API_TOKEN="<YOUR_API_TOKEN>" <image-name>                                           
Copied!
You can set up and passNEPTUNE_API_TOKEN as a:
 Docker run command options environment variable​
 Dockerfile environment variable​
 Docker secret​
Finally, you can run your docker container image as you see fit, it could be using Github Actions, on your local or remote machine, and so on. 
Neptune will work with any of the methods described above.
After you run the docker container you will get a link on the terminal similar to: ​https://app.neptune.ai/o/common/showroom/e/SHOW-3102/ with common/showroom replaced by your project, and SHOW-3102 replaced by your Run ID. Click on the link to open the Run in Neptune to watch your model training live.
Training charts in the Run created inside docker container image
Summary
In this guide you have learned how to:
How to set up and use the Neptune Python Client to log metadata to Neptune Runs inside a docker container.
Set up and pass NEPTUNE_API_TOKEN as a docker environment variable.
​
 
​
 
​See code examples​
​See results in Neptune​
See also
Continuous Integration and Delivery(CI/CD)
Resume run
Monitor model training live

How-to guides - Previous

Automation Pipelines

Continuous Integration and Delivery(CI/CD)

Last modified 13d ago

Export as PDF

Copy link

Contents

Before you start

Step 1: Add Neptune to your Python script

Step 2: Create Dockerfile that installs dependencies

Step 3: Build and run Docker container passing API token

Summary