Use Neptune with Docker
How to use Neptune with Docker
You can use Neptune in any Python environment to log and retrieve ML metadata, including containerized Python scripts or applications.
In this guide, you will learn how to use the Neptune-client library inside docker to log experimentation metadata.
Keywords: Docker, Docker support, Logging from docker

Before you start

Before you begin, make sure you meet the following prerequisites:

Step 1: Add Neptune to your Python script

Create a Python script file named training.py that logs ML model training metadata to Neptune.
Neptune snippets
Full script
training.py
1
# Step 1: Initialize Neptune and create new Neptune Run
2
run = neptune.init(
3
project="<YOUR_WORKSPACE/YOUR_PROJECT>", # use common/showroom
4
api_token="<YOUR_API_TOKEN>" # use ANONYMOUS
5
)
6
7
# Step 2: Log config & pararameters
8
run["config/dataset/path"] = data_dir
9
run["config/dataset/transforms"] = data_tfms
10
run["config/dataset/size"] = dataset_size
11
run["config/params"] = params
12
13
# Step 3: Log losses & metrics
14
for i, (x, y) in enumerate(trainloader, 0):
15
16
# Log batch loss
17
run["metrics/training/batch/loss"].log(loss)
18
19
# Log batch accuracy
20
run["metrics/training/batch/acc"].log(acc)
21
22
23
# Stop logging
24
run.stop()
Copied!
training.py
1
import torch
2
import torch.nn as nn
3
import torch.optim as optim
4
from torchvision import datasets, transforms
5
import neptune.new as neptune
6
import os
7
8
# Step 1: Initialize Neptune and create new Neptune Run
9
run = neptune.init(
10
project="<YOUR_WORKSPACE/YOUR_PROJECT>", #common/showroom
11
api_token="<YOUR_API_TOKEN>" #ANONYMOUS
12
)
13
14
data_dir = "data/CIFAR10"
15
compressed_ds = "./data/CIFAR10/cifar-10-python.tar.gz"
16
data_tfms = {
17
"train": transforms.Compose(
18
[
19
transforms.RandomHorizontalFlip(),
20
transforms.ToTensor(),
21
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
22
]
23
)
24
}
25
26
27
params = {
28
"lr": 1e-2,
29
"batch_size": 128,
30
"input_size": 32 * 32 * 3,
31
"n_classes": 10,
32
"model_filename": "basemodel",
33
}
34
35
36
trainset = datasets.CIFAR10(data_dir, transform=data_tfms["train"], download=True)
37
trainloader = torch.utils.data.DataLoader(trainset, batch_size=params["batch_size"], shuffle=True)
38
dataset_size = {"train": len(trainset)}
39
40
model = BaseModel(params["input_size"], params["input_size"], params["n_classes"])
41
criterion = nn.CrossEntropyLoss()
42
optimizer = optim.SGD(model.parameters(), lr=params["lr"])
43
44
# Step 2: Log config & pararameters
45
run["config/dataset/path"] = data_dir
46
run["config/dataset/transforms"] = data_tfms
47
run["config/dataset/size"] = dataset_size
48
run["config/params"] = params
49
50
51
# Step 3: Log losses & metrics
52
for i, (x, y) in enumerate(trainloader, 0):
53
54
optimizer.zero_grad()
55
outputs = model.forward(x)
56
_, preds = torch.max(outputs, 1)
57
loss = criterion(outputs, y)
58
acc = (torch.sum(preds == y.data)) / len(x)
59
60
# Log batch loss
61
run["metrics/training/batch/loss"].log(loss)
62
63
# Log batch accuracy
64
run["metrics/training/batch/acc"].log(acc)
65
66
loss.backward()
67
optimizer.step()
68
69
# Stop logging
70
run.stop()
Copied!
If you pass your NEPTUNE_API_TOKEN as a docker environment variable, either through docker run command options (described here) or through Dockerfile (described here), you can skip the api_token argument.
Optionally, you can also pass NEPTUNE_API_TOKEN as a docker secret.

Step 2: Create Dockerfile that installs dependencies

Create a Dockerfile and a requirements.txt with all the dependencies needed for this guide, especially Neptune-client.
First, create a file requirements.txt that is going to contain all the libraries you will need for this guide.
requirements.txt
1
neptune-client
2
torch==1.9.0
3
torchvision==0.10.0
Copied!
Now create a Dockerfile that will :
    Specify the base container image from which we will build ours from
    Install dependencies in the requirements.txt
    Copy our training script to the container image and define the command to execute it.
Dockerfile
1
# syntax=docker/dockerfile:1
2
FROM python:3.8-slim-buster
3
4
RUN apt-get update
5
RUN apt-get -y install gcc
6
7
COPY requirements.txt requirements.txt
8
RUN pip3 install -r requirements.txt
9
10
# Copy all files in the current dir to the main dir of the container
11
COPY . .
12
CMD [ "python3", "-W ignore" ,"training.py" ]
Copied!

Step 3: Build and run Docker container passing API token

In this step, you are going to do the following:
Build a docker image from the Dockerfile created in the last step by running the following command:
1
docker build --tag <image-name> . # image-name: neptune-docker
Copied!
Run the docker container image created by passing NEPTUNE_API_TOKEN as a docker environment variable using the following command:
1
# NEPTUNE_API_TOKEN: ANONYMOUS
2
# image-name: neptune-docker
3
docker run -e NEPTUNE_API_TOKEN="<YOUR_API_TOKEN>" <image-name>
Copied!
You can set up and passNEPTUNE_API_TOKEN as a:
Finally, you can run your docker container image as you see fit, it could be using Github Actions, on your local or remote machine, and so on.
Neptune will work with any of the methods described above.
After you run the docker container you will get a link on the terminal similar to: https://app.neptune.ai/o/common/showroom/e/SHOW-3102/ with common/showroom replaced by your project, and SHOW-3102 replaced by your Run ID. Click on the link to open the Run in Neptune to watch your model training live.
Training charts in the Run created inside docker container image

Summary

In this guide you have learned how to:
    How to set up and use the Neptune Python Client to log metadata to Neptune Runs inside a docker container.
    Set up and pass NEPTUNE_API_TOKEN as a docker environment variable.

See also

Last modified 13d ago