You should know
How-to guides
Integrations and Supported Tools
API reference
Administration
Use Neptune with Docker
How to use Neptune with Docker
You can use Neptune in any Python environment to log and retrieve ML metadata, including containerized Python scripts or applications.
In this guide, you will learn how to use the Neptune-client library inside docker to log experimentation metadata.
Keywords: Docker, Docker support, Logging from docker
Before you start
Before you begin, make sure you meet the following prerequisites:
Step 1: Add Neptune to your Python script
Neptune snippets
Full script
training.py
1
# Step 1: Initialize Neptune and create new Neptune Run
2
run = neptune.init(
3
project="<YOUR_WORKSPACE/YOUR_PROJECT>", # use common/showroom
4
api_token="<YOUR_API_TOKEN>" # use ANONYMOUS
5
)
6
​
7
# Step 2: Log config & pararameters
8
run["config/dataset/path"] = data_dir
9
run["config/dataset/transforms"] = data_tfms
10
run["config/dataset/size"] = dataset_size
11
run["config/params"] = params
12
​
13
# Step 3: Log losses & metrics
14
for i, (x, y) in enumerate(trainloader, 0):
15
​
16
# Log batch loss
17
run["metrics/training/batch/loss"].log(loss)
18
​
19
# Log batch accuracy
20
run["metrics/training/batch/acc"].log(acc)
21
​
22
​
23
# Stop logging
24
run.stop()
Copied!
training.py
1
import torch
2
import torch.nn as nn
3
import torch.optim as optim
4
from torchvision import datasets, transforms
5
import neptune.new as neptune
6
import os
7
​
8
# Step 1: Initialize Neptune and create new Neptune Run
9
run = neptune.init(
10
project="<YOUR_WORKSPACE/YOUR_PROJECT>", #common/showroom
11
api_token="<YOUR_API_TOKEN>" #ANONYMOUS
12
)
13
​
14
data_dir = "data/CIFAR10"
15
compressed_ds = "./data/CIFAR10/cifar-10-python.tar.gz"
16
data_tfms = {
17
"train": transforms.Compose(
18
[
19
transforms.RandomHorizontalFlip(),
20
transforms.ToTensor(),
21
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
22
]
23
)
24
}
25
​
26
27
params = {
28
"lr": 1e-2,
29
"batch_size": 128,
30
"input_size": 32 * 32 * 3,
31
"n_classes": 10,
32
"model_filename": "basemodel",
33
}
34
​
35
​
36
trainset = datasets.CIFAR10(data_dir, transform=data_tfms["train"], download=True)
37
trainloader = torch.utils.data.DataLoader(trainset, batch_size=params["batch_size"], shuffle=True)
38
dataset_size = {"train": len(trainset)}
39
​
40
model = BaseModel(params["input_size"], params["input_size"], params["n_classes"])
41
criterion = nn.CrossEntropyLoss()
42
optimizer = optim.SGD(model.parameters(), lr=params["lr"])
43
​
44
# Step 2: Log config & pararameters
45
run["config/dataset/path"] = data_dir
46
run["config/dataset/transforms"] = data_tfms
47
run["config/dataset/size"] = dataset_size
48
run["config/params"] = params
49
​
50
​
51
# Step 3: Log losses & metrics
52
for i, (x, y) in enumerate(trainloader, 0):
53
​
54
optimizer.zero_grad()
55
outputs = model.forward(x)
56
_, preds = torch.max(outputs, 1)
57
loss = criterion(outputs, y)
58
acc = (torch.sum(preds == y.data)) / len(x)
59
​
60
# Log batch loss
61
run["metrics/training/batch/loss"].log(loss)
62
​
63
# Log batch accuracy
64
run["metrics/training/batch/acc"].log(acc)
65
​
66
loss.backward()
67
optimizer.step()
68
​
69
# Stop logging
70
run.stop()
Copied!
If you pass your
NEPTUNE_API_TOKEN as a docker environment variable, either through docker run command options (described here) or through Dockerfile (described here), you can skip the api_token argument.Step 2: Create Dockerfile that installs dependencies
Create a
Dockerfile and a requirements.txt with all the dependencies needed for this guide, especially Neptune-client.First, create a file
requirements.txt that is going to contain all the libraries you will need for this guide.requirements.txt
1
neptune-client
2
torch==1.9.0
3
torchvision==0.10.0
Copied!
- Specify the base container image from which we will build ours from
- Copy our training script to the container image and define the command to execute it.
Dockerfile
1
# syntax=docker/dockerfile:1
2
FROM python:3.8-slim-buster
3
​
4
RUN apt-get update
5
RUN apt-get -y install gcc
6
​
7
COPY requirements.txt requirements.txt
8
RUN pip3 install -r requirements.txt
9
​
10
# Copy all files in the current dir to the main dir of the container
11
COPY . .
12
CMD [ "python3", "-W ignore" ,"training.py" ]
Copied!
Step 3: Build and run Docker container passing API token
In this step, you are going to do the following:
- Build a container image using the Dockerfile we created in the previous step.
Build a docker image from the Dockerfile created in the last step by running the following command:
1
docker build --tag <image-name> . # image-name: neptune-docker
Copied!
Run the docker container image created by passing
NEPTUNE_API_TOKEN as a docker environment variable using the following command:1
# NEPTUNE_API_TOKEN: ANONYMOUS
2
# image-name: neptune-docker
3
docker run -e NEPTUNE_API_TOKEN="<YOUR_API_TOKEN>" <image-name>
Copied!
Finally, you can run your docker container image as you see fit, it could be using Github Actions, on your local or remote machine, and so on.
Neptune will work with any of the methods described above.
After you run the docker container you will get a link on the terminal similar to: ​https://app.neptune.ai/o/common/showroom/e/SHOW-3102/ with common/showroom replaced by your project, and SHOW-3102 replaced by your Run ID. Click on the link to open the Run in Neptune to watch your model training live.
Training charts in the Run created inside docker container image
Summary
In this guide you have learned how to:
- How to set up and use the Neptune Python Client to log metadata to Neptune Runs inside a docker container.
- Set up and pass
NEPTUNE_API_TOKENas a docker environment variable.
See also
Last modified 6mo ago