Field types

neptune.types

Types in Neptune can be separated into few categories sharing similar traits and logging/fetching methods:

Atom fields

Fields that you use to store a single value of the given type or a single file. It's possible to upload and fetchFloat, Integer, Boolean, String, Datetime and File fields. GitRef, RunState fields are possible to access, but manual creation or modification of the fields of these types is not possible.

Series fields

Fields that you use to store a series of values or a series of files (e.g. images). Available types are FloatSeries, StringSeries and FileSeries.

FileSet

Field type for holding a larger number of files, when access to a single file is rare. FileSet can be used for a folder with source files, a folder with image examples, etc. Things that can be easily browsed through the web interface, and are typically accessed as a whole.

Sets fields

Fields holding unorganized sets of values. Only one set type currenlty exists - StringSet that is used to interact with the tags of the experiment. It's possible to access it and modify it, but manual creation of the fields of this type is not possible.

Handler

When you access a run's path that doesn't exist yet you obtain a Handler object. Think of it as a wildcard that can become any of the types once you invoke on it specific logging method. If you invoke .upload() it becomes a File type, if you invoke .log(2)it will become a FloatSeries etc. Note that it can also become a Namespace handler if you create a field at a lower level of organization.

Namespace handler

If you access a run's path that is one or more levels higher than an existing field you obtain a Namespace handler object, e.g. when you access params when you have a field params/max_epochs. Namespace handler can be used as a shortcut for accessing fields by only specifying a relative path within the namespace.

Float

Field type representing a floating-point number.

=

Convenience alias for .assign().

.assign()

Assigns the provided value to the field.

Parameters

value

(float) - Value to be stored in a field.

wait

(Boolean, optional, default is False) - If True the client will wait to send all tracked metadata to the server before making the assignment. This makes the call synchronous, see Connection modes guide.

Examples

import neptune.new as neptune
run = neptune.init()
# You can both use the Python assign operator (=)
run["params/lr"] = 0.8
# as well as directly use the .assign method
run["params/lr"].assign(0.8)

.fetch()

Fetches field value from Neptune servers.

Returns

float value stored in the field.

Examples

import neptune.new as neptune
# Continue run with ID "SAN-23"
run = neptune.init(run="SAN-23")
# Fetch best accuracy so far
best_acc = run["train/acc/best"].fetch()

Integer

Field type representing an integer number.

=

Convenience alias for .assign().

.assign()

Assigns the provided value to the field.

Parameters

value

(int) - Value to be stored in a field.

wait

(Boolean, optional, default is False) - If True the client will wait to send all tracked metadata to the server before making the assignment. This makes the call synchronous, see Connection modes guide.

Examples

import neptune.new as neptune
run = neptune.init()
# You can both use the Python assign operator (=)
run["params/max_epochs"] = 10
# as well as directly use the .assign method
run["params/max_epochs"].assign(10)

.fetch()

Fetches field value from Neptune servers.

Returns

int value stored in the field.

Examples

import neptune.new as neptune
# Continue run with ID "SAN-23"
run = neptune.init(run="SAN-23")
# Fetch epoch count
epoch = run["train/epoch"].fetch()

Boolean

Field type representing a boolean value.

=

Convenience alias for .assign().

.assign()

Assigns the provided value to the field.

Parameters

value

(Boolean ) - Value to be stored in a field.

wait

(Boolean, optional, default is False) - If True the client will wait to send all tracked metadata to the server before making the assignment. This makes the call synchronous, see Connection modes guide.

Examples

import neptune.new as neptune
run = neptune.init()
# You can both use the Python assign operator (=)
run["params/use_preprocessing"] = True
# as well as directly use the .assign method
run["params/use_preprocessing"].assign(True)

.fetch()

Fetches field value from Neptune servers.

Returns

bool value stored in the field.

Examples

import neptune.new as neptune
# Continue run with ID "SAN-23"
run = neptune.init(run="SAN-23")
# Fetch use_proprocessing parameter
use_preprocessing = run['params/use_preprocessing'].fetch()

String

Field type representing a small piece of text.

=

Convenience alias for .assign().

.assign()

Assigns the provided value to the field.

Parameters

value

(str or String ) - Value to be stored in a field.

wait

(Boolean, optional, default is False) - If True the client will wait to send all tracked metadata to the server before making the assignment. This makes the call synchronous, see Connection modes guide.

Examples

import neptune.new as neptune
run = neptune.init()
# You can both use the Python assign operator (=)
run["params/optimizer"] = "Adam"
# as well as directly use the .assign method
run["params/optimizer"].assign("Adam")

.fetch()

Fetches field value from Neptune servers.

Returns

str value stored

File

Field type for holding a single file of any type.

.upload()

Uploads provided file under specified field path

Parameters

value

(stror File) - Path to the file to be uploaded or File value object

wait

(Boolean, optional, default is False) - If True the client will wait to send all tracked metadata to the server before making the assignment. This makes the call synchronous, see Connection modes guide.

Examples

import neptune.new as neptune
run = neptune.init()
# Upload example data
run["dataset/data_sample"].upload("sample_data.csv")
# Both the content and the extension is stored
# When downloaded the filename is a combination of path and the extension
run["dataset/data_sample"].download() # data_sample.csv
# Explicitely create File value object
from neptune.new.types import File
run["dataset/data_sample"].upload(File("sample_data.csv"))
# Lot's of types will be implicitly converted to Files on the fly
# E.g. image-like objects such as Matplotlip figures etc.
import matplotlib.pyplot as plt
plt.plot(data)
run["dataset/distribution"].log(plt.gcf()) # .gcf() gets the figure object

=

Convenience alias for .assign().

.assign()

You can also upload a file by assigning the File value object to the specified field path.

Parameters

value

(File) - File value object

wait

(Boolean, optional, default is False) - If True the client will wait to send all tracked metadata to the server before making the assignment. This makes the call synchronous, see Connection modes guide.

Examples

import neptune.new as neptune
from neptune.new.types import File
run = neptune.init()
run["dataset/data_sample"] = File("sample_data.csv")

.download()

Downloads the stored file to the working directory or specified destination.

Parameters

destination

(str, optional, default is None) - Path to where the file should be downloaded. If None file will be downloaded to the working directory.

If destination is a directory, the file will be downloaded to the specified directory with a filename composed from field name and extension (if present).

If destination is a path to a file, the file will be downloaded under the specified name.

.fetch_extension()

A programmatic way to discover an extension of the stored file.

Returns

Str with the extensions of the stored File.

Examples

import neptune.new as neptune
from neptune.new.types import File
run = neptune.init()
# Upload model as a File field
run['model/last'].upload('model.pt')
# Check extension of the uploaded File
ext = run['model/last'].fetch_extension()
ext == 'pt' # True

.as_image()

Static method for converting image objects or image-like objects to an image File value object. This way you can upload Matplotlib figures, PIL images, NumPy arrays, as static images.

Parameters

image

Image-like object to be converted. Supported are PyTorch tensors, TensorFlow/Keras tensors, NumPy arrays, PIL images, Matplotlib figures.

Returns

File value object with converted image

Examples

import neptune.new as neptune
from neptune.new.types import File
run = neptune.init()
# Convert NumPy array to File value object and upload it
run["train/prediction_example"].upload(File.as_image(numpy_array))
# Convert PIL image to File value object and upload it
pil_file = File.as_image(pil_image)
run["dataset/data_sample/img1"].upload(pil_file)
# You can upload PIL image without explicit conversion
run["dataset/data_sample/img2"].upload(pil_image)

.as_html()

Converts an object to an HTML File value object. This way you can upload Altair, Bokeh, Plotly, Matplotlib interactive charts or upload directly Pandas DataFrame objects to explore them in Neptune UI.

Parameters

value

An object to be converted. Supported are Altair, Bokeh, Plotly, Matplotlib interactive charts, and Pandas DataFrame objects.

Returns

File value object with converted object

Examples

import neptune.new as neptune
from neptune.new.types import File
run = neptune.init()
# Convert Pandas DataFrame to File value object and upload it
run["train/results"].upload(File.as_html(df_predictions))
# Convert Altair interactive chart to File value object and upload it
altair_file = File.as_html(altair_chart)
run["dataset/data_sample/img1"].upload(altair_file)
# You can upload Altair interactive chart without explicit conversion
run["dataset/data_sample/img2"].upload(altair_chart)

.as_pickle()

Pickles a Python object and stores it in File value object. This way you can upload any Python object for future use.

Parameters

value

An object to be converted. Supported are Altair, Bokeh, Plotly, Matplotlib interactive charts, and Pandas DataFrame objects.

Returns

File value object with pickled object

Examples

import neptune.new as neptune
from neptune.new.types import File
run = neptune.init()
# Pickle model object and upload it
run["results/pickled_model"].upload(File.as_pickle(trained_model))

.from_content()

Factory method for creating File value objects directly from binary and text content. In the case of text content, UTF-8 encoding will be used.

Parameters

content

(strorbytes) - Text or binary content to stored in the File value object.

extension

(str, optional, default is None) - Extension of the file created that will be used for interpreting the type of content for visualization. If None it will be bin for binary content and txt for text content.

Returns

File value object created from the content

.from_stream()

Factory method for creating File value objects directly from binary and text streams. In the case of text stream, UTF-8 encoding will be used.

Parameters

stream

(IOBase) - Stream to be converted.

seek

(int, optional, default is 0) - See IOBase documentation

extension

(str, optional, default is None) - Extension of the file created that will be used for interpreting the type of content for visualization. If None it will be bin for binary stream and txt for text stream.

Returns

File value object created from the stream.

Datetime

A field representing a time. Datetime fields are possible to access, but manual creation or modification of the fields of these types is not possible.

.fetch()

Fetches field value from Neptune servers.

Returns

datetime value stored

RunState

A field containing the state of the tracked run. RunState fields are possible to access, but manual creation or modification of the fields of these types is not possible. RunState does not expose any methods.

GitRef

A field containing the information about the git repository at the time of starting the tracked run. GitRef fields are possible to access, but manual creation or modification of the fields of these types is not possible. GitRef does not expose any methods.

FloatSeries

A field containing a series of numbers for example training metrics or change of performance of the model on production. The series can be indexed both by index and by time.

.log()

Logs the provided number or a collection of numbers.

Parameters

value

(float or int, or collection of float or int) - The value to be logged or collections of values to be logged. If a collection is provided step and timestamp parameters must be None.

step

(float or int, optional, default is None) - Index of the log entry being appended. Must be strictly increasing.

timestamp

(float or int, optional, default is None) - Time index of the log entry being appended in form of Unix time. If None current time (time.time()) will be used as a timestamp.

wait

(Boolean, optional, default is False) - If True the client will wait to send all tracked metadata to the server. This makes the call synchronous, see Connection modes guide.

.fetch_last()

Fetches last value stored in the series from Neptune servers.

Returns

Last float value logged.

.fetch_values()

Fetches all values stored in the series from Neptune servers.

Parameters

include_timestamp

(Boolean, optional, default is True) - Whether to include the fetched data should include the timestamp field.

Returns

Pandas.DataFrame containing all the values and their indexes stored in the series field.

StringSeries

A field containing a series of text values.

Single text log is limited to 1000 characters, but there is no limit to the number of logs in the series. If the logged text exceeds this limit it will be truncated to match it.

.log()

Logs the provided text or a collection of texts.

Parameters

value

(str or collection of str) - The value to be logged or collections of values to be logged. If a collection is provided step and timestamp parameters must be None.

step

(float or int, optional, default is None) - Index of the log entry being appended. Must be strictly increasing.

timestamp

(float or int, optional, default is None) - Time index of the log entry being appended in form of Unix time. If None current time (time.time()) will be used as a timestamp.

wait

(Boolean, optional, default is False) - If True the client will wait to send all tracked metadata to the server. This makes the call synchronous, see Connection modes guide.

.fetch_last()

Fetches last value stored in the series from Neptune servers.

Returns

Last str value logged.

.fetch_values()

Fetches all values stored in the series from Neptune servers.

Parameters

include_timestamp

(Boolean, optional, default is True) - Whether to include the fetched data should include the timestamp field.

Returns

Pandas.DataFrame containing all the values and their indexes stored in the series field.

FileSeries

A field containing a series of files of any type.

We are still working on support for files of any type. In the meantime, FileSeries fields accept only image-like files.

.log()

Logs the provided file.

Parameters

value

(File value object or collection of File) - The file to be logged or collections of files to be logged. If a collection is provided step and timestamp parameters must be None.

step

(float or int, optional, default is None) - Index of the log entry being appended. Must be strictly increasing.

timestamp

(float or int, optional, default is None) - Time index of the log entry being appended in form of Unix time. If None current time (time.time()) will be used as a timestamp.

name

(str, optional, default is None) - name of the logged file.

description

(str, optional, default is None) - short description of the logged file.

wait

(Boolean, optional, default is False) - If True the client will wait to send all tracked metadata to the server. This makes the call synchronous, see Connection modes guide.

Examples

import neptune.new as neptune
from neptune.new.types import File
run = neptune.init()
# Log image file
run['train/prediction_example'].log(File(path_to_file))
# Log Matplotlib figure with implicit casting
run["train/distribution"].log(plt_histogram, step=epoch)
# This is equivalent to following explicit casting
run["train/distribution"].log(File.as_image(plt_histogram), step=epoch)
# Read more about supported conversion in File.as_image
# Convert NumPy array to File value object and log it
run["train/prediction_example"].log(File.as_image(numpy_array))
# You can also log name and description for the image
# Log data sample
for plt_image, class_name in data_sample:
run['data/sample'].log(plt_image, name=class_name)
# Log predictions with class probabilities
for image, y_pred in zip(x_test_sample, y_test_sample_pred):
description = "\n".join(
["class {}: {}".format(i, pred) for i, pred in enumerate(y_pred)]
)
run['train/predictions'].log(image, description=description)

.download_last()

Downloads the last File stored in the series from Neptune servers and save it locally.

Parameters

destination

(str , optional, default is None) - The directory where the file will be downloaded. If None is passed, the file will be downloaded to the current working directory.

If destination is a directory, the file will be downloaded to the specified directory with a filename composed from field name and extension (if present).

If destination is a path to a file, the file will be downloaded under the specified name.

.download()

Downloads all files stored in the series from Neptune servers.

Parameters

destination

(str , optional, default is None) - The directory where the file will be downloaded. If None is passed, the file will be downloaded to the current working directory.

If destination is a directory, the file will be downloaded to the specified directory with a filename composed from field name and extension (if present).

If destination is a path to a file, the file will be downloaded under the specified name.

StringSet

A field containing unorganized set of strings used for interacting with your run's tags. It's possible to access it and modify it, but manual creation of new fields of this type is not possible.

.add()

Adds the provided tag or tags to the run's tags.

Parameters

values

(str or collection of str) - Tag or tags to be added. Tip: If you want you can use emojis in your tags eg. Exploration 🧪

wait

(Boolean, optional, default is False) - If True the client will wait to send all tracked metadata to the server first. This makes the call synchronous, see Connection modes guide.

.remove()

Removes the provided tag or tags from the set.

Parameters

values

(stror collection of str) - Tag or tags to be removed.

wait

(Boolean, optional, default is False) - If True the client will wait to send all tracked metadata to the server first. This makes the call synchronous, see Connection modes guide.

.clear()

Removes all tags from the StringSet.

Parameters

wait

(Boolean, optional, default is False) - If True the client will wait to send all tracked metadata to the server first. This makes the call synchronous, see Connection modes guide.

.fetch()

Fetches all tags from Neptune servers.

Returns

Run's tags in a form of set of str.

FileSet

Field type for holding a larger number of files, when access to a single file is rare. FileSet can be used for a folder with source files, a folder with image examples, etc. Things that can be easily browsed through the web interface, and are typically accessed as a whole.

.upload_files()

Uploads the provided file or files and stores them inside the FileSet.

Parameters

globs

(stror collection of str) - Path or paths to the files to be uploaded.

wait

(Boolean, optional, default is False) - If True the client will wait to send all tracked metadata to the server before making the assignment. This makes the call synchronous, see Connection modes guide.

.delete_files()

Delete the file or files specified by paths from the FileSet stored on the Neptune servers.

Parameters

paths

(stror collection of str) - Path or paths to files or folders to be deleted. Note that these are paths relative to the FileSet itself e.g. if the FileSet contains file example.txt, varia/notes.txt, varia/data.csv to delete whole subfolder you would pass varia as the argument.

wait

(Boolean, optional, default is False) - If True the client will wait to send all tracked metadata to the server before making the assignment. This makes the call synchronous, see Connection modes guide.

.download()

Downloads all files stored in the field from Neptune servers.

Parameters

destination

(str , optional, default is None) - The directory where the files will be downloaded. If None is passed, the files will be downloaded to the current working directory.

If destination is a directory, the file will be downloaded to the specified directory with a filename composed from field name and extension (if present).

If destination is a path to a file, the file will be downloaded under the specified name.

Artifact

To use artifacts you need at least 0.10.10 version of neptune client.

pip install neptune-client>=0.10.10

Field type for holding datasets, models, and other artifacts.Artifact can be used for a dataset in a CSV format, a folder with training images as PNG files, etc. You should use Artifacts for things that are large and you don't want to upload them but you would still like to version and compare them between Runs.

=

Convenience alias for .assign().

.assign()

Assigns the provided value to the field.

Parameters

value

(str or String ) - Value to be stored in a field.

wait

(Boolean, optional, default is False) - If True the client will wait to send all tracked metadata to the server before making the assignment. This makes the call synchronous, see Connection modes guide.

Examples

import neptune.new as neptune
run = neptune.init(project='<YOUR_WORKSPACE/YOUR_PROJECT>',
api_token='<YOUR_API_TOKEN>')
run["datasets/images"].track_files("./images")
run["datasets/images-copy"].assign(run["datasets/images"].fetch())

.download()

Downloads all the files that are referenced in the field.

Parameters

destination

(str , optional, default is None) - The directory where the files will be downloaded. If None is passed, the files will be downloaded to the current working directory.

If destination is a directory, the file will be downloaded to the specified directory with a filename composed from field name and extension (if present).

If destination is a path to a file, the file will be downloaded under the specified name.

Examples

import neptune.new as neptune
run = neptune.init(project='<YOUR_WORKSPACE/YOUR_PROJECT>',
api_token='<YOUR_API_TOKEN>',
run='AR-2', # Neptune Run ID of a run with artifact
mode='read-only')
run['artifacts/images'].download(destination='datasets/train/images')

.fetch()

Fetches field value from Neptune servers. You should only use .fetch() to copy the artifact.

Returns

Artifact object

Examples

import neptune.new as neptune
run = neptune.init(project='<YOUR_WORKSPACE/YOUR_PROJECT>',
api_token='<YOUR_API_TOKEN>')
run["datasets/images"].track_files("./images")
run["datasets/images-copy"].assign(run["datasets/images"].fetch())

.fetch_files_list()

Fetches the Hash of the artifact from Neptune servers.

Returns

List of ArtifactFileData objects for all the files referenced in the artifacts.

ArtifactFileDatahas the following fields that you can use:

  • file_hash: str, Hash of the file

  • file_path: str, URL of the file in the Neptune UI

  • size: int, Size of the file in KB

  • metadata: dict, dictionary with the following keys:

    • file_path: a location (path) of the file either on local storage or S3-compatible storage

    • last_modified: when was the last the artifact content was changed

Examples

import neptune.new as neptune
run = neptune.init(project='<YOUR_WORKSPACE/YOUR_PROJECT>',
api_token='<YOUR_API_TOKEN>',
run='AR-2', # Neptune Run ID of a run with artifact
mode='read-only')
artifact_list = run['artifacts/images'].fetch_files_list()
artifact_list[0].file_hash
artifact_list[0].file_path
artifact_list[0].metadata['last_modified']

.fetch_hash()

Fetches the Hash of the artifact from Neptune servers.

Returns

Hash of a Neptune artifact as astr.

Examples

import neptune.new as neptune
run = neptune.init(project='<YOUR_WORKSPACE/YOUR_PROJECT>',
api_token='<YOUR_API_TOKEN>',
run='AR-2', # Neptune Run ID of a run with artifact
mode='read-only')
run['artifacts/images'].fetch_hash()

.track_files()

Saves the artifact metadata: version (hash), location (path), size, folder structure, and contents to Neptune. Works for files, folders or S3-compatible storage.

Parameters

path

(str) - File path or S3-compatible path to the file or folder that you want to track.

destination

(str, optional default is None) - location inside Neptune artifact namespace where you want to log the metadata.

wait

(Boolean, optional, default is False) - If True the client will wait to send all tracked metadata to the server before making the assignment. This makes the call synchronous, see Connection modes guide.

Examples

import neptune.new as neptune
run = neptune.init(project='<YOUR_WORKSPACE/YOUR_PROJECT>',
api_token='<YOUR_API_TOKEN>')
run['artifacts/images'].track_files('datasets/train/images')

Namespace handler

An object representing a namespace in your run's metadata structure. You can think of namespaces as folders for organizing your metadata. When you have a fields params/max_epochs and params/lr they will be grouped under a params namespace.

Namespace handler exposes similar methods as the Run object, however, all field paths are relative to the namespace:

import neptune.new as neptune
# You can upload the value through an absolute path
run["params/max_epochs"] = 20
# Or with a relative path and a namepsace handler
params_ns = run["params"]
params_ns["max_epochs"] = 20

[ ] (field lookup)

See Run's [] field lookup. Remember that the resulting paths will be a combination of the namespace path and the provided relative path.

=

Convenience alias for .assign(). Can be used only in combination with [] (field lookup)

.assign()

See Run's .assign(). Remember that the resulting paths will be a combination of the namespace path and the provided relative path.

Examples

import neptune.new as neptune
run = neptune.init()
# Assign a single parameter
run["model/params/batch_size"] = 64
# Access params namespace handler
params_ns = run["model/params"]
# Assign additional parameters in batch
PARAMS = {"max_epochs": 20, "optimizer": "Adam"}
params_ns.assign(PARAMS)
# = needs to be used in combination with [] field lookup
params_ns = PARAMS # Does not work
run["model/params"] = PARAMS # Works
# This also works with relative pahts
model_ns = run["model"]
model_ns["params"] = PARAMS

.fetch()

Fetch values of all non-File Atom fields as a dictionary.

The result will preserve the hierarchical structure of the run's metadata but will contain only non-File Atom fields. You can use this method e.g. to quickly retrieve run parameters.

Returns

dict containing all non-File Atom fields values.

Examples

import neptune.new as neptune
# Resume run with SUN-123 identifier
run = neptune.init(project="my_workspace/my_project", run="SUN-123")
# Fetch the runs parameters
params = run["model/params"].fetch()

Handler

When you access a run's path that doesn't exist yet you obtain a Handler object. Think of it as a wildcard that can become any of the types once you invoke on it specific logging method. If you invoke .upload() it becomes a File type, if you invoke .log(2)it will become a FloatSeries etc. Note that it can also become a Namespace handler if you create a field at a lower level of organization.

Handler object exposes all tracking methods: .assign(), .log(), .upload(), .upload_files(), etc. of all other field types as well as Namespace handler methods.

Examples

import neptune.new as neptune
run = neptune.init()
# Accessing "train/batch/acc" returns a Handler as the field does not exist yet
handler_object = run["train/batch/acc"]
# You can use it as if you would use any other field
handler_object.log(0.7)
# Once the field is created under the path, only the fields methods are accepted
handler_object.log(0.6) # Works
handler_object.upload("image.png") # Error, handler_object is a FloatSeries now