Skip to main content
App version: 3.20251215

Query metadata

To query the logged metadata programmatically, use the Neptune Query API. You can:

  • Fetch experiment metadata as a data frame.
  • Define filters to query experiments, runs, and attributes that meet certain criteria.
  • List experiments, runs, and attributes of a project.

The extended regular expression syntax is supported for matching names and other string values.

Setup​

To install the Neptune Query API client:

pip install "neptune-query<2.0.0"

Set your Neptune API token and project name as environment variables:

In your .profile or other shell initialization file, use the export commands:

export NEPTUNE_API_TOKEN="h0dHBzOi8aHR0cHM.4kl0jvYh3Kb8...ifQ=="
export NEPTUNE_PROJECT="workspace-name/project-name"

For help, see Get started.

You can also pass the project path to the project argument of any querying function.

Usage​

import neptune_query as nq

Available functions:

FunctionDescription
download_files()Download files from the specified experiments.
fetch_experiments_table()Fetch experiment metadata as a data frame with attributes as columns.
fetch_metrics()Fetch values of numerical series, one step per row.
fetch_metric_buckets()Fetch metric summary values split by X-axis ranges.
fetch_series()Fetch series of strings, files, or histograms.
list_attributes()List all logged attributes of the target project's experiments.
list_experiments()List names of experiments in the target project.
set_api_token()Set the Neptune API token to use for the session.

Runs API​

You can target individual runs by ID instead of experiment runs by name. To use the corresponding methods for runs, import the runs module:

import neptune_query.runs as nq_runs


nq_runs.fetch_metrics(...)

Available functions:

FunctionDescription
download_files()Download files from the specified runs.
fetch_metrics()Fetch values of numerical series, one step per row.
fetch_metric_buckets()Fetch summary values split by X-axis ranges.
fetch_runs_table()Fetch run metadata as a data frame with attributes as columns.
fetch_series()Fetch series of strings, files, or histograms.
list_attributes()List all logged attributes of the target project's runs.
list_runs()List IDs of runs in the target project.
tip

To get runs belonging to a specific experiment, see Define filters: Runs of a specific experiment.

Example 1: Fetch metric values per step​

To fetch numerical values at each step, use fetch_metrics():

  • To specify or filter experiments to return, use the experiments parameter.
  • To specify attributes to include as columns, use the attributes parameter.
  • To limit the returned values, use the available parameters.
Fetch last 5 values from specific experiment's matching attributes
nq.fetch_metrics(
    experiments=["exp_dczjz"],
    attributes=r"metrics/val_.+_estimated$",
    tail_limit=5,
)
Sample output
                    metrics/val_accuracy_estimated  metrics/val_loss_estimated
experiment    step
exp_dczjz   1000.0                         0.72187                     0.14989
            1001.0                         0.72683                     0.14781
            1002.0                         0.76981                     0.14233
            1003.0                         0.76995                     0.13552
            1004.0                         0.78161                     0.12020

For a full guide, see Query metric values.

Example 2: Fetch experiments data frame​

To fetch experiment metadata from your project, use fetch_experiments_table(). The output mimics the runs table of the web app.

  • To specify or filter experiments to return, use the experiments parameter.
  • To specify attributes to include as columns, use the attributes parameter.
Fetch specific attribute values from matching experiments
nq.fetch_experiments_table(
    experiments=r"^exp_",
    attributes=["metrics/train_accuracy", "metrics/train_loss", "learning_rate"],
)
Sample output
            metrics/train_accuracy   metrics/train_loss   learning_rate
experiment
exp_ergwq                 0.278149             0.336344            0.01
exp_qgguv                 0.160260             0.790268            0.02
exp_hstrj                 0.365521             0.459901            0.01

For series attributes, the value of the last logged step is returned.

Example 3: Define filters​

List my experiments that have a "dataset_version" attribute and "val/loss" less than 0.1:

from neptune_query.filters import Filter


owned_by_me = Filter.eq("sys/owner", "sigurd")
dataset_check = Filter.exists("dataset_version")
loss_filter = Filter.lt("val/loss", 0.1)

interesting = owned_by_me & dataset_check & loss_filter
nq.list_experiments(experiments=interesting)
Sample output
['exp_ergwq', 'exp_qgguv', 'exp_hstrj']

Then fetch configs from the experiments, including also the interesting metric:

nq.fetch_experiments_table(
    experiments=interesting,
    attributes=r"config/ | val/loss",
)
Sample output
            config/optimizer  config/batch_size  config/learning_rate   val/loss
experiment
exp_ergwq               Adam                 32                 0.001     0.0901
exp_qgguv           Adadelta                 32                 0.002     0.0876
exp_hstrj           Adadelta                 64                 0.001     0.0891

For a full guide, see Define filters.

Exclude archived runs​

To exclude archived experiments or runs from the results, create a filter on the sys/archived attribute:

import neptune_query as nq
from neptune_query.filters import Filter


exclude_archived = Filter.eq("sys/archived", False)
nq.fetch_experiments_table(experiments=exclude_archived)

To use this filter in combination with other criteria, use the & operator to join multiple filters:

name_matches_regex = Filter.name(r"^exp_")
exclude_archived = Filter.eq("sys/archived", False)

nq.fetch_experiments_table(experiments=name_matches_regex & exclude_archived)