neptunecontrib.monitoring.fairness
¶
Module Contents¶
Functions¶
|
Creates fairness metric charts, calculates fairness classification metrics and logs them to Neptune. |
|
|
|
|
|
|
|
|
|
|
|
|
|
-
neptunecontrib.monitoring.fairness.
log_fairness_classification_metrics
(y_true, y_pred_class, y_pred_score, sensitive_attributes, favorable_label, unfavorable_label, privileged_groups, unprivileged_groups, experiment=None, prefix='')¶ Creates fairness metric charts, calculates fairness classification metrics and logs them to Neptune.
Class-based metrics that are logged: ‘true_positive_rate_difference’,’false_positive_rate_difference’, ‘false_omission_rate_difference’, ‘false_discovery_rate_difference’, ‘error_rate_difference’, ‘false_positive_rate_ratio’, ‘false_negative_rate_ratio’, ‘false_omission_rate_ratio’, ‘false_discovery_rate_ratio’, ‘error_rate_ratio’, ‘average_odds_difference’, ‘disparate_impact’, ‘statistical_parity_difference’, ‘equal_opportunity_difference’, ‘theil_index’, ‘between_group_theil_index’, ‘between_all_groups_theil_index’, ‘coefficient_of_variation’, ‘between_group_coefficient_of_variation’, ‘between_all_groups_coefficient_of_variation’, ‘generalized_entropy_index’, ‘between_group_generalized_entropy_index’, ‘between_all_groups_generalized_entropy_index’
Charts are logged to the ‘metric_by_group’ channel: ‘confusion matrix’, ‘TPR’, ‘TNR’, ‘FPR’, ‘FNR’, ‘PPV’, ‘NPV’, ‘FDR’, ‘FOR’, ‘ACC’, ‘error_rate’, ‘selection_rate’, ‘power’, ‘precision’, ‘recall’, ‘sensitivity’, ‘specificity’.
- Parameters
y_true (array-like, shape (n_samples)) – Ground truth (correct) target values.
y_pred_class (array-like, shape (n_samples)) – Class predictions with values 0 or 1.
y_pred_score (array-like, shape (n_samples)) – Class predictions with values from 0 to 1. Default None.
sensitive_attributes (pandas.DataFrame, shape (n_samples, k)) – datafame containing only sensitive columns.
favorable_label (str or int) – label that is favorable, brings positive value to a person being classified.
unfavorable_label (str or int) – label that is unfavorable, brings positive value to a person being classified.
privileged_groups (dict) – dictionary with column names and list of values for those columns that belong to the privileged groups.
unprivileged_groups (dict) – dictionary with column names and list of values for those columns that belong to the unprivileged groups.
experiment (neptune.experiments.Experiment) – Neptune experiment. Default is None.
prefix (str) – Prefix that will be added before metric name when logged to Neptune.
Examples
Train the model and make predictions on test. Log metrics and performance curves to Neptune:
import neptune from neptunecontrib.monitoring.fairness import log_fairness_classification_metrics neptune.init() with neptune.create_experiment(): log_fairness_classification_metrics(y_true, y_pred_class, y_pred_score, test[['race']], favorable_label='granted_parole', unfavorable_label='not_granted_parole', privileged_groups={'race':['Caucasian']}, privileged_groups={'race':['African-American','Hispanic]}, )
Check out this experiment https://ui.neptune.ai/jakub-czakon/model-fairness/e/MOD-92/logs.
-
neptunecontrib.monitoring.fairness.
_make_dataset
(features, labels, scores=None, protected_columns=None, privileged_groups=None, unprivileged_groups=None, favorable_label=None, unfavorable_label=None)¶
-
neptunecontrib.monitoring.fairness.
_fmt_priveleged_info
(privileged_groups, unprivileged_groups)¶
-
neptunecontrib.monitoring.fairness.
_log_fairness_metrics
(aif_metric, experiment, prefix)¶
-
neptunecontrib.monitoring.fairness.
_plot_confusion_matrix_by_group
(aif_metric, figsize=None)¶
-
neptunecontrib.monitoring.fairness.
_plot_performance_by_group
(aif_metric, metric_name, ax=None)¶
-
neptunecontrib.monitoring.fairness.
_add_annotations
(ax)¶
-
neptunecontrib.monitoring.fairness.
_format_aif360_to_sklearn
(aif360_mat)¶