Introduction to fairGATE

fairGATE corrects an existing prediction model so that its errors are spread more evenly across protected groups, and reports for every person how far the prediction was moved and how stable that adjustment is. This vignette walks through the method on the Adult income sample shipped with the package (1,000 people from the UCI Adult data; outcome: income above USD 50,000; protected attribute: sex), then shows a continuous outcome on simulated data with the neural-network base learner.

The code chunks that fit models need the suggested packages torch (with its LibTorch backend) and xgboost. Both were available when this vignette was built, so every chunk was run.

Data

library(fairGATE)
data("adult_ready_small", package = "fairGATE")
d <- adult_ready_small
y <- d$income
group <- as.integer(d$sex == "Female")        # 0 = male (reference), 1 = female
X <- as.matrix(d[, setdiff(names(d), c("income", "sex"))])
dim(X)
#> [1] 1000  106
table(sex = group, income = y)
#>    income
#> sex   0   1
#>   0 457 164
#>   1 339  40

The protected attribute is kept out of X: fairGATE uses it only in the fairness objective, in the target of the disparity signal and in evaluation.

Step 1: a cross-fitted base learner

The base learner is any accurate model fitted without the protected attribute. fairgate_base_learner() fits XGBoost or a neural network and cross-fits it: training rows receive 5-fold out-of-fold scores, held-out rows the prediction of a fit on all training rows.

set.seed(2024)
test <- sort(sample(nrow(X), 200))
train <- setdiff(seq_len(nrow(X)), test)
b <- fairgate_base_learner(X, y, "binomial", train = train, test = test,
                           base_learner = "xgboost",
                           params = list(nrounds = 150, eta = 0.05, max_depth = 3))
b
#> Cross-fitted XGBoost base learner (binary outcome), 5 folds
#>   training rows: 800 (out-of-fold AUC 0.8936)
#>   held-out rows: 200 (AUC 0.8774)
#>   hyperparameters: nrounds = 150, eta = 0.05, max_depth = 3, min_child_weight = 1, subsample = 0.8, colsample_bytree = 0.8, lambda = 1, alpha = 0, early_stopping_rounds = 50

With params = NULL the hyperparameters are tuned first on an inner split by fairgate_tune_base_learner(); the default grids are listed by fairgate_base_grid("xgboost") and fairgate_base_grid("nn").

How unequal are the base learner’s errors? fairgate_group_disparity() reports the average odds difference (the mean of the true- and false-positive rate gaps) at the prevalence-matched threshold, which flags positive the same share of people as the training prevalence.

thr <- stats::quantile(b$test, 1 - mean(y[train]), names = FALSE)
fairgate_group_disparity(y[test], b$test, group[test], reference = 0,
                         family = "binomial", threshold = thr)$per_group
#>   group  n         gap eligible
#> 1     1 80 -0.06321839     TRUE

Step 2: fairGATE in one call

fairgate() fits the base learner, the cross-fitted disparity signal, searches the configuration space on an inner split, selects the configuration with the smallest worst-group disparity among those within 0.01 of the best validation AUC, and fits it with per-person stability intervals. The defaults are the full protocol of the accompanying study (48 configurations, up to 300 epochs, 10 bootstrap replicates). To keep this vignette fast we search 4 configurations with at most 60 epochs and 3 replicates.

fit <- fairgate(X, y, group, family = "binomial", reference_group = 0,
                splits = list(list(train = train, test = test)),
                base_params = list(nrounds = 150, eta = 0.05, max_depth = 3),
                n_draws = 4, epochs = 60, interval_reps = 3, interval_epochs = 5)
#> Warning: No non-reference group passes the minimum-cell rule; the worst group
#> is taken over all groups.
fit
#> fairGATE fit (binary outcome), 1 partition(s), reference group 0
#>   base learner: XGBoost
#>   configurations searched: 4
#>   selected: coupling residual, penalty random_effects, lambda 1, lambda_thr 0, intervention cost 0.02
#>   held-out AUC: base 0.8811, fairGATE 0.8813 (mean over partitions)
#>   mean |adjustment|: 0.0015

With only 1,000 people, women with high income are too few for the minimum-cell rule of the study (at least 10 expected positives and negatives per held-out set), so fairgate() warns and takes the worst group over all groups. Missing values (24 people have no recorded country of birth) are imputed with training-row medians inside every fit.

The search table shows each configuration’s validation AUC and worst-group disparity, and which one the rule selected.

fit$search[, c("config_id", "coupling", "penalty", "lambda", "intervention_cost",
               "perf", "worst", "selected")]
#>   config_id coupling        penalty lambda intervention_cost      perf
#> 1         1   expert    thresholded      1              0.10 0.8890934
#> 2         2 residual random_effects      1              0.02 0.8870992
#> 3         3   expert       variance     64              0.02 0.8285013
#> 4         4   expert       variance      1              0.00 0.8826507
#>         worst selected
#> 1 0.066289124    FALSE
#> 2 0.007043397     TRUE
#> 3 0.057138798    FALSE
#> 4 0.007043397    FALSE

Disparity before and after the correction, on the held-out rows:

p <- fit$predictions
cmp <- function(score) {
  t <- stats::quantile(score, 1 - mean(y[train]), names = FALSE)
  fairgate_group_disparity(p$y, score, p$group, reference = 0,
                           family = "binomial", threshold = t)$worst
}
c(base = cmp(p$base), fairGATE = cmp(p$pred))
#>       base   fairGATE 
#> 0.08531435 0.08531435

With 200 held-out people and one partition these numbers are noisy; the study compares methods over 50 partitions with paired tests.

Step 3: who was moved, and how reliably

fairgate_export() returns one row per held-out person: base score, corrected score, adjustment, the weight left on the base learner, the routing weights, and the bootstrap interval of the adjustment.

ex <- fairgate_export(fit)
head(ex[, c("row", "group", "base", "pred", "adjustment", "base_weight",
            "adj_lo", "adj_hi", "interval_excludes_zero")])
#>   row group       base       pred    adjustment base_weight        adj_lo
#> 1   6     1 0.02085921 0.02068808 -1.711231e-04   0.4346066 -0.0005894727
#> 2  11     1 0.21899530 0.22451966  5.524355e-03   0.2607865  0.0113193608
#> 3  16     0 0.09430359 0.09420569 -9.790436e-05   0.3590426 -0.0009867174
#> 4  24     0 0.43796626 0.43760532 -3.609388e-04   0.2743821 -0.0029882150
#> 5  26     0 0.23683140 0.23663478 -1.966193e-04   0.8952325 -0.0018347357
#> 6  29     0 0.81087232 0.81298816  2.115847e-03   0.1931165 -0.0032726052
#>          adj_hi interval_excludes_zero
#> 1 -4.599560e-04                   TRUE
#> 2  1.392832e-02                   TRUE
#> 3  4.588545e-04                  FALSE
#> 4  4.188494e-06                  FALSE
#> 5 -8.887821e-04                   TRUE
#> 6  3.107396e-03                  FALSE

Summaries by group show how the correction is distributed:

aggregate(cbind(abs_adjustment = abs(adjustment), base_weight,
                stable = interval_excludes_zero) ~ group, data = ex, FUN = mean)
#>   group abs_adjustment base_weight stable
#> 1     0    0.001399574   0.2815211   0.70
#> 2     1    0.001716993   0.2490203   0.75

The intervals come from a small number of warm-started bootstrap replicates and are not calibrated; they rank adjustments by stability rather than give coverage. To write the export to a file use fairgate_export(fit, file.path(tempdir(), "fairgate_export.csv")).

A continuous outcome with the neural-network base learner

For a continuous outcome the fairness criterion is the conditional mean prediction difference within outcome strata, in outcome standard deviations. Here the base learner is a torch multilayer perceptron, tuned on a small grid.

set.seed(7)
n <- 1200
g2 <- sample(0:2, n, replace = TRUE, prob = c(0.6, 0.25, 0.15))
X2 <- cbind(x1 = rnorm(n) + 0.6 * g2, x2 = rnorm(n), x3 = rnorm(n))
y2 <- X2[, 1] + 0.5 * X2[, 2]^2 + rnorm(n)
fit2 <- fairgate(X2, y2, g2, family = "gaussian", reference_group = 0,
                 base_learner = "nn", signal_learner = "nn",
                 base_tune_args = list(n_search = 3, grid = list(epochs = 40L)),
                 n_draws = 3, epochs = 40, interval_reps = 0)
fit2
#> fairGATE fit (continuous outcome), 1 partition(s), reference group 0
#>   base learner: neural network
#>   configurations searched: 3
#>   selected: coupling residual, penalty random_effects, lambda 1, lambda_thr 0, intervention cost 0.02
#>   held-out R-squared: base 0.6402, fairGATE 0.6030 (mean over partitions)
#>   mean |adjustment|: 0.2602

Working with the building blocks

The one-call wrapper is built from exported functions that can be used separately:

fairgate_base_learner() and fairgate_tune_base_learner() for base scores per partition (for example to compare base learners);
train_gnn() with base_score = and coupling = "residual" or "expert" to correct base scores computed elsewhere;
fairgate_disparity_target() for the disparity signal’s target;
fairgate_group_disparity() for per-group and worst-group disparity;
analyse_experts(), analyse_gnn_results() and plot_sankey() for routing and expert analyses, and export_f360_csv() for the AI Fairness 360 toolkit.

For reproducible neural-network fits call torch::torch_set_num_threads(1) before fitting; with several threads the order of floating-point reductions can change between runs.