Accumulated Local Effects (ALE) were initially developed as a model-agnostic approach for global explanations of the results of black-box machine learning algorithms (Apley, Daniel W., and Jingyu Zhu. ‘Visualizing the effects of predictor variables in black box supervised learning models.’ Journal of the Royal Statistical Society Series B: Statistical Methodology 82.4 (2020): 1059-1086 doi:10.1111/rssb.12377). ALE has at least two primary advantages over other approaches like partial dependency plots (PDP) and SHapley Additive exPlanations (SHAP): its values are not affected by the presence of interactions among variables in a model and its computation is relatively rapid. This package reimplements the original algorithm from the {ALEPlot} package and reimplements the plotting of ALE values. It also extends the original ALE concept to add bootstrap-based confidence intervals and ALE-based statistics that can be used for statistical inference.
For more details, see Okoli, Chitu. 2023. “Statistical Inference Using Machine Learning and Classical Techniques Based on Accumulated Local Effects (ALE).” arXiv. doi:10.48550/arXiv.2310.09877.
This vignette demonstrates the basic functionality of the {ale} package on standard large datasets used for machine learning. A separate vignette is devoted to its use on small datasets, as is often the case with statistical inference. (How small is small? That’s a tough question, but as that vignette explains, most datasets of less than 2000 rows are probably “small” and even many datasets that are more than 2000 rows are nonetheless “small”.) Other vignettes introduce ALE-based statistics for statistical inference and show how the {ale} package handles various datatypes of input variables.
We begin by loading the necessary libraries.
library(ale)
Attaching package: 'ale'
The following object is masked from 'package:base':
get
library(dplyr)
Attaching package: 'dplyr'
The following objects are masked from 'package:stats':
filter, lag
The following objects are masked from 'package:base':
intersect, setdiff, setequal, union
1 diamonds dataset
For this introduction, we use the diamonds dataset, included with the {ggplot2} graphics system. We cleaned the original version by removing duplicates and invalid entries where the length (x), width (y), or depth (z) is 0.
# Clean up some invalid entriesdiamonds <- ggplot2::diamonds |>filter(!(x ==0| y ==0| z ==0)) |># https://lorentzen.ch/index.php/2021/04/16/a-curious-fact-on-the-diamonds-dataset/distinct( price, carat, cut, color, clarity,.keep_all =TRUE ) |>rename(x_length = x,y_width = y,z_depth = z,depth_pct = depth )summary(diamonds)
carat cut color clarity depth_pct
Min. :0.2000 Fair : 1492 D:4658 SI1 :9857 Min. :43.00
1st Qu.:0.5200 Good : 4173 E:6684 VS2 :8227 1st Qu.:61.00
Median :0.8500 Very Good: 9714 F:6998 SI2 :7916 Median :61.80
Mean :0.9033 Premium : 9657 G:7815 VS1 :6007 Mean :61.74
3rd Qu.:1.1500 Ideal :14703 H:6443 VVS2 :3463 3rd Qu.:62.60
Max. :5.0100 I:4556 VVS1 :2413 Max. :79.00
J:2585 (Other):1856
table price x_length y_width
Min. :43.00 Min. : 326 Min. : 3.730 Min. : 3.680
1st Qu.:56.00 1st Qu.: 1410 1st Qu.: 5.160 1st Qu.: 5.170
Median :57.00 Median : 3365 Median : 6.040 Median : 6.040
Mean :57.58 Mean : 4686 Mean : 6.009 Mean : 6.012
3rd Qu.:59.00 3rd Qu.: 6406 3rd Qu.: 6.730 3rd Qu.: 6.720
Max. :95.00 Max. :18823 Max. :10.740 Max. :58.900
z_depth
Min. : 1.070
1st Qu.: 3.190
Median : 3.740
Mean : 3.711
3rd Qu.: 4.150
Max. :31.800
Here is the description of the modified dataset.
Variable
Description
price
price in US dollars ($326–$18,823)
carat
weight of the diamond (0.2–5.01)
cut
quality of the cut (Fair, Good, Very Good, Premium, Ideal)
color
diamond color, from D (best) to J (worst)
clarity
a measurement of how clear the diamond is (I1 (worst), SI2, SI1, VS2, VS1, VVS2, VVS1, IF (best))
x_length
length in mm (0–10.74)
y_width
width in mm (0–58.9)
z_depth
depth in mm (0–31.8)
depth_pct
total depth percentage = z / mean(x, y) = 2 * z / (x + y) (43–79)
table
width of top of diamond relative to widest point (43–95)
Min. 1st Qu. Median Mean 3rd Qu. Max.
326 1410 3365 4686 6406 18823
Interpretable machine learning (IML) techniques like ALE should be applied on the same dataset that was used to train the model. An explanation is an explanation of a trained model and a trained model is intrinsically linked to the dataset on which it is trained. (When a dataset is too small to feasibly split into training and test sets, then the ale package has tools to appropriately handle such small datasets.
2 Modelling with a random forest
ALE is a model-agnostic IML approach, that is, it works with any kind of machine learning model. As such, {ale} works with any R model with the only condition that it can predict numeric outcomes (such as raw estimates for regression and probabilities or odds ratios for classification). For this demonstration, we use a random forest from the {ranger} package. A random forest combines predictions from multiple decision trees and can capture nonlinear effects and interactions. The examples here will work with any statistical or machine learning algorithm that produces numeric predictions.
We train a random forest to predict diamond prices:
rf_diamonds <- ranger::ranger( price ~ ., diamonds, # train for IML on full datasetmtry =7, # number of variables per treemin.node.size =4, # minimum leaf (terminal node) sizesample.fraction =0.465, # fraction of rows to sample# small forest still accurate for a large dataset; 10× faster than 500 treesnum.trees =50,seed =0# keep the random forest consistent)rf_diamonds
Ranger result
Call:
ranger::ranger(price ~ ., diamonds, mtry = 7, min.node.size = 4, sample.fraction = 0.465, num.trees = 50, seed = 0)
Type: Regression
Number of trees: 50
Sample size: 39739
Number of independent variables: 9
Mtry: 7
Target node size: 4
Variable importance mode: none
Splitrule: variance
OOB prediction error (MSE): 379704.8
R squared (OOB): 0.9784723
3 Creating an ALE object
The core object in the {ale} package is the {S7}ALE object. Its effect property stores the ALE data and, optionally, ALE statistics and bootstrap data for one or more outcome categories. The first argument to the ALE() constructor is a model object–any R model object that can generate numeric predictions is acceptable. By default, ALE() calculates 1D (or “first-order”) ALE for every predictor in the model’s training data. Ranger models do not retain their training data, so we supply data = diamonds explicitly. The outcome column can be detected from this model; for models where it cannot be detected, also supply y_col.
# For faster processing, set the number of CPU cores available. See help(ALE).options(ale.parallel =2)
For faster demonstrations, this vignette uses precreated ALE objects. For the full experience, you can uncomment the relevant lines in the code below.
# For speed, load pre-created objects from the current wip branch.serialized_objects_site <-"https://github.com/tripartio/ale/raw/wip/download"
# To run the slow code yourself, uncomment and execute the following call.# ale_rf_diamonds <- ALE(# rf_diamonds,# data = diamonds# )ale_rf_diamonds <- serialized_objects_site |>file.path("ale_rf_diamonds.0.5.3.rds") |>url() |>readRDS()# Print a statistical summary of the ALE objectsummary(ale_rf_diamonds)
The summary() method reports the model and data represented by the object, followed by its ALE statistics and statistically significant confidence regions when available. It returns the ALE object invisibly, so it is intended for inspection rather than data extraction. The default constructor calculates ALE statistics, although p-values and their confidence regions are not calculated unless p-values are supplied or generated during bootstrapping. See the ALE statistics vignette for their interpretation.
Most core functions in the {ale} package support parallel processing. You can enable parallelization globally with options(ale.parallel = x), where “x” is the number of CPU cores you want to use. It can also be enabled for specific function runs with the parallel argument. Parallelization is disabled by default with parallel = 0. See the documentation for ALE() for details.
To access the plot for a specific variable, we must first create an ALEPlots object by calling the plot() method on the ALE object which internally generates ggplot objects with the full flexibility of {ggplot2}:
# Generate an ALEPlots objectdiamonds_plots <-plot(ale_rf_diamonds)
To retrieve a specific variable plot, you can use the get() method of the ALEPlots object. For example, to access and print the carat ALE plot, we can simply refer to get(diamonds_plots, 'carat'):
# Print a plot by entering its referenceget(diamonds_plots, 'carat')
To display all the ALE plots in an ALEPlots object, we can simply call its print() or plot() methods. Behind the scenes, they use the {patchwork} package to arrange multiple plots in a common plot grid using patchwork::wrap_plots(), so we can pass arguments from that function. For example, we can specify that we want two plots per row with the ncol argument:
# Print all plotsplot(diamonds_plots, ncol =2)
4 Selecting variables and interactions
The x_cols argument controls the effects that ALE() calculates. Its list syntax can request 1D effects, 2D interactions, or both in the same object. For example, the following requests every 1D effect and only the interaction between carat and clarity:
These formats can be combined with other selection options. See help(resolve_x_cols) for the complete set of formats, including ways to request all interactions involving a particular variable and to exclude variables.
5 Bootstrapped ALE
One of the key features of the ALE package is bootstrapping of the ALE results to ensure that the results are reliable, that is, generalizable to data beyond the sample on which the model was trained. As mentioned above, this assumes that IML analysis is carried out on a model whose hyperparameters were determined by cross-validation. When samples are too small for cross-validation, we provide a different approach by bootstrapping the entire model with a ModelBoot object, explained in the vignette for small datasets.
Although ALE is faster than most other IML techniques for global explanation such as partial dependence plots (PDP) and SHAP, it still requires some time to run. Bootstrapping multiplies much of that time by the number of bootstrap iterations.
We create bootstrapped ALE data and plots using the boot_it argument. ALE is a relatively stable IML algorithm (compared to others like PDP), so 100 bootstrap samples should be sufficient for relatively stable results, especially for model development. Final results could be confirmed with 1000 bootstrap samples or more, but there should not be much difference in the results beyond 100 iterations.
# To run the slow code yourself, uncomment and execute the following call.# ale_rf_diamonds_boot <- ALE(# rf_diamonds,# data = diamonds,# # Request all 1D effects and two specific 2D interactions.# x_cols = list(d1 = TRUE, d2 = c("carat:clarity", "cut:table")),# boot_it = 100# )ale_rf_diamonds_boot <- serialized_objects_site |>file.path("ale_rf_diamonds_boot.0.5.3.rds") |>url() |>readRDS()# Bootstrapping produces confidence intervals for the 1D effectsplot(ale_rf_diamonds_boot) |>subset(list(d1 =TRUE)) |>print(ncol =2)
Compare the bootstrapped confidence intervals with the single (non-bootstrapped) ALE curves above. The intervals show how much the estimated effects vary when the training data are resampled while keeping the fitted forest fixed. Use the bootstrapped results to assess the reliability of the patterns rather than relying on a single curve.
By default, requesting bootstrapping while retaining ALE statistics also asks ALE() to generate fast surrogate p-values. These are useful for exploratory analysis but should not be used for definitive or published conclusions. See help(ALEpDist) and the ALE statistics vignette for more rigorous options.
6 ALE interactions
Another advantage of ALE is that it provides data for 2D interactions between variables. Because the bootstrapped object contains the selected carat:clarity and cut:table interactions as well as all 1D effects, the same ALEPlots object provides both kinds of plots. We can use subset() to select just the interactions for a separate grid:
The interaction plot shows the joint effect beyond the separate 1D effects. Grey values fall in the middle range of ALE values; colours away from grey indicate an additional joint effect. The sizes of the squares indicate the relative percentage of actual data in each interaction intersection, helping us assess where the data support the estimated interaction.
Note that ALE interactions are very particular: an ALE interaction means that two variables have a composite effect over and above their separate independent effects. Two predictors can each have strong one-way effects without their combination having an additional interaction effect.
7 Custom prediction functions
ALE() normally calls a model’s prediction method and handles ranger’s prediction format automatically. To demonstrate a custom pred_fun, we explicitly call ranger’s prediction method and extract its numeric predictions. The function must accept object, newdata, and type, and return a numeric vector or matrix. Explicitly invoking ranger’s method also makes this custom function available during parallel processing:
custom_predict <-function(object, newdata, type ='response') { ranger:::predict.ranger( # must explicitly invoke ranger for parallelization to workobject = object,data = newdata,type = type )$predictions}# To run the slow code yourself, uncomment and execute the following call.# ale_rf_diamonds_custom <- ALE(# rf_diamonds,# data = diamonds,# pred_fun = custom_predict# )ale_rf_diamonds_custom <- serialized_objects_site |>file.path("ale_rf_diamonds_custom.0.5.3.rds") |>url() |>readRDS()plot(ale_rf_diamonds_custom) |>print(ncol =2)
A custom function is especially useful for model classes whose prediction method has a non-standard signature or returns a structured object.
8 Retrieving ALE data and statistics
Plots are useful for interpretation, but the get() method retrieves the underlying data for further analysis. To demonstrate all of its principal outputs, we create a smaller selection of 1D and 2D effects, run ten bootstrap iterations, and retain the otherwise-discarded iteration-level data with output_boot_data = TRUE:
# To run the slow code yourself, uncomment and execute the following call.# ale_diamonds_with_boot_data <- ALE(# rf_diamonds,# data = diamonds,# x_cols = ~ carat + cut + clarity + carat:clarity + cut:table,# output_boot_data = TRUE,# boot_it = 10# )ale_diamonds_with_boot_data <- serialized_objects_site |>file.path("ale_rf_diamonds_with_boot_data.0.5.3.rds") |>url() |>readRDS()
By default, get() returns ALE data. A formula can limit the result to selected 1D and 2D terms:
# All confidence regions, or only statistically significant regionsget(ale_diamonds_with_boot_data, stats ="conf_regions")
! Note that confidence regions are not reliable with fewer than 100 bootstrap
iterations or p-values based on fewer than 100 random iterations.
ℹ There are 10 bootstrap iterations.
ℹ p-values are based on 100 iterations.
$d1
# A tibble: 16 × 12
term x start_x end_x x_span_pct n pct y start_y end_y trend
<chr> <chr> <dbl> <dbl> <dbl> <int> <dbl> <dbl> <dbl> <dbl> <dbl>
1 carat <NA> 0.2 0.71 10.6 16609 41.8 NA 1631. 2549. 0.501
2 carat <NA> 0.85 1.01 3.33 8804 22.2 NA 2818. 4145. 2.31
3 carat <NA> 1.07 5.01 81.9 14326 36.1 NA 4290. 7546. 0.230
4 cut Fair NA NA NA 1492 3.75 3357. NA NA NA
5 cut Good NA NA NA 4173 10.5 3430. NA NA NA
6 cut Very… NA NA NA 9714 24.4 3489. NA NA NA
7 cut Prem… NA NA NA 9657 24.3 3522. NA NA NA
8 cut Ideal NA NA NA 14703 37.0 3555. NA NA NA
9 clarity I1 NA NA NA 704 1.77 2342. NA NA NA
10 clarity SI2 NA NA NA 7916 19.9 4347. NA NA NA
11 clarity SI1 NA NA NA 9857 24.8 5241. NA NA NA
12 clarity VS2 NA NA NA 8227 20.7 5558. NA NA NA
13 clarity VS1 NA NA NA 6007 15.1 5259. NA NA NA
14 clarity VVS2 NA NA NA 3463 8.71 4696. NA NA NA
15 clarity VVS1 NA NA NA 2413 6.07 3547. NA NA NA
16 clarity IF NA NA NA 1152 2.90 2136. NA NA NA
# ℹ 1 more variable: aler_band <ord>
$d2
# A tibble: 47 × 8
term1 x1 term2 x2 aler_band n pct y
<chr> <chr> <chr> <chr> <ord> <int> <dbl> <dbl>
1 carat [0.2,0.71] clarity I1 above 95 0.239 4901.
2 carat (0.71,1.07] clarity I1 above 27 0.0679 4899.
3 carat (0.71,1.07] clarity I1 overlap 206 0.518 5001.
4 carat (1.07,5.01] clarity I1 overlap 376 0.946 5140.
5 carat [0.2,0.71] clarity SI2 overlap 1697 4.27 4088.
6 carat (0.71,1.07] clarity SI2 overlap 2894 7.28 5457.
7 carat (1.07,5.01] clarity SI2 overlap 1881 4.73 6631.
8 carat (1.07,5.01] clarity SI2 above 1444 3.63 7921.
9 carat [0.2,0.71] clarity SI1 overlap 3572 8.99 4023.
10 carat (0.71,1.07] clarity SI1 overlap 994 2.50 4635.
# ℹ 37 more rows
! Note that confidence regions are not reliable with fewer than 100 bootstrap
iterations or p-values based on fewer than 100 random iterations.
ℹ There are 10 bootstrap iterations.
ℹ p-values are based on 100 iterations.
$d1
# A tibble: 11 × 12
term x start_x end_x x_span_pct n pct y start_y end_y trend
<chr> <chr> <dbl> <dbl> <dbl> <int> <dbl> <dbl> <dbl> <dbl> <dbl>
1 carat <NA> 0.2 0.71 10.6 16609 41.8 NA 1631. 2549. 0.501
2 carat <NA> 0.85 1.01 3.33 8804 22.2 NA 2818. 4145. 2.31
3 carat <NA> 1.07 5.01 81.9 14326 36.1 NA 4290. 7546. 0.230
4 clarity I1 NA NA NA 704 1.77 2342. NA NA NA
5 clarity SI2 NA NA NA 7916 19.9 4347. NA NA NA
6 clarity SI1 NA NA NA 9857 24.8 5241. NA NA NA
7 clarity VS2 NA NA NA 8227 20.7 5558. NA NA NA
8 clarity VS1 NA NA NA 6007 15.1 5259. NA NA NA
9 clarity VVS2 NA NA NA 3463 8.71 4696. NA NA NA
10 clarity VVS1 NA NA NA 2413 6.07 3547. NA NA NA
11 clarity IF NA NA NA 1152 2.90 2136. NA NA NA
# ℹ 1 more variable: aler_band <ord>
$d2
# A tibble: 32 × 8
term1 x1 term2 x2 aler_band n pct y
<chr> <chr> <chr> <chr> <ord> <int> <dbl> <dbl>
1 carat [0.2,0.71] clarity I1 above 95 0.239 4901.
2 carat (0.71,1.07] clarity I1 above 27 0.0679 4899.
3 carat (0.71,1.07] clarity I1 overlap 206 0.518 5001.
4 carat (1.07,5.01] clarity I1 overlap 376 0.946 5140.
5 carat [0.2,0.71] clarity SI2 overlap 1697 4.27 4088.
6 carat (0.71,1.07] clarity SI2 overlap 2894 7.28 5457.
7 carat (1.07,5.01] clarity SI2 overlap 1881 4.73 6631.
8 carat (1.07,5.01] clarity SI2 above 1444 3.63 7921.
9 carat [0.2,0.71] clarity SI1 overlap 3572 8.99 4023.
10 carat (0.71,1.07] clarity SI1 overlap 994 2.50 4635.
# ℹ 22 more rows
These calls return data rather than plots: "estimate" provides a compact overview, named statistics return their estimates and inferential details, and the confidence-region options identify portions of an ALE curve or surface that exceed the relevant threshold. See help("get-ALE-method") for the precise return structures and the ALE statistics vignette before drawing inferential conclusions.
9 Where to go next
This introduction has covered the main ALE() workflow: default effects, selected 1D and 2D terms, bootstrapping, custom prediction functions, plotting, and data extraction. The other vignettes cover ALE-based statistical inference, different predictor datatypes, and small datasets and full-model bootstrapping. For exhaustive argument and selection details, see help(ALE), help(resolve_x_cols), and help("get-ALE-method").