BOSO is an R package to perform feature selection in linear regression models.
BOSO can be installed from CRAN repository:
install.packages("BOSO")
Note that it is necessary to have IBM ILOG CPLEX installed, with a version greater than 12.1, and the cplexAPI package, which is used to invoke CPLEX from R environment.
Furthermore, check if packages are installed. Check if bestsubset is installed. If not, source necessary functions from bestsubset.
if (!requireNamespace('cplexAPI', quietly = TRUE)) {
testthat::skip('Package cplexAPI not installed (required for this vignette)!\n
Install it from CRAN: https://cran.r-project.org/web/packages/cplexAPI/index.html')
stop('Package cplexAPI not installed (required for this vignette)!\n
Install it from CRAN: https://cran.r-project.org/web/packages/cplexAPI/index.html', call. = FALSE)
}
if (!requireNamespace('bestsubset', quietly = TRUE)) {
devtools::source_url("https://raw.githubusercontent.com/ryantibs/best-subset/master/bestsubset/R/fs.R")
devtools::source_url("https://raw.githubusercontent.com/ryantibs/best-subset/master/bestsubset/R/lasso.R")
devtools::source_url("https://raw.githubusercontent.com/ryantibs/best-subset/master/bestsubset/R/sim.R")
enlist <- function (...)
{
result <- list(...)
if ((nargs() == 1) & is.character(n <- result[[1]])) {
result <- as.list(seq(n))
names(result) <- n
for (i in n) result[[i]] <- get(i)
} else {
n <- sys.call()
n <- as.character(n)[-1]
if (!is.null(n2 <- names(result))) {
which <- n2 != ""
n[which] <- n2[which]
}
names(result) <- n
}
result
}
} else {require("bestsubset")}
## ℹ SHA-1 hash of file is "cc12499ddd2cc538f318f3ee14ad4a3911945a27"
## ℹ SHA-1 hash of file is "e7c4bf157a3c33b0a11179541be100c0037a4fea"
## ℹ SHA-1 hash of file is "531fcc6ad06024eae37afaa3e15d83e0ea3ece89"
The package is designed to run the BOSO algorithm in one single function, which comprises two steps: block strategy and final problem.
The block strategy is a pre-processing step in which the BOSO MILP (Mixed-Integer Liner Programming) is applied to random subproblems of small dimension, aiming to discard variables for the final problem. The usual block size is 10 variables. This process is done iteratively until convergence.
Final problem applies the BOSO MILP with all the remaining variables from the block strategy.
Figure 1 summarizes the BOSO algorithm. An example dataset with 7 features is split into training and validation sets. Green boxes represent optimal selected features for a Specific K value. For example, for K=2, the subset of features that minimizes the validation error is {X3, X6}. The problem of selecting the best subset of features of length K is formulated via mixed-integer quadratic programming (MIQP) and solved using IBM ILOG CPLEX This process is repeated for each K value until an information criterion, in this case the extended Bayesian Information Criterion (eBIC), is not further improved. Minimal eBIC is found in this example for K=2. The final model is derived from Ridge regression with only these two selected variables.
Function BOSO
has a number of parameters that can be changed:
On the other hand, there are some parameters that are identical to the ones found in the glmnet
function: lambda.min.ratio, lambda, intercept, standardize and dfmax.
Finally, some of the parameters are solver-specific:
Given a synthetic data set (see Methods section in the article), we evaluate the capacity of BOSO to extract the features that are related with the response variable. First o, we set the parameters for the generation of synthetic data:
# Set some overall simulation parameters
n = 100; # Size of training set
nval = n # Size of validation set
p = 10; # Number of variables
s = 5 # Number of nonzero coefficients
beta.type = 1 # Type of coefficiente structure
rho.vec = 0.35 # Variable autocorrelation level
snr.vec = exp(seq(log(0.05),log(6),length=10)) # Signal-to-noise ratios
nrep = 10 # Number of repetitions for a given setting
seed = 0 # Random number generator seed
Benchmarked methods are defined: Forward Stepwise, Lasso, Relaxed lasso and BOSO.
# Regression functions: lasso, forward stepwise and BOSO
reg.funs = list()
reg.funs[["Forward stepwise"]] = function(x,y) fs(x,y,intercept=FALSE, max=min(n,p))
reg.funs[["Lasso"]] = function(x,y) lasso(x,y,intercept=FALSE,nlam=50)
reg.funs[["Relaxed lasso"]] = function(x,y) lasso(x,y,intercept=FALSE,
nrelax=10,nlam=50)
reg.funs[["BOSO - AIC"]] <- function(x,y,xval,yval) BOSO(x,y,xval,yval,
IC = "AIC",
nlambda = 100,
Threads = 4,
timeLimit = 60,
intercept = F,
standardize = F,
verbose = 0,
warmstart=T,
seed = 123456)
reg.funs[["BOSO - BIC"]] <- function(x,y,xval,yval) BOSO(x,y,xval,yval,
IC = "BIC",
nlambda = 100,
Threads = 4,
timeLimit = 60,
intercept = F,
standardize = F,
verbose = 0,
warmstart=T,
seed = 123456)
reg.funs[["BOSO - eBIC"]] <- function(x,y,xval,yval) BOSO(x,y,xval,yval,
IC = "eBIC",
nlambda = 100,
Threads = 4,
timeLimit = 60,
intercept = F,
standardize = F,
verbose = 0,
warmstart=T,
seed = 123456)
We have adapted the sim.master
function from the bestsubset library function to assess the different methods.
sim.results <- list()
for (i in 1:length(snr.vec)) {
set.seed(i)
sim.results[[i]] <- sim.master(n = n, p = p, nval = nval,
rho=rho.vec, s=s, beta.type=beta.type,
snr = snr.vec[[i]],
reg.funs, nrep=nrep, seed=i,
verbose=T)
}
Finally, the next part of the code shows the comparison of BOSO-eBIC with some of the most widely used algorithms in the literature: Lasso, Relaxed lasso and Forward Stepwise.
method.nums = c(6,1,2,3)
method.names = c("BOSO - eBIC","Forward stepwise","Lasso","Relaxed lasso")
plot.BOSO <- list(
"F" = plot.from.sim(sim.list = sim.results, what="F", rel.to=NULL, tuning="val",
method.nums=method.nums, method.names=method.names),
"nonzeros" = plot.from.sim(sim.list = sim.results, what="nonzero", rel.to=NULL, tuning="val",
method.nums=method.nums, method.names=method.names),
"fpos" = plot.from.sim(sim.list = sim.results, what="fpos", rel.to=NULL, tuning="val",
method.nums=method.nums, method.names=method.names),
"fneg" = plot.from.sim(sim.list = sim.results, what="fneg", rel.to=NULL, tuning="val",
method.nums=method.nums, method.names=method.names))
## Coordinate system already present. Adding new coordinate system, which will
## replace the existing one.
## Coordinate system already present. Adding new coordinate system, which will
## replace the existing one.
## Coordinate system already present. Adding new coordinate system, which will
## replace the existing one.
## Coordinate system already present. Adding new coordinate system, which will
## replace the existing one.
ggarrange(plotlist = plot.BOSO, ncol = 2, nrow = 2, common.legend = T, legend = "bottom")
The following part of code is designed to compare the performance of the BOSO algorithm using different information criteria. The options available are AIC, BIC and eBIC.
method.nums = c(4, 5, 6)
method.names = c("BOSO - AIC","BOSO - BIC","BOSO - eBIC")
plot.BOSO <- list(
"F" = plot.from.sim(sim.list = sim.results, what="F", rel.to=NULL, tuning="val",
method.nums=method.nums, method.names=method.names),
"nonzeros" = plot.from.sim(sim.list = sim.results, what="nonzero", rel.to=NULL, tuning="val",
method.nums=method.nums, method.names=method.names),
"fpos" = plot.from.sim(sim.list = sim.results, what="fpos", rel.to=NULL, tuning="val",
method.nums=method.nums, method.names=method.names),
"fneg" = plot.from.sim(sim.list = sim.results, what="fneg", rel.to=NULL, tuning="val",
method.nums=method.nums, method.names=method.names))
## Coordinate system already present. Adding new coordinate system, which will
## replace the existing one.
## Coordinate system already present. Adding new coordinate system, which will
## replace the existing one.
## Coordinate system already present. Adding new coordinate system, which will
## replace the existing one.
## Coordinate system already present. Adding new coordinate system, which will
## replace the existing one.
ggarrange(plotlist = plot.BOSO, ncol = 2, nrow = 2, common.legend = T, legend = "bottom")
Luis V. Valcarcel, Edurne San José-Enériz, Xabier Cendoya, Ángel Rubio, Xabier Agirre, Felipe Prósper, Francisco J. Planes ‘BOSO: a novel feature selection algorithm for linear regression with high-dimensional data.’ submitted.
Hastie, Trevor, Robert Tibshirani, and Ryan Tibshirani. “Best Subset, Forward Stepwise or Lasso? Analysis and Recommendations Based on Extensive Comparisons.” Statistical Science 35.4 (2020): 579-592.
sessionInfo()
## R version 4.3.3 (2024-02-29)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 22.04.4 LTS
##
## Matrix products: default
## BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
## LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so; LAPACK version 3.10.0
##
## locale:
## [1] LC_CTYPE=C.UTF-8 LC_NUMERIC=C LC_TIME=C.UTF-8
## [4] LC_COLLATE=C LC_MONETARY=C.UTF-8 LC_MESSAGES=C.UTF-8
## [7] LC_PAPER=C.UTF-8 LC_NAME=C LC_ADDRESS=C
## [10] LC_TELEPHONE=C LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C
##
## time zone: Europe/Madrid
## tzcode source: system (glibc)
##
## attached base packages:
## [1] stats graphics grDevices utils datasets methods base
##
## other attached packages:
## [1] ggpubr_0.6.0 ggplot2_3.5.0 glmnet_4.1-8 Matrix_1.6-5
## [5] kableExtra_1.4.0 dplyr_1.1.4 BOSO_1.0.4 knitr_1.46
## [9] BiocStyle_2.30.0
##
## loaded via a namespace (and not attached):
## [1] tidyselect_1.2.1 viridisLite_0.4.2 farver_2.1.1
## [4] fastmap_1.1.1 promises_1.3.0 digest_0.6.35
## [7] mime_0.12 lifecycle_1.0.4 ellipsis_0.3.2
## [10] survival_3.5-8 magrittr_2.0.3 compiler_4.3.3
## [13] rlang_1.1.3 sass_0.4.9 tools_4.3.3
## [16] utf8_1.2.4 yaml_2.3.8 ggsignif_0.6.4
## [19] labeling_0.4.3 htmlwidgets_1.6.4 pkgbuild_1.4.4
## [22] curl_5.2.1 xml2_1.3.6 pkgload_1.3.4
## [25] abind_1.4-5 miniUI_0.1.1.1 withr_3.0.0
## [28] purrr_1.0.2 grid_4.3.3 fansi_1.0.6
## [31] urlchecker_1.0.1 profvis_0.3.8 xtable_1.8-4
## [34] colorspace_2.1-0 scales_1.3.0 iterators_1.0.14
## [37] tinytex_0.50 cli_3.6.2 rmarkdown_2.26
## [40] generics_0.1.3 remotes_2.5.0 rstudioapi_0.16.0
## [43] httr_1.4.7 sessioninfo_1.2.2 cachem_1.0.8
## [46] stringr_1.5.1 splines_4.3.3 BiocManager_1.30.22
## [49] vctrs_0.6.5 devtools_2.4.5 jsonlite_1.8.8
## [52] carData_3.0-5 bookdown_0.38 car_3.1-2
## [55] rstatix_0.7.2 systemfonts_1.0.6 foreach_1.5.2
## [58] tidyr_1.3.1 jquerylib_0.1.4 glue_1.7.0
## [61] codetools_0.2-20 cowplot_1.1.3 stringi_1.8.3
## [64] shape_1.4.6.1 gtable_0.3.4 later_1.3.2
## [67] munsell_0.5.1 tibble_3.2.1 pillar_1.9.0
## [70] htmltools_0.5.8.1 R6_2.5.1 evaluate_0.23
## [73] shiny_1.8.1.1 lattice_0.22-6 highr_0.10
## [76] backports_1.4.1 memoise_2.0.1 broom_1.0.5
## [79] httpuv_1.6.15 bslib_0.7.0 Rcpp_1.0.12
## [82] gridExtra_2.3 svglite_2.1.3 xfun_0.43
## [85] fs_1.6.3 usethis_2.2.3 pkgconfig_2.0.3