The goal of alphaN is to help the user set their significance level
as a function of the sample size. The function alphaN
allows users to set the significance level as function of the sample
size based on the evidence and the prior features they desire. The
function JABt and JABp converts test
statistics and \(p\)-values into sample
size dependent Bayes factors. JAB_plot plots the Bayes
factor as a function of the \(p\)-value, and alphaN_plot
plots the alpha level as a function of sample size for a given Bayes
factor.
Calculations are based on Wulff & Taylor
(2024). If you enjoy the package, please consider citing the paper
(see citation("alphaN")).
As of version 0.2.0, alphaN() can also calibrate the
alpha level to the effect-size and moment Bayes factors of Klauer, Meyer-Grant
& Kellen (2024), which center the alternative hypothesis on an
effect size of your choosing (method = "ES" and
method = "moment").
If you’re not an R user, you may also be interested in the associated Shiny app.
To install the latest release version from CRAN use:
install.packages("alphaN")You can install the development version of alphaN from GitHub with:
# install.packages("devtools")
devtools::install_github("jespernwulff/alphaN")Here is an example: We are planning to run a linear regression model
with 1000 observations. We thus set n = 1000. The default
BF is 1 meaning that we want to avoid Lindley’s paradox,
i.e., we just want the null and the alternative to be at least equally
likely when we reject the null.
library(alphaN)
alpha <- alphaN(n = 1000, BF = 1)
alpha
#> [1] 0.008582267Therefore, to obtain evidence of at least 1, we should set our alpha to 0.0086.
Raising BF asks for more evidence before you are allowed
to reject, which lowers alpha. The method argument selects
the prior behind Jeffreys’ approximate Bayes factor:
# Moderate (BF = 3) and strong (BF = 10) evidence at n = 1000
alphaN(n = 1000, BF = 3)
#> [1] 0.002549145
alphaN(n = 1000, BF = 10)
#> [1] 0.0006911392
# Balancing Type I and Type II error rates instead of the default prior
alphaN(n = 1000, BF = 3, method = "balanced")
#> [1] 0.024221The methods "ES" and "moment" (new in
0.2.0) answer the question: which alpha do I need so that a significant
result corresponds to a Bayes factor of at least BF against
an alternative centered on the effect size de I actually
care about?
# Moderate evidence, targeting a medium-sized effect (Cohen's d = 0.5)
alphaN(1000, BF = 3, method = "ES", de = 0.5)
#> [1] 0.002189564
alphaN(1000, BF = 3, method = "moment", de = 0.5)
#> [1] 0.0004913521Because the moment prior treats effects near zero as a priori implausible, the alpha it implies falls much faster with the sample size than under JAB:
ns <- c(100, 1000, 10000)
tab <- rbind(JAB = alphaN(ns, BF = 3),
ES = alphaN(ns, BF = 3, method = "ES"),
moment = alphaN(ns, BF = 3, method = "moment"))
colnames(tab) <- paste0("n = ", ns)
round(tab, 5)
#> n = 100 n = 1000 n = 10000
#> JAB 0.00910 0.00255 0.00073
#> ES 0.01185 0.00219 0.00058
#> moment 0.00788 0.00049 0.00002JAB() computes Jeffreys’ approximate Bayes factor for a
coefficient directly from a fitted lm() or
glm() object; JABt() and JABp()
do the same from a t-statistic or a p-value if that is all you have
(e.g., from a published paper):
set.seed(1)
d <- data.frame(x = rnorm(200), z = rnorm(200))
d$y <- 0.2 * d$x + rnorm(200)
m <- lm(y ~ x + z, data = d)
JAB(m, covariate = "x")
#> [1] 22.50664
# From summary statistics alone
JABt(n = 200, t = 2.8)
#> [1] 3.56385
JABp(n = 200, p = 0.005)
#> [1] 3.634824alphaN_plot() compares alpha as a function of sample
size across the four JAB-type priors:
alphaN_plot(BF = 3)
JAB_plot() shows how the Bayes factor maps onto the
p-value for a given sample size, marking the alpha levels needed for
evidence thresholds of 1, 3, and 10:
JAB_plot(n = 1000, BF = 3)