probcal 0.2.0
- Added a calibration-inference layer for binary and multiclass
predictions.
skce() computes the squared kernel calibration error of
Widmann et al. (2019) with three estimators: the unbiased U-statistic
("uq"), the unbiased linear-time estimator
("ul"), and the biased V-statistic ("biased").
It supports the binary/confidence reduction, the strong (canonical)
multiclass form via a matrix-valued kernel, and the classwise
one-vs-rest reduction. In the binary and confidence cases
mmce() equals
sqrt(skce(..., estimator = "biased")).
cal_test() performs a kernel calibration hypothesis
test (H0: the model is calibrated). The default
method = "bootstrap" uses the more powerful quadratic
estimator with a wild bootstrap (Widmann et al. 2019, Theorem G.2);
method = "asymptotic" uses the faster linear-estimator
normal test (Lemma 3). It returns an object of class
c("cal_test", "htest"). The test targets are binary,
"confidence", and the strong "canonical"
multiclass form; the classwise average is available only as a point
estimate from skce().
cal_ci() returns a percentile bootstrap confidence
interval for ece(), skce(),
mmce(), mce(), or ace(), as a
classed cal_ci object with a print()
method.
skce() and cal_test() accept
bandwidth = "median" for the median-heuristic kernel scale
(Widmann et al. 2019), recommended for the canonical multiclass form;
the fixed 0.2 remains the default.
ece() gains debiased (the debiased
squared-ECE estimator of Kumar, Liang & Ma 2019, Definition 5.2),
strategy (equal-width or equal-mass bins, Roelofs et
al. 2022), and norm ("l1" or
"l2"). The norm and debiased
choices are independent; debiasing is defined only for
norm = "l2". Defaults (norm = "l1",
debiased = FALSE, strategy = "width")
reproduce the previous numeric output exactly.
stratified_folds() (used by cal_cv()) now
scopes the optional fold-assignment seed with
withr::local_seed() instead of touching
.Random.seed directly; withr was added to
Imports. The fold assignments for a given seed are
unchanged.
inst/CITATION now reads the version from the package
metadata, and the package author is also declared as copyright holder in
Authors@R.
- The README documents the calibration-inference layer
(
skce(), cal_test(), cal_ci(),
and the debiased, strategy, and
norm arguments of ece()) with a worked
example.
probcal 0.1.0
probcal 0.0.0.9000
- Added multiclass calibration.
cal_temperature() and
cal_cv() accept a logit or probability matrix, and new
constructors cal_vector_scaling(),
cal_dirichlet(), and cal_ovr() cover vector
scaling, Dirichlet calibration, and one-vs-rest calibration.
ece(), mce(), ace(), and
reliability_diagram() accept a probability matrix with a
type argument for classwise or top-label confidence
evaluation.
- Added
mmce(), a binning-free Maximum Mean Calibration
Error metric for binary and multiclass predictions.
- Added
inst/CITATION so users can cite the package with
citation("probcal").
- Added applied workflow, calibrator selection, and numerical
validation vignettes.
print() and summary() respect
options(probcal.emoji = FALSE) to suppress the decorative
glyph in console output.
reliability_diagram() now reports ECE in the subtitle
by default and can use either count-scaled or fixed-size points.
- Added optional development validation tests against Python
netcal and R betacal.
- Initial development version with binary calibration methods,
calibration metrics, reliability diagrams, and out-of-fold calibration
support.