robust_gandh(): a Tukey g-and-h transformation
fitted by robust quantile least squares (rQLS), with separate skewness
and tail parameters, adaptive or explicit trimming cutoffs, and fit
diagnostics. It has the fitted-object interface of
robust_boxcox() and robust_yeojohnson(),
including new-data prediction, inverse transformation and residual
smearing. Direct inversion preserves extreme normal scores without
rounding cumulative probabilities. step_robust_gandh()
estimates the transformation within recipe training.
robust = FALSE gives the classical quantile least squares
fit (QLS) of all the observations, for comparison: the examples show, on
synthetic data, outliers that inflate the classical g and
h and are pulled in among the other values, but are trimmed
by the robust fit.
wavelet_denoise(): denoising of spectra by
thresholding their undecimated wavelet transform (MODWT), the
translation-invariant denoising of Coifman and Donoho (1995). Narrow
lines keep their height and width, unlike with a smoothing filter. The
noise of each level is estimated robustly in each spectrum, the
threshold is SureShrink (default) or universal, hard (default) or soft,
and the segments of a multi-spectrometer system are denoised separately.
step_wavelet_denoise() is its recipe step, with a tunable
level. In the single shots of forageShots,
whose lines are 2 to 3 channels wide and whose noise has the scale of
the lines, the defaults lower the error of the continuum by about 10%
for 4% of the height of the weak lines, while a Savitzky-Golay filter of
5 channels lowers the strong lines by 10%; the PLS calibrations of the
averaged spectra of forageLIBS hardly change.
rsimpls(): robust PLS regression by the RSIMPLS
algorithm of Hubert and Vanden Branden (2003), the SIMPLS algorithm
computed from a ROBPCA covariance matrix of the spectra and the
responses, followed by a robust regression on the scores. Outlying
spectra and wrong reference values have little influence on the model.
One or several responses; the response is block-scaled before ROBPCA so
that wrong reference values are detected whatever its units. The fit
gives the score, residual and orthogonal distances of each observation,
and the robust R² of the model. predict() gives the
predictions or scores of new spectra, with their score and orthogonal
distances, and plot_outlier_map() draws the regression
outlier map (vertical outliers, good and bad leverage points) or, with
map = "score", the score outlier map of the spectra.
step_rsimpls() is its recipe step, the robust counterpart
of recipes::step_pls(): it replaces the spectra by their
robust PLS scores, and optionally their score and orthogonal
distances.
The "rsimpls" engine of parsnip::pls()
fits rsimpls() models in tidymodels, with one or several
outcomes; the other arguments of rsimpls() are engine
arguments (set_engine("rsimpls", ncomp_max = , alpha = )).
multi_predict() gives the predictions of the models with
fewer components from the same fit, so tune::tune_grid()
fits one model per resample for a grid of num_comp.
predict() of an rsimpls() model gains
ncomp, for the models with 1 to ncomp_max
components, which the fit now keeps (models).
rpcr(): robust principal component regression (RPCR)
of Hubert and Verboven (2003), the robust PCA of the spectra
(robpca()) followed by the LTS regression of the response
on the scores, or the MCD regression of several responses. It has the
outputs, predict() method and outlier maps of
rsimpls().
robust_rmsecv(): the robust RMSECV of
rsimpls() or rpcr() models with 1 to
ncomp_max components, which leaves the outliers out of the
cross-validated error, and the robust component selection (RCS)
criterion of Engelen and Hubert (2005), to choose the number of
components. The folds are fitted in parallel when a
future::plan() is set.
The "lts" engine of
parsnip::linear_reg() fits the reweighted LTS regression
(lts_fit(), from robustbase::ltsReg()); after
step_robpca() in a workflow, it gives robust PCR in
tidymodels.
rsimca(): robust SIMCA classification of Vanden
Branden and Hubert (2005), a robust PCA model of each class and the
assignment to the closest class from the scaled score and orthogonal
distances, for whole spectra. predict() gives the classes,
or the distances to the classes and whether a spectrum is outlying for
all of them, and plot_coomans() draws the distances to two
classes against each other (Coomans plot), with the class and
classification boundaries. simca() is a new parsnip model
of SIMCA classification, with rsimca() as its
"rsimca" engine, to fit and tune it in tidymodels:
num_comp, and the rule of classification,
gamma and squared, with the new dials
parameters simca_gamma() and
simca_squared().
robust_da(): robust linear and quadratic
discriminant analysis of Hubert and Van Driessen (2004), from the MCD
estimates of the classes, for a few variables such as line intensities
or robust principal component scores. predict() gives the
classes or posterior probabilities. It is the "mcd" engine
of parsnip::discrim_linear() and
parsnip::discrim_quad().
select_wavelengths(): selection of the informative
wavelengths for PLS regression, by the variable importance in projection
(VIP), the selectivity ratio (SR), backward elimination on either, or
forward interval PLS (iPLS). With robust = TRUE, the
selection is made from an rsimpls() model, without the
regression outliers. The candidate intervals of iPLS are evaluated in
parallel when a future::plan() is set.
plot_wavelength_selection() shows the selected regions on
the mean spectrum with the importance of each variable.
plot_shots(): views of the laser shots and of their
rejection by reject_shots(), to judge the rejection before
averaging the shots: a map of the samples by the shot number, colored by
the robust z-scores in classes up to the cutoff, the rejected shots
outlined ("heatmap", the default; acquisition
orders the samples as measured, arrange = "extreme" puts
the most extreme first, and smooth gives running medians
along the acquisition, which show the drift of the shots during a
session); the robust z-scores of the shots with the cutoff
("criteria"); the shots of a sample with the rejected ones
in color ("spectra"); the intensity, shape and rejected
share along the shot number, which shows cleaning shots and the drift of
the ablation ("order"); and the rejected shots per sample
with the RSD of the samples before and after the rejection
("samples"). plot = FALSE returns the table of
each view.
forageShots: the single laser shots (8 per
measurement, in firing order) of 20 of the measurements of
forageLIBS, in the windows 380-430 nm and 760-780 nm, to
study the shot-to-shot variability.
step_select_wavelengths(): the selection as a recipe
step, repeated on every resample; num_terms,
num_intervals (new dials parameter
num_intervals()) and num_comp can be
tuned.
gap_derivative(): the gap-segment derivative of
Norris and Williams (1984), the first or second derivative computed from
the means of segments of segment channels separated by
gap channels, so that the smoothing and the distance of the
difference are set separately (the window of
savitzky_golay() sets both). As in
savitzky_golay(), every channel is kept and the detectors
of a multi-spectrometer system are filtered separately.
step_gap_derivative() is its recipe step;
derivative, gap and segment can
be tuned.
bin_spectra(): the means of groups of
width adjacent channels, within each detector, which reduce
the number of channels and the noise at the cost of resolution.
step_bin_spectra() is its recipe step; width
can be tuned. With step_spectral_norm(method = "area") and
step_gap_derivative(), it gives the preprocessing of the
winning team of the EMSLIBS 2019 classification contest (Vrábel et al.,
2020).
plasma_uncertainty(): the uncertainty of the
temperature of boltzmann() and
saha_boltzmann(), and of the temperature and composition of
cf_libs(), by Monte Carlo simulation (JCGM 101:2008). Each
draw repeats the fit with the intensities, transition probabilities,
electron density and, for a given temperature, the temperature perturbed
within their uncertainties; the result gives the standard uncertainty
and the interval of each quantity, and the draws. The uncertainties of
the intensities are the column intensity_se of the lines
and those of the transition probabilities their NIST accuracy grades (or
aki_rsd). The reduced chi-squared of the fit tells whether
these uncertainties explain the scatter of the points, or whether other
errors (spectral response, self-absorption) dominate. The fits now keep
their lines (and, for saha_boltzmann() and
cf_libs(), their other arguments in settings)
for this purpose.
electron_density_uncertainty(): the electron density
of electron_density() with its standard uncertainty and
interval, from the standard error of the width and the uncertainty of
the Stark parameter or of the H-alpha width law.
aki_uncertainty(): the relative standard
uncertainties of transition probabilities from the accuracy grades of
the NIST Atomic Spectra Database (one standard deviation; upper
bounds).
line_intensities() gives the standard error of each
intensity (intensity_se): the noise of the spectrum
propagated through the integration and the baseline, or the standard
error of the fitted area for the Voigt method.
correct_self_absorption() turns it into the standard error
of the corrected intensity.
response_calibration(): the relative spectral
response of the instrument from spectra of a calibration lamp (such as a
deuterium-halogen lamp) and its certified irradiance, with its
uncertainty. The certified values are interpolated by a monotone spline
of their logarithm, the ratio is smoothed by a local linear fit within
each detector, ranges such as the Balmer lines of deuterium can be left
out, and the relative standard uncertainty combines those of the
certificate and of the measurement (repeated lamp spectra, or the noise
of one). correct_response() corrects spectra on the same
axis, or the intensities of a table of lines (with their standard
errors, for plasma_uncertainty()), and
plot_response() draws the response.
peak_fit() and multipeak_fit() gain
fixed, to hold the Gaussian (or Lorentzian) width at a
given value, such as the instrumental width measured on a lamp line
combined with the Doppler width: the Lorentzian (Stark) width of a Voigt
fit is then much better determined. doppler_width() gives
the Doppler width of a line from the temperature and the mass of the
emitter.
lte_criteria(): the criteria of local thermodynamic
equilibrium of Cristoforetti et al. (2010) for transient and
inhomogeneous plasmas, with the McWhirter criterion: the relaxation time
to equilibrium and the diffusion length of the atoms meanwhile, from the
temperature, the electron density and the gap and oscillator strength of
the slowest (resonance) transition. With measurements at several delays,
the relative changes of the temperature and density during the
relaxation time give the temporal criterion; with the length over which
they change (such as the size of the plasma), the diffusion length gives
the spatial one.
multipeak_fit() gains constraints for multiplets:
lock_centers keeps the separations of the given wavelengths
(such as those of the NIST database) with one fitted shift,
share_widths gives the lines common widths, and
area_ratios fixes the ratios of their areas (for optically
thin lines, the relative intensities of libs_lines()). Each
constraint applies to all lines, or to groups of lines, so that an
interfering line can be left free. tidied lists the center,
widths and area of every line, with standard errors propagated from the
fitted parameters, and the common shifts.
calibration_curve() fits straight lines with errors
in the concentrations, such as the certified values of reference
materials: regression = "york" (York et al., 2004) with the
standard uncertainties of the concentration and signal of each standard
(concentration_se, signal_se), and
regression = "deming" with a constant ratio of their
variances (variance_ratio). Least squares, which assume
exact concentrations, bias the slope toward zero. The York fit
reproduces the Pearson-York benchmark; its standard errors are scaled by
the square root of the reduced chi-squared when it exceeds 1.
predict() and plot_calibration() use the
fitted line.
calibration_curve() gains the limits of ISO 11843-2
and IUPAC (lod_convention = "iso11843"): the critical value
and the limit of detection for the risks alpha and
beta (noncentral t), with the uncertainty of the intercept
and the number of replicates of a sample, and the limit of
quantification at 10 standard deviations. predict() marks
the samples above the critical value as detected, and flags
the concentrations outside the range of the standards
(outside_range).
applicability_domain() describes the calibration
spectra by a PCA, and its predict() flags new spectra
outside the domain, by their Hotelling T-squared or Q residual (see
q_residuals()), so that extrapolated predictions of any
model can be checked.
curve_of_growth(): self-absorption by the curve of
growth of a homogeneous plasma in LTE (Gornushkin et al., 1999), the
model of the C-sigma graphs of Aragón and Aguilera (2014): from the
intensities of the lines of one or several species, their Stark widths
and the temperature, it fits a common instrument factor and one column
density per species, and returns the optical depth at the center, the
self-absorption coefficient and the optically thin intensity of each
line, ready for boltzmann() and cf_libs(). The
line profiles are Voigt profiles of the Stark and Doppler widths.
curve_of_growth_model() computes the intensities for given
optical depths, and plot_curve_of_growth() draws the
reduced intensities against the optical depth.
Calibration transfer: ds() (direct standardization)
and sst() (spectral space transformation, Du et al., 2011)
transfer spectra of a secondary instrument to the primary, from transfer
samples measured on both, with low-rank transfer matrices;
slope_bias() corrects the predictions of a model on the
secondary instrument instead, with tests of the bias and slope.
pds() now returns a specproc_transfer object
(with the same transfer_matrix and intercept),
and predict() transfers spectra with any of the
three.
control_chart() and
plot_control_chart(): control charts of a quantity that
should stay constant, such as an internal standard line or a check
sample, to detect drifts of the instrument: Shewhart chart with the
Western Electric rules, EWMA and CUSUM, with the center and standard
deviation (mean moving range) from a baseline.
line_finder() builds a Boltzmann plot interactively:
clicking the dot of a line marker adds the line to the plot of its
species (or removes it), Add shown lines adds the lines
shown, and clicking a point of the plot leaves it out of the fit. The
intensities are measured on the displayed spectrum with
line_intensities() and the temperature is fitted with
boltzmann(); a table lists the upper-level energy,
signal-to-noise ratio and residual of each line. Use T
ranks the candidate lines at the fitted temperature, and
Done closes the app and returns the
boltzmann() fit to R.
Molecular bands: molecular_bands() lists the band
heads of the electronic systems of diatomic molecules that emit in
laser-induced plasmas (CN, C2, CH, OH, NH, N2, N2+, NO, CO, AlO, CaF,
CaCl, SrF, SrCl, MgO, TiO, BO, SiO), with their origins, the direction
in which they are degraded, Franck-Condon factors (Morse wavefunctions)
and relative intensities at a vibrational temperature. They are computed
from the constants of Huber and Herzberg (1979), included in the
package, and agree with measured heads to about 0.05 nm for the
well-studied systems. plot_lines() gains bands
to mark them on a spectrum (and lines may be
NULL), and line_finder() overlays the bands of
the selected molecules. molecular_spectrum() simulates the
emission spectrum of a molecule in LTE from a line list of the ExoMol
database (the MoLLIST line lists by default), downloaded on demand and
convolved with the instrument profile, and
exomol_datasets() lists the line lists of a
molecule.
read_vald() and read_kurucz() read the
Stark widths of a line list saved from the Vienna Atomic Line Database
(VALD3, free with a registration; “Extract All”, “Extract Element” or
“Extract Stellar”, short or long format, in any wavelength unit) or from
the line lists of Kurucz (gfall*.dat or the list of one
ion), as a table in the format of starkb_lines(). Both
give, for each line, the quadratic Stark damping constant at 10 000 K,
converted to a full width at half maximum and scaled with temperature as
T^(1/6). These widths are approximate, but cover many lines that STARK-B
does not have. The VAMDC service of VALD could not be queried (it fails
on every query, as of October 2026), so VALD is read from saved
extracts.
stark_width(), electron_density() and
electron_density_uncertainty() gain species: a
table of several species (as read by read_vald() and
read_kurucz(), or combined from several sources) needs the
species of the line, as the line is otherwise matched by its wavelength
alone and could be a line of another species.
Persistent cache of downloaded data: with
options(specProc.cache = TRUE) (the user cache directory of
tools::R_user_dir()) or a directory, the data of
nist_lines(), nist_ionization_energy(),
nist_levels() (and so libs_lines() and
line_finder()), starkb_lines() and the ExoMol
line lists of molecular_spectrum() are kept on disk and
reused in later sessions, without an internet connection. Entries older
than specProc.cache_max_age days (default 180) are
downloaded again, and an older copy is used, with a warning, when the
database cannot be reached. download_cache() lists the
cached files and clear_download_cache() removes them. The
cache is off by default.
tidy() and glance() methods for peak
fits, calibration_curve(),
wavelength_calibration(), slope_bias(),
boltzmann(), saha_boltzmann(),
cf_libs(), plasma_uncertainty() and
curve_of_growth(): the estimates with their standard errors
and confidence intervals, and a one-row summary of the fit, in the
conventions of broom (see ?"specproc-tidy"), to bind the
results of several spectra or samples. The generics are re-exported, and
generics (already a dependency of dplyr) moves from Suggests to Imports.
The fit column of peak_fit() and
multipeak_fit() holds nls fits of the new
class specproc_peak_fit.
tidy() and glance() methods for the
chemometric models (see ?"specproc-tidy-models"): robust
PCA (robpca(), rospca(),
macropca(), cellpca(); eigenvalues, loadings
or scores, as for prcomp in broom), rsimpls()
and rpcr() (coefficients, components, loadings or scores),
lts_fit(), rsimca(), robust_da(),
opls(), som(),
select_wavelengths(), nas() and
applicability_domain(), with the outlier counts and the
figures of merit of each model in glance(). Through
parsnip, they also summarize the fits and workflows of the
"rsimpls", "lts", "rsimca" and
"mcd" engines. lts_fit() now keeps the
standard errors (std_error), residual degrees of freedom
and robust R² of the reweighted fit.
medcouple(): the medcouple of Brys, Hubert and
Struyf (2004), a robust measure of skewness with a breakdown value of
25%, of a numeric vector. sapply(spectra, medcouple) gives
one per wavelength of a set of spectra (channels that saturate the
detector have a strongly negative medcouple). It was already used by
adjusted_boxplot(), iqr_outliers(skew = TRUE),
medcouple_weight() and
summary_stats(robust = TRUE).
plot_medcouple(): the medcouple of the intensities
at each wavelength, as bars along the wavelength axis, or with
group as a heatmap with one row per group of spectra (such
as the shots of each sample). The color scale is fixed from -1 to 1: two
colors through white at 0 (blue and red by default), or any gradient
(color = grDevices::hcl.colors(9, "Spectral")). Interactive
versions with plotly.
robust_boxcox() and robust_yeojohnson()
replace robust_bcyj(), with the recipe steps
step_robust_boxcox() and
step_robust_yeojohnson() (the robust counterparts of
recipes::step_BoxCox() and
recipes::step_YeoJohnson()). Each fits one transformation,
so that the lambda of all the variables are on the same
scale. They take a vector, matrix or data frame, have a
standardize argument, and return a fitted object that
predict() applies to new data, or to any subset of its
variables. robust_boxcox() warns about the variables with
zero or negative values, which it leaves unchanged.
robust = FALSE gives the classical fit by maximum
likelihood (for Box-Cox, the lambda of
MASS::boxcox()), for comparison: the examples show, on
synthetic data, outliers that pull the classical lambda,
skew the transformed regular values and hide among them, but not with
the robust fit.
predict(inverse = TRUE) transforms values back to
the original scale after robust_boxcox() or
robust_yeojohnson(), such as the predictions of a
calibration fitted to the transformed contents: as the median
(estimate = "median", default) or, with the residuals of
the model, as the mean (estimate = "mean", the smearing
estimate of Duan, 1983). Values beyond the range of the transformation
become NA, with a warning.
title, xlab, ylab and
base_size on every plot (added where they were missing, as
in plot_calibration(), plot_boltzmann(),
plot_lines() and the outlier maps);point_size and point_alpha on the plots of
points: plot_calibration(), whose replicate measurements
overlap, plot_boltzmann(),
plot_curve_of_growth(), plot_control_chart(),
plot_wavelength_calibration(),
plot_outlier_map(), plot_influence(),
plot_coomans(), plot_embedding(),
plot_outliers(), adjusted_boxplot(),
generalized_boxplot(), plot_shots(),
plot_som() and plot_fit();wavelength = c(min, max) on the plots along the
wavelength axis, to draw a window without subsetting the spectra first:
plot_lines() (its markers are then scaled within the
window, and the lines and bands outside it left out),
plot_loadings() and plot_contributions() (the
largest peaks within the window are labeled),
plot_cell_map(), plot_response(),
plot_wavelength_selection() and the prototypes of
plot_som(), as in plot_spectra() and
plot_shots();linewidth for the spectra and curves drawn along the
wavelength axis, with spectrum_color,
marker_linewidth and marker_alpha in
plot_lines(), whose intensity axis is now titled “Intensity
(arb. units)”, as in plot_spectra();interactive for the plots with a plotly version, and
color_by for what colors the points.plot_outlier_map(),
plot_influence(), plot_coomans(),
plot_outliers() and plot_embedding() (and its
biplot loadings), the peaks of plot_loadings(),
plot_contributions(), plot_cell_map() and
plot_fit(), the outliers of adjusted_boxplot()
and generalized_boxplot() (where many crowd together, some
are left unlabeled), the rejected shots of plot_shots(),
the spectra of plot_spectra(label_spectra = TRUE), and the
lines of plot_wavelength_calibration(), which are now all
labeled.robpca() and rospca() are faster on wide
data (up to about 4 times on full LIBS spectra): the reduction to the
subspace spanned by the observations is now computed in C++ from the
smaller cross-product matrix instead of an SVD. The results are
unchanged up to numerical precision.plot_spectra() has new arguments: panel, a
grouping variable that splits the spectra into panels, arranged one
above the other or side by side (layout = "vertical" or
"horizontal"), with the offset restarting in
each panel; grid, to draw grid lines; and
color_as = "fill", to show the colors of
colvar (or id) as the fill of the area under
each spectrum instead of the line color. Spectra stacked with a vertical
offset are then filled with opaque colors, the front ones hiding the
back ones, as in a waterfall plot.plot_spectra() makes figures ready for publication:
legend shows the legend of colvar (a color
bar) or id, on any side or inside the plot, titled by
legend_title (such as "K (%)");xlab, ylab and title set the
axis titles and the plot title (bold, split into a title and a subtitle
when long), and the intensities are written in plain notation (“100,000”
rather than “1e+05”);lines, a line list from libs_lines(),
marks the emission lines with dashed lines, labeled with their species
and wavelength above the plot (close lines share a label);palette sets the colors: a viridis palette
("viridis", the default, "magma",
"cividis", …) or any colors;base_size and linewidth size the text and
the lines for the printed figure (such as 8 points and 0.3 for a journal
column);wavelength = c(min, max) limits the wavelength range,
as in peak_fit() and nist_lines(); a list of
ranges draws them side by side, each with its own wavelength axis, such
as
list("Ca II H and K" = c(392, 398), "K I" = c(765, 771)),
with the groups of panel in rows;
scales = "free_y" fits the intensity axis to each
panel;transform applies a preprocessing function of specProc
(such as snv, baseline_als() or
savitzky_golay()), a function of your own, or a list of
them in order, to the whole spectra before they are drawn, keeping the
id, colvar and panel
columns;panel_tags tags the panels (a), (b), (c), …;label_spectra labels each spectrum at its right end,
with its id or value of colvar;summary = "mean" or "median" draws the
mean spectrum of each group (panel and id) with a band of
one standard deviation, or the median with the quartiles;rasterize draws the spectra as an image within an
otherwise vector plot, for small PDF files of many long spectra (needs
the ragg package).colvar now go from dark purple to yellow-green (viridis),
instead of from blue to red, readable by colorblind readers and in
grayscale; use palette = c("blue", "red") for the former
colors. The axis lines are slightly thinner.adjusted_boxplot() and
generalized_boxplot() make figures ready for publication:
scales = "free_y", the default), so that variables of
different scales stay readable; scales = "fixed" keeps a
common axis;group draws the boxes of groups (sites, treatments, …)
side by side, and id names the observations: the
outliers table gives the row and
id of every outlying value, and label_outliers
labels them;show_n), and show_mean marks the means;points = "all" draws every observation
("outliers" by default, spread sideways so that equal
values do not hide each other, or "none");caption); the help pages give a
figure legend;xlab, ylab, title (bold),
base_size, horizontal, log (a
logarithmic axis) and fill (one color, or one per group or
variable);stats table gives n, the fences, the
notch limits, the mean and n_outliers of each
variable (and group);annotate writes statistics with each box: the shape
measured by the method (medcouple, or g and h), the number of flagged
values with the number expected in clean data, the median with its
distribution-free 95% confidence interval, the IQR and the robust
coefficient of variation; it draws the fences and the biweight location,
gives the p-value of a Kruskal-Wallis test between the groups, and the
number of missing values. The stats table gains these
values (n_missing, median_lower,
median_upper, iqr, rcv,
biweight, expected_outliers), and a
tests table gives the tests.plot_outliers() makes figures ready for publication:
readable variable names, a “Robust z-score” axis, outliers in red among
grey samples (“Regular” and “Outlier”), dashed univariate limits (z =
-2.5 and 2.5), and a caption saying how the outliers are flagged and how
to read the plot. type = "distance" draws the robust
distance of each sample with the cutoffs. id names the
samples, in the table and with label_outliers;
color_by ("outlier", "distance"
or "both"), plot = FALSE (the table: row, id,
distance, cutoff, outlier, weight and robust z-scores),
title, ylab and base_size. Rows
with missing values are left out, with a message, instead of
stopping.plot_fit() gives, above each panel, the center, FWHM
and area of each peak with their standard errors (rounded to two
significant digits of them), and R^2 and RMSE. The FWHM of a Voigt
profile combines its widths (Olivero and Longbothum, 1977), with a
standard error from the covariance of the estimates.
show_fwhm draws the FWHM at half maximum; the baseline is
dotted, the peaks of a multi-peak fit are numbered, the residuals have
dashed limits at plus or minus twice the RMSE, and a caption names the
profiles. The data are small open circles, with a legend;
xlab, ylab, base_size,
ncol (several spectra: a pair of panels each) and
caption are added.plot_embedding() makes figures ready for publication:
plain axis numbers (“-100,000” rather than “-1e+05”);
base_size, which scales the text, points, lines and labels;
a caption saying what the ellipses and limits are;
legend_title; a point shape per group
(shapes), for grayscale printing; palette;
aspect_ratio = "equal"; legend (position) and
panel = "white". The labels of the flagged samples
alternate above and below the points, and the short loadings of a biplot
are drawn without labels, which overlapped near the origin.hotelling_t2() and plot_embedding() give
the limit of a new sample, method = "new" (the F limit
times (n + 1) / n). The docs no longer call the F limit the limit of a
new sample.plot_spectra() and
plot_fit() give the units in parentheses, as in the other
plots: “Wavelength (nm)” and “Intensity (arb. units)”.wavelet_features() keeps the wavelengths: when the
names of the spectra are wavelengths, the attribute
"wavelength" of the coefficients gives the wavelength at
the center of the window of channels of each one (the centroid of its
weights for the approximation, their center of energy for the details,
which are not centered on the block of 2^level channels:
about 37 channels later for "la8" at level 4).
tidy() of step_wavelet() gains the
wavelength of the kept coefficients, to trace them back to
the spectrum. A named vector now keeps its names.zscore(), iqr_outliers()
and directional_outlyingness() start with synthetic data
whose outliers are known, drawn as a density plot: a cluster of outliers
that hides itself from the classical z-score (masking), and a
right-skewed sample with low and high outliers, on which the fences that
account for the skewness and the directional outlyingness flag the
outliers and no regular value of the long tail.peakfit(), multipeakfit(),
plotfit(), plotSpec(),
summaryStats(), tukeyGH(),
boltzmann_plot() and the others listed in
?"specProc-defunct"), deprecated until 0.8.3, are now
defunct: they stop with an error that names the new function, and will
be removed in a later release.max.iter becomes max_iter in
baseline_als(), baseline_arpls(),
baseline_lsp(), osc(), peak_fit()
and multipeak_fit();drop.na becomes drop_na (as in
plot_spectra()) in biweight_location(),
biweight_midvariance(), biweight_scale(),
center(), iqr_outliers(),
medcouple_weight(), minmax(),
msc(), normalize(),
pareto_scale(), poisson_scale(),
quantile_weight(), rousseeuw_croux(),
snv(), summary_stats(), umad()
and zscore();drop.offset becomes drop_offset in
msc() and step_msc(), and
pls.ncomp becomes pls_ncomp in
osc();.group_by becomes group_by in
average(), log.p becomes log_p in
tukey_gh(), and launch.browser becomes
launch_browser in line_finder();directional_outlyingness(),
cutoff.quantile, rmZeroes,
maxRatio and precScale become
cutoff_quantile, rm_zeroes,
max_ratio and prec_scale;peak_fit() and multipeak_fit(),
wavelength = c(min, max) replaces wlgth.min
and wlgth.max, as the wavelength range of
nist_lines() and libs_lines(). Calls that gave
the arguments after A by position must name them.ncomp in all the functions,
as in rsimpls(), rpcr(),
rsimca(), the PLS and OSC functions and the pls package,
and the largest number of components considered is
ncomp_max. The former names still work, with a deprecation
warning, and the new names keep the positions of the former ones:
k becomes ncomp in robpca(),
rospca(), macropca(), cellpca(),
q_residuals(), dmodx(),
contributions(), hotelling_t2() and
plot_embedding();kmax becomes ncomp_max in
robpca(), macropca() and
cellpca(), and so in the options of
step_robpca(), step_macropca() and
step_cellpca().robpca(), rospca(),
macropca() and cellpca() keep their number of
components in ncomp instead of k, and the
results of q_residuals(), dmodx() and
contributions() in the attribute ncomp: code
that reads fit$k now gets NULL. The
k of iqr_outliers(), the fence factor, keeps
its name.rospca() chooses its number of components when
ncomp is not given, as robpca(): the smallest
number that explains var_explained (new, 0.8) of the
variance of the least outlying observations, with at most
ncomp_max (new, 10). ncomp was 2 by default;
rospca(x, ncomp = 2) gives the former result.generalized_boxplot() flags fewer observations: its
default detection rate alpha is now about 0.7%
(2 * pnorm(-4 * qnorm(0.75))), that of Tukey’s boxplot for
normal data, as in the authors’ implementation (the Stata command
robbox), instead of 5%. With 5%, about 18 of the 368
forageLIBS samples were flagged per mineral even in clean
data; use alpha = 0.05 for the former fences (see also the
bug fixes).adjusted_boxplot() and
generalized_boxplot() draw one panel per variable, boxes in
light grey (the colors of each variable added nothing, and depended on
whether ggsci was installed) and horizontal labels.
scales = "fixed" and fill give the former
layout and colors.xlabels.angle, box.width,
notchwidth and staplewidth of
adjusted_boxplot() and generalized_boxplot()
are renamed x_labels_angle, box_width,
notch_width and staple_width, and
xlabels.hjust and xlabels.vjust are deprecated
(the justification follows the angle). The former names still work, with
a warning.outliers table of adjusted_boxplot()
has the columns row and out (the tail), as
that of generalized_boxplot().reject_shots() (and step_reject_shots())
rejects far fewer regular shots. With the few shots of a sample, the MAD
of the sample is unstable: shots that differed by little from very
similar shots got large z-scores, and the correlations near 1, bounded
and skewed, made small differences look extreme. On the 2944 shots of
the forage data set, 235 shots were rejected, many with a correlation
above 0.99 with the median spectrum of their sample; in simulations,
2.5% to 5% of regular shots were. The z-scores of the correlation are
now those of Fisher’s z, the intensity and the distance are on a log
scale, and the scale of a sample is its MAD but not less than the pooled
MAD of all the samples (scale = "floor", the default;
"sample" and "pooled" are the others): 10
shots of the forage data set are rejected (weak or strong plasmas of
other shapes), and 0.1% to 0.5% of regular shots in simulations, while
the shots of a plasma weakened by a quarter or more are still detected.
The result also gives the shot number .shot (new argument
shot), the raw criteria .intensity (relative
to the median shot of the sample), .correlation and
.distance, and its settings.plot_fit() are
renamed point_size, point_color,
linewidth and fit_color (from
pt.size, pt.colour, line.size and
line.colour); pt.shape, pt.fill,
linetype and the resid.* arguments are
deprecated and ignored.plot_outliers() flags the samples beyond the adaptive
cutoff of Filzmoser, Garrett and Reimann (2005), as
aq.plot() of the mvoutlier package, instead of the fixed
97.5% chi-squared quantile, which flags about 2.5% of clean data (the
adaptive part had no effect: the cutoff was the smaller of the two). In
forageLIBS, 29 of the 368 samples are flagged on Ca, Mg, P
and K instead of 38. cutoff = "quantile" gives the former
flags. show.outlier and show.mahal are
deprecated in favor of color_by and
plot = FALSE, and the table gains the columns
row, cutoff and weight.plot_spectra(.interactive) becomes
interactive;plot_outlier_map(colour_by) and
plot_influence(colour_by) become color_by, and
plot_som(colour) becomes color_by;plot_embedding(colour, size, alpha) become
color_by, point_size and
point_alpha. A number in point_size is the
size of the points; the default, and a number in the former
size, scale with base_size.plot_outlier_map(), plot_influence() and
plot_coomans() can still be given in ...
(size, alpha), as other arguments of
ggplot2::geom_point().robpca() now have a fixed sign, their
largest element positive, as those of macropca() and
cellpca(). The signs of loadings and scores may differ from
those of earlier versions.robust_bcyj() and step_robust_bcyj() are
deprecated in favor of robust_boxcox(),
robust_yeojohnson() and their recipe steps (see New
features). They still work, with a deprecation warning, and the recipes
trained with step_robust_bcyj() can still be baked. Their
default, type = "bestObj", chose Box-Cox or Yeo-Johnson for
each variable by a difference in fit that is often negligible (each for
about half of the forageLIBS channels), and mixed two scales of
lambda. The defunct robustBCYJ() now names the
new functions.biweight_location(), biweight_scale(),
biweight_midvariance(), umad(),
iqr_outliers(), medcouple_weight(),
quantile_weight(), zscore(),
minmax() and directional_outlyingness(), as
medcouple()) stop when x is a matrix, instead
of pooling all its values into one result (one location of all the
channels of a set of spectra, for example). The message shows how to get
one result per column, with sapply(x, f) for a data frame
or apply(x, 2, f) for a matrix. Data frames already gave an
error.directional_outlyingness() flags the observations whose
score exceeds the (1 + cutoff_quantile) / 2 quantile of the
standard normal distribution (2.81 by default), about 0.5% of the
observations of a normal distribution. The cutoff came from a robust
normal fit to log(0.1 + score): on one variable it lay near
5 to 6 scales from the median, so that no observation of normal data was
flagged, nor, in a right-skewed sample, a high outlier at 6 scales or
low outliers below the distribution. More observations are flagged.
cutoff_quantile must be strictly between 0 and 1. The
scales of the two half samples now follow equation (2) of Rousseeuw,
Raymaekers and Hubert (2018) exactly, with the consistency constant
alpha of Huber’s rho function instead of a rounded one: the scores are
0.4% higher, and the help page gives the formula.nist_lines() accepts several species, such as
c("Ca I", "Ca II"), as in the plasma diagnostics vignette,
and returns their lines in one table; it accepted only one (“‘species’
must be a single character string”).starkb_lines(), nist_lines(),
nist_levels() and nist_ionization_energy() try
a download again, up to three times, when the server is busy (HTTP 5xx
errors, such as 504 Gateway Timeout), and report the HTTP status when
the download fails, instead of a less telling message such as
“downloaded length 0 != reported length 160”.timeout of the functions that download data
(nist_lines(), nist_levels(),
nist_ionization_energy(), libs_lines(),
starkb_lines(), molecular_spectrum(),
exomol_datasets()) is the time after which a download is
given up, even when the option timeout is larger: a server
that did not answer was waited for as long as the option.generalized_boxplot() now follows Bruffaerts et
al. (2014) and the authors’ implementation: the transformed data are
standardized by their interquartile range divided by
diff(qnorm(c(0.25, 0.75))) = 1.349 (was 1.3426); a negative
tail parameter h (tails lighter than normal) is kept
instead of being set to 0, which widened the fences of such variables
(in forageLIBS, at alpha = 0.7%, the upper fence of Ca was
1.25 instead of 1.19); and the fences are never inside the box.notch = TRUE failed in adjusted_boxplot()
and generalized_boxplot() (“object ‘notchlower’ not
found”); the notches span the median plus or minus 1.58 IQR / sqrt(n)
(McGill et al., 1978).plot_spectra() drew straight lines across the gaps
between detectors (in forageLIBS, 781.5 to 789.2 nm and
800.7 to 813.9 nm), and joined the channels of overlapping detectors in
a zigzag (near 766 nm, on the K I 766.49 nm line). Each detector is now
drawn as its own line.robpca() and rospca() failed, or flagged
most observations as orthogonal outliers, when ncomp
equaled the rank of the data (for example ncomp = 5 with 5
variables, ncomp = n - 1 with more variables than
observations, or an ncomp chosen by
var_explained for data with few variables): the orthogonal
distances to a subspace that is the whole space are rounding errors, and
their cut-off was zero. As in Hubert, Rousseeuw and Vanden Branden
(2005), ROBPCA then reduces to the reweighted MCD of the data: the
reweighting on the orthogonal distances is skipped, and H1 is the subset
H0 of the least outlying observations (as in the rospca package). The
orthogonal distances and their cut-off are now zero, with no orthogonal
outliers, in robpca(), rospca() and
macropca(), and for new observations of
predict() that lie in the subspace.robpca(), rospca() and
macropca() no longer depend on the units of the data: their
numerical tolerances (singular subsets of FAST-MCD, zero scales of the
outlyingness, the cut-off of the orthogonal distances, the rank and the
C-steps of macropca(), and the constant variables of the
detection of deviating cells) are relative to the scale of the data
instead of absolute. Robust PCA failed on data with a small scale (“No
non-singular subset found” for values around 1e-8, or a zero cut-off of
the orthogonal distances when they were all below 1.5e-8), and
macropca() gave different results. With the same seed,
robpca(x * c) now gives the same subsets and outlier types
as robpca(x) for any c > 0. The final step
of macropca() now always uses the deterministic MCD when it
succeeds: cellWise compares its log-determinant with the determinant of
the C-steps, a comparison that depends on the units, and keeps it in the
usual case.quantile_weight() says what
p and q are (the probabilities that choose the
quantiles compared in each half of the distribution) and gives the right
breakdown value and normal values: the defaults have a breakdown value
of 6.25%, not 12.5% (which needs p = 0.25 and
q = 0.75), and give LQW = RQW = 0.249 at the normal
distribution, not 0.2 (the value of the medcouple weights). The
breakdown value of medcouple_weight() is 12.5%, not 25%:
the medcouple resists 25% of outliers, but is computed on each half of
the observations.cellpca(): robust PCA by casewise and cellwise
weighting (cellPCA, Centofanti, Hubert and Rousseeuw). It handles
outlying observations, outlying cells and missing values by minimizing a
single objective, with weights between 0 and 1 for every cell and
observation. The iteratively reweighted least squares and the
predictions of new data are computed in C++; the results agree with the
reference code of the authors to numerical precision.
predict(), plot_outlier_map() (enhanced
outlier map), plot_cell_map() and
flagged_regions() accept its fits, and
step_cellpca() is its recipe step.macropca() and cellpca() now follows the DDC
algorithm of the cellWise package: 1-step M estimators for the
standardization and the residual scales, the robust
Gnanadesikan-Kettenring correlations for the neighbors (wrapped
correlations, found exactly, for more than 750 variables), robust slopes
refined by least squares, each cell included in its own prediction, and
the standalone variables. It flags the same cells as cellWise, and more
of the outlying cells with far fewer false flags than before.macropca() now follows the MacroPCA algorithm of
Hubert, Rousseeuw and Van den Bossche (2019), built on ROBPCA, as
implemented in cellWise, whose results it reproduces (with
scale = FALSE): projection pursuit on the data where only
the observations with the fewest flagged cells are cell-imputed,
iterative subspace estimation on a fixed set of observations, one
reweighting step on the orthogonal distances, concentration steps and
the deterministic MCD within the subspace, and distances from the data
with only the missing cells imputed. New data are analyzed as by
MacroPCApredict (with the DDC model of the fit). Its results change.
tol is now the largest angle between successive subspaces
as a fraction of a right angle (default 0.005, as in the paper), the
cut-offs of the score and orthogonal distances are at the 99% level as
in the paper, the residual scales are 1-step M scales, and the fit
returns the observations set aside by DDC
(flagged_rows).baseline_lsp() uses the improved modified polynomial
fit (IModPoly) of Zhao et al. (2007) by default
(method = "imodpoly"). The modified polynomial fit of
Lieber and Mahadevan-Jansen (2003), still available with
method = "modpoly", is pulled up by strong lines, and its
baseline could then fall far below the background at the ends of the
spectrum (for example above 290 nm in the 240-300 nm window of the
forageLIBS spectra). The default max.iter is
now 100.ggsci, prospectr and XICOR
move from Imports to Suggests. generalized_boxplot() uses
the default ggplot2 palette when ggsci is not installed,
and correlation(method = "chatterjee") asks for
XICOR. y_gradient_glsw() computes its
Savitzky-Golay derivatives with the package’s own filter (same
results).mt is no longer suggested.step_*() function checks that
recipes is installed.step_*()
functions, predict() methods, reject_shots(),
pds(), robust_bcyj(), umad(), the
wavelength calibration helpers and the tuning parameters).rospca() example runs on a spectral window, and the
plot_loadings() example that queries the NIST database is
run with \donttest and try().q_residuals(), and the Q limits of
plot_influence()) was wrong when its exponent h0 is close
to zero or negative, which happens for spectra (a few large eigenvalues
of the components left out, then a long tail of small ones): it fell
below the mean of Q, so that nearly every sample was flagged. For
example, with 3 components on the forageLIBS spectra, the
99% limit was 1.8e8 instead of 2.0e9, and 355 of the 368 spectra were
flagged. The sign of h0 is now kept, as its derivation requires, and the
limit as h0 tends to zero is used near zero.opls() has the arguments of the other orthogonalization
methods:
opls(x, y, ncomp = NULL, center = TRUE, scale = FALSE, crossval = 7, permutation = 0).
ncomp (the number of orthogonal components,
NULL for the automatic choice) replaces
ncomp.ortho, and the logical center and
scale replace the character scale; for Pareto
scaling, apply pareto_scale() to x first (or
step_pareto_scale() before step_opls()), which
gives the same filter. The former arguments still work, with a
deprecation warning. The permutation test is now skipped by default
(permutation = 0), as it refits the model as many
times.projected_osc()’s ncomp is now the number
of orthogonal components removed, as in opls(),
osc() and direct_osc(), instead of the number
of PLS components (one more): replace ncomp = k + 1 with
ncomp = k. The default is 4 (formerly 5, the same filter).
step_projected_osc() is unchanged.macropca() and robust_bcyj() (and
step_macropca(), step_robust_bcyj()) are
implemented natively, after the published algorithms, and no longer
depend on cellWise, which moves to Suggests.
macropca() detects the deviating cells with DDC
(Rousseeuw and Van den Bossche, 2018), whose neighbor search and cell
predictions are computed in C++ by blocks of variables: on the 368 x
7152 forageLIBS spectra, DDC takes about 5 s instead of 13
s, and macropca() about 12 s instead of 16 s. The results
are close to, but not the same as, those of
cellWise::MacroPCA() (orthogonal distances correlated at
0.99 on forageLIBS, nearly the same subspace); the score
and orthogonal distance cut-offs are those of robpca(). In
simulations with known rowwise and cellwise outliers and missing values,
it estimates the PCA subspace more accurately than
cellWise::MacroPCA(), imputes the missing values as
accurately, and flags far fewer regular observations as outliers
(cellWise compares the distances computed with the outlying cells to a
cut-off estimated with them imputed). Its arguments scale,
ndir, maxiter and tol replace the
... passed to cellWise::MacroPCA(), the result
has no fit element any more, and imputed holds
the data with the missing values imputed.robust_bcyj() fits the transformations of Raymaekers
and Rousseeuw (2021), with nearly the same parameters as
cellWise::transfo(), and leaves the variables that cannot
be transformed unchanged (method "none").boltzmann_plot() and saha_boltzmann_plot()
are renamed boltzmann() and saha_boltzmann(),
so that they are not confused with plot_boltzmann(), which
draws their result. The old names still work, with a deprecation
warning.dmodx(), the nominal degrees of freedom (K - k) are
now selected with df = "nominal".soilLIBS data set is removed, to keep the package
data under the 5 MB CRAN guideline: data(soilLIBS) no
longer works. The examples use forageLIBS instead; the
example of reject_shots(), which relied on the soil
replicates, is removed. The raw data and the script that built it remain
in data-raw/ of the source repository.som() fits a self-organizing map of spectra (batch SOM,
in C++), which maps similar spectra onto the same or neighboring units
of a hexagonal or rectangular grid and follows nonlinear relations
(matrix effects, self-absorption, plasma changes) that PCA can miss. It
starts from the plane of the first two principal components
(deterministic training), chooses the grid from the number of spectra,
and reports the quantization and topographic errors.
robust = TRUE down-weights outlying spectra by Huber
weights of their quantization errors. predict() places new
spectra on their best-matching unit and flags as novel those beyond a
robust cut-off of the training quantization errors.
plot_som() draws the counts, the U-matrix, the quantization
errors, the component planes (one map per wavelength or line), the
spectra on the map and the prototype spectra, and
som_stability() measures how consistently the spectra keep
their neighbors over resampled fits. On the 368 x 7152
forageLIBS spectra, a map takes about 2 s.plot_spectra() has an offset argument, to
shift successive spectra vertically and horizontally
(offset = c(horizontal, vertical), or a single number for a
vertical offset), for example to stack them.plot_embedding() recognizes UMAP maps (axes named
UMAP1, UMAP2, …, as by
embed::step_umap()): it leaves out the lines through the
origin, whose position means nothing on such a map, and ignores
hotelling ("all" or "group"),
with a warning, since its T-squared limits assume linear scores. On
other maps, the lines through the origin are light grey instead of
black.correlation() takes several responses
(var = c(K, Ca), or dplyr::all_of(minerals)),
each with its own observations, so that a response with many missing
values does not reduce the data of the others. The result gains an
outcome column, and plot = TRUE draws a
heatmap of the responses against the wavelengths (or the other
variables), on a fixed scale from -1 to 1. Pearson and Spearman
correlations of spectra are computed in one step: the 12 minerals of
forageLIBS against its 7152 channels take under a second.
With cluster = TRUE, the rows are ordered by a hierarchical
clustering of the responses on their correlation profiles (1 - r,
average linkage), so that responses whose correlations rise and fall at
the same wavelengths are adjacent, with the dendrogram on the left.plot_calibration()’s “Linear
calibration” above its R², LOD and LOQ. This applies to every plot with
a title, including the interactive (plotly) ones and
plot_cell_map(). plot_fit() now shows its
title as a title rather than a subtitle.forageLIBS spectra and mineral
contents instead of simulated data, wherever the function applies to
them (some taken from the vignettes). Examples of pure helper functions
(line profiles, plasma criteria, tuning parameters) keep simple inputs,
and the examples of pds() and
step_reject_shots(), for which forageLIBS has
no suitable data, are removed. plot_influence() gains an
example.dplyr::, so
that it runs without dplyr attached.forageLIBS spectra: fitting emission
lines (vignette("peak-fitting"), which replaces
"line-fitting"), preprocessing with recipe steps
(vignette("preprocessing")), calibration curves and figures
of merit (vignette("calibration")) and plasma diagnostics
(vignette("plasma-diagnostics")). The orthogonalization
vignette is folded into the preprocessing one.reject_shots(),
msc() and emsc() with
robust = TRUE, center(method = "median"),
robpca()): reject_shots() is about 20 times
faster on 400 spectra of 7152 channels.q_residuals() and dmodx() compute the
residuals of the calibration samples from the scores of the components
left out, without reconstructing the data (about 100 times faster), and
line_intensities() measures all the spectra of a line at
once (about 3 times faster). The results are unchanged.opls() is implemented natively and no longer needs the
Bioconductor package ropls. It reproduces ropls::opls()
(scores, loadings, R2X, R2Y, Q2, RMSEE, VIP and permutation p-values),
but returns an object of class specproc_opls instead of a
list holding the ropls model: the model element is removed,
and the model has new elements correction (the
OPLS-filtered data), fitted, coefficients,
vip, ortho_vip, components (R2X,
R2Y and Q2 of each component) and permutation.
x and y can be matrices or vectors as well as
data frames. When ncomp.ortho = NA and only the predictive
component is significant, the model has no orthogonal component (with a
warning) instead of failing. predict() filters new data, or
predicts the response or the scores (type), and
crossval = 0 skips the cross-validation.specLIBS is now
soilLIBS, and fourrage is now
forageLIBS. Replace data(specLIBS) with
data(soilLIBS) and data(fourrage) with
data(forageLIBS).macropca() (and step_macropca()) now only
centers the variables by default, instead of also scaling them as
cellWise::MacroPCA() does: scaling spectra gives noise and
continuum channels as much weight as emission lines. Use
macropca(x, scale = TRUE) for the previous behavior.plot_cell_map() is redrawn with ggplot2 and patchwork
instead of cellWise::cellMap(), for spectra: the map fills
the plot, with a wavelength axis, cells combined into at most
resolution blocks (within detector segments), flagged cells
colored by the sign and size of their residual, a strip of the outlier
type of each observation, rows optionally sorted by orthogonal distance
(order = "od"), and a top panel of the share of flagged
cells at each wavelength (profile). The arguments
nrowsinblock, ncolumnsinblock and
... are replaced by resolution. The profile
shows the mean spectrum in grey (of the imputed data, or of
spectra) and labels the most flagged regions, with the
emission lines they match (lines), and
order = "cluster" groups the observations with similar
flagged cells.flagged_regions() lists the wavelength regions where
many observations have cells flagged by macropca() (at
least threshold of them), with their extent, share,
direction and matching emission lines.plot_influence() is drawn like
plot_outlier_map(): points outlined in black and filled by
type (in the colors of the outlier map), dashed limits at the highest
confidence level (dotted, … at the others, with a legend only for
several levels), a compact legend at the bottom, and labels on the
labels (default 3) most outlying samples only, instead of
all the flagged ones. It gains the options of
plot_outlier_map(): relative (distances
divided by their limits), shade (the outlying regions),
log, colour_by = "distance" and point styles
in ..., including a size per sample. The arguments after
label changed order: name log and
title. In both plots, labels near the right edge are placed
on the left of their point, so that they are not cut off.
step_opls() removes the orthogonal components of an
opls() model in a recipe, with num_comp
tunable. It gives the same filtered data as
step_projected_osc(), and also offers Pareto
scaling.
step_o2pls() removes the outcome-orthogonal
components of an o2pls() model in a recipe, estimated
against one or several outcomes (recipe(K + Ca ~ ., ...)),
with num_comp and joint_comp tunable for a
single outcome. Only the predictors are filtered. Its help page shows a
workflow predicting several outcomes and, since tune_grid()
does not support several outcomes, how to tune one workflow per outcome
with a workflow set (workflow_map("tune_grid", ...)), each
filter still estimated against all the outcomes.
plot_embedding() is drawn with a grey panel outside
the outermost ellipse (of T-squared, or of each group) and white inside,
with no grid, a fixed aspect_ratio (0.7) and lines through
the origin. The axis titles give the explained variance of the
components of PCA fits, and the T-squared ellipses of the groups
(hotelling = "group") are filled with their color.
size can be a variable (such as a concentration) that sets
the size of the points, and biplot = TRUE draws the
loadings of a PCA over the scores, with the biplot_top most
important variables (emission lines, for spectra) as labeled arrows.
With hotelling = "all", the T-squared of a PCA model is
that of the model (its eigenvalues, centered at 0): for a robust fit,
the ellipses are no longer inflated or tilted by the outlying samples,
and the flagged samples are the leverage points of
plot_outlier_map(). Without groups,
ellipse = TRUE with hotelling = "all" draws
only the T-squared ellipse, with a warning, instead of two ellipses of
all the samples. The samples beyond the T-squared limit are counted in
the subtitle, circled in red only with flag = TRUE, and
labeled only with label (TRUE, or a column or
vector of labels), independently of each other.
plot_outlier_map() takes a size per sample
(size), and colour_by = "distance" colors its
points by their reduced distance from the origin (the larger of SD and
OD over their cut-offs), on a rainbow from dark red to blue; with
relative = TRUE, where both cut-offs are at 1, yellow marks
the cut-offs, between the regular observations (dark red to orange) and
the outlying ones (green to blue).
The legends of plot_embedding() and
plot_outlier_map() are more compact: smaller text and keys,
three sizes, the T-squared levels written on their ellipses instead of
in a legend, and no key for the kind of samples without new
samples.
contributions() computes the contributions of each
variable to the Q residual or to Hotelling’s T-squared of samples (whose
squares add up to Q and T-squared), for a prcomp() fit or a
robust fit (where they decompose the orthogonal and score distances),
and relative contributions against reference samples
(reference, or "regular" for all regular
samples). plot_contributions() draws them against
wavelength with the largest labeled and matched to emission
lines.
plot_loadings() plots the loadings of a PCA
(prcomp(), robpca(), rospca() or
macropca()) against wavelength, one panel per component,
and labels the wavelengths that contribute most to each component, with
the emission lines they match in a line list from
libs_lines(). The mean spectrum can be drawn behind the
loadings, type = "contribution" plots the variance of each
wavelength explained by the components, and
interactive = TRUE makes a plotly figure.
loading_peaks() returns the labeled peaks as a table, with
their candidate lines and a flag for derivative shapes (line
shifts).
plot_outlier_map() gains relative, to
plot the reduced distances (divided by their cut-offs) so that maps of
different models share one scale, shade, to shade the good
leverage, orthogonal outlier and bad leverage regions in greys of
increasing darkness and name them, and log, for logarithmic
axes. The points are outlined in black, and ... passes
styling such as alpha or size to
geom_point().
dmodx() computes the distance to the model in the
space of the variables (DModX) of a PCA model, normalized or absolute,
for the calibration samples or new ones. Its limits use the effective
number of residual dimensions by default
(df = "effective"), because the nominal degrees of freedom
(df = "nominal") flag a large share of ordinary samples
when spectra have far more channels than samples.
plot_influence() draws DModX or Q against
T-squared.
hotelling_t2(), q_residuals(),
dmodx(), plot_influence() and the T-squared
ellipses of plot_embedding() take any confidence level(s)
(conf_level, 0.975 by default; in
plot_embedding(), one conf_level sets both its
confidence and T-squared ellipses), and the Beta distribution of
T-squared for the samples of the model (method = "beta").
hotelling_t2() and the T-squared ellipses of
plot_embedding() now use the HotellingEllipse package
(1.3.0 or later).
wavelength_calibration() fits a correction of the
wavelength axis from reference lines (for example, from the NIST
database), located to a fraction of a channel by Gaussian interpolation,
with one correction per detector segment (constant, linear or quadratic)
and automatic rejection of edge, weak, saturated and outlying lines.
apply_calibration() relabels the wavelengths of spectra
without changing their intensities, predict() corrects any
wavelength, and plot_wavelength_calibration() draws the
offsets and the fit.
savitzky_golay() also splits the axis where
overlapping detectors make the wavelengths step back.
The periodic table of line_finder() can be hidden
(button Hide table, or show_table = FALSE
at start) to give the spectrum the whole height of the window; the
selected elements stay listed in the panel header.
savitzky_golay() and step_savgol():
Savitzky-Golay smoothing and first or second derivatives, keeping every
channel (polynomial fits at the edges) and filtering the segments
between detector gaps separately. The window, polynomial order and
derivative are tunable (new dials parameter
savgol_derivative()).
normalize() gains the L1 ("l1"), L2 or
vector ("l2") and maximum ("max") norms, and
step_spectral_norm() normalizes each spectrum in a recipe
by its L1 norm, total area, L2 norm or maximum, with the method tunable
(new dials parameter spectral_norm_method()).
wavelet_features() and step_wavelet():
discrete wavelet transform of spectra (Haar, Daubechies d4/d6/d8 and
least asymmetric la8 wavelets) as compressed features: the approximation
at a level or all coefficients, optionally the coefficients of largest
variance in the training data. The level and number of coefficients are
tunable (new dials parameter wavelet_level()).
q_residuals() computes Hotelling’s T-squared and the
Q residual (SPE) of each sample for a PCA model, for the calibration
samples or new ones, with their 95% and 99% limits (Jackson-Mudholkar or
Box for Q), and classifies the samples as regular, extreme, residual or
both. plot_influence() draws Q against T-squared with the
limits.
calibration_curve() gains prediction intervals:
plot_calibration() draws the confidence band, the
prediction band or both (interval), and shows new samples
(newdata) at their predicted concentrations with their
intervals; predict(type = "signal") gives the expected
signal at given concentrations with a confidence or prediction interval;
the coefficients have confidence intervals.
The plots of correlation() are redesigned. For
spectra (variables named by wavelength), the plot is a correlation
spectrum with the 5% significance thresholds; otherwise, a sorted chart
colored by sign and labeled with the values, with top to
show only the strongest correlations. The interactive versions are built
directly with plotly (WebGL for spectra) instead of converted from
ggplot2. color now takes one color or two (positive and
negative correlations).
plot_embedding() draws two-dimensional embeddings
(UMAP from embed::step_umap(), principal components from
recipes::step_pca(), prcomp() or
robpca()), colored by a variable, with optional confidence
ellipses per group (classical or robust, normal or Hotelling quantiles)
from the ConfidenceEllipse package, and Hotelling’s T-squared 95% and
99% ellipses with the outlying samples labeled (hotelling,
for all samples or within groups, on k components). The
preprocessing vignette uses it to compare PCA and UMAP maps of the
samples.
hotelling_t2() computes Hotelling’s T-squared of
each sample on k components of an embedding, with its 95%
and 99% limits, for all samples or within groups.
macropca() failed when k was not given:
cellWise::MacroPCA() with k = 0 only reports
the explained variance. k is now chosen from the new
arguments var_explained (0.8) and kmax (10),
as in robpca(), and MacroPCA’s message and scree plot are
no longer shown.cf_libs(): calibration-free LIBS quantification (Ciucci
et al., 1999). The temperature is estimated from the common slope of
parallel Boltzmann plots (one per species) or, when the electron density
is known, Saha-Boltzmann plots (one per element, much more precise). The
densities follow from the intercepts and the partition functions, and
the ionization stages not observed from the Saha equation. The
composition (atomic and mass fractions) is normalized by closure or by
the known concentration of one element. plot_boltzmann()
draws the plots of the fit. The plasma diagnostics vignette applies it
to the forage samples and compares it with their laboratory values.nist_levels() retrieves energy levels from the NIST
Atomic Spectra Database, and partition_function() computes
partition functions from them, optionally truncated at a maximum energy
(such as the lowered ionization energy).reject_shots() flags the outlying laser shots of each
sample (total intensity, correlation with or distance to the median
spectrum, by robust z-scores), before they are averaged with
average().line_intensities() measures the area (or height, or
Voigt-fitted area) of emission lines in spectra, searching each peak
near its tabulated wavelength, with a signal-to-noise ratio and
saturation check. It keeps the columns of a table of lines, so its
result feeds boltzmann_plot(), cf_libs() and
calibration_curve(). step_line_intensities()
does the same in a recipe.correct_self_absorption() corrects line intensities for
self-absorption by the internal reference method of Sun and Yu (2009),
with the temperature given or estimated from a Saha-Boltzmann plot.calibration_curve() fits univariate (linear or
quadratic, optionally weighted) calibration curves, with the
sensitivity, LOD and LOQ, Mandel’s and lack-of-fit tests of linearity,
inverse prediction of concentrations with confidence intervals
(predict()) and plot_calibration().step_reject_shots() removes the
outlying shots from the training data (skipped on new data), and
step_line_ratio() normalizes each spectrum to the area or
height of a reference line (internal standard).line_finder(): a Shiny app to identify the emission
lines of LIBS spectra. Elements are selected on a periodic table, and
the strongest lines of their ionization stages, from the NIST Atomic
Spectra Database, are overlaid on a spectrum in an interactive plotly
graph (one color per stage), with controls for the temperature,
wavelength range, number of lines, wavelength shift and spectrum (mean,
group mean or single spectrum). Fetched lines can be saved and reloaded
for offline use.libs_lines() lists the lines of selected species
expected to be the strongest in a plasma at a given temperature
(relative intensities in LTE), and plot_lines() overlays
them on a spectrum (plotly or ggplot2).This release adds plasma diagnostics for LIBS: electron density from Stark broadening, excitation temperature from Boltzmann and Saha-Boltzmann plots, and checks of LTE, self-absorption and detector saturation, with atomic data from the NIST Atomic Spectra Database and Stark parameters from STARK-B.
starkb_lines() retrieves Stark widths and shifts of the
lines of an atom or ion from the STARK-B database (Sahal-Bréchot,
Dimitrijević and Moreau) through its VAMDC service, on demand;
read_starkb() reads STARK-B data saved as XSAMS files, for
offline and reproducible work; stark_table() builds the
same table from user-supplied widths or from fitted temperature laws.
stark_width() interpolates the width at a given temperature
and electron density, and scales multiplet data to a line of the
multiplet (lambda-squared rule).electron_density() estimates the electron density from
the Stark (Lorentzian) width of a line, with STARK-B data or a reference
width, or from the H-alpha line (Gigosos et al., 2003).boltzmann_plot() and saha_boltzmann_plot()
estimate the excitation temperature from line intensities and atomic
data, and plot_boltzmann() draws the plots.mcwhirter_criterion() checks the McWhirter criterion
for LTE, and self_absorption() computes self-absorption
coefficients from line widths (El Sherbini et al., 2005).saturation_summary() finds channels at the saturation
limit of the detector.nist_lines() and nist_ionization_energy()
retrieve transition probabilities, level energies, statistical weights
and ionization energies from the NIST Atomic Spectra Database, on
demand.voigt_fwhm() computes the full width at half maximum of
a Voigt profile (Olivero and Longbothum, 1977).specLIBS and fourrage
spectra.This release adds robust PCA, a proper net analyte signal with figures of merit, and rewrites the documentation around tidymodels.
nas() now computes the net analyte signal (Lorber,
Faber and Kowalski, 1997; Faber, 1998) of an inverse PLS or PCR
calibration model, with its figures of merit: sensitivity, selectivity
and, given the noise level, analytical sensitivity, limits of detection
and quantification, and signal-to-noise ratios. A predict()
method gives the NAS, selectivity and predicted concentration of new
samples. Previously, nas() returned spectra filtered by
direct orthogonalization, identical to direct_orthogonal(),
which remains available for that purpose.robpca() (ROBPCA, Hubert, Rousseeuw and
Vanden Branden,
2005) and rospca() (robust sparse PCA, Hubert, Reynkens,
Schmitt and Verdonck, 2016), implemented from the published algorithms
with C++ kernels for the Stahel-Donoho outlyingness, FAST-MCD and the
grid-search sparse PCA; and macropca(), a wrapper around
cellWise::MacroPCA() for cellwise outliers and missing
values. All three return score and orthogonal distances with cut-offs,
an outlier classification, and a predict() method for new
observations.plot_outlier_map() draws the outlier map (score
distance against orthogonal distance) of any of the three, optionally
with new observations; plot_cell_map() draws the cell map
of a MacroPCA fit.step_robpca(),
step_rospca() and step_macropca(), robust
counterparts of recipes::step_pca() that can also add the
score and orthogonal distances as columns, and
step_robust_bcyj(), a robust counterpart of
recipes::step_BoxCox() and
recipes::step_YeoJohnson().rsample::nested_cv() and
tune::tune_grid(), and the orthogonalization vignette tunes
every filter together with the PLS model. Vignettes that need suggested
packages stop with a message when those are not installed.This release connects specProc to the tidymodels framework.
Preprocessing and orthogonalization can now be estimated on calibration
data and applied to new spectra, either directly with
predict() or as recipe steps that are re-estimated on every
resample and tuned together with the model.
epo(), osc(),
direct_orthogonal(), direct_osc(),
projected_osc() and o2pls() return classed
objects with a predict() method, which corrects new data
with the centers, scales and components estimated on the calibration
data. Existing fields are unchanged.step_epo(),
step_glsw(), step_osc(),
step_direct_orthogonal(), step_direct_osc(),
step_projected_osc() and
step_y_gradient_glsw(), with tidy() and
tunable() methods, and the dials parameter
glsw_alpha(). In a workflow, the filters are re-estimated
on every resample and can be tuned with the model. The GLSW steps store
the filter in factored form instead of a p x p matrix. recipes and dials
are suggested, not required.emsc(): extended multiplicative signal correction
(Martens and Stark, 1991), with a polynomial baseline and optional
interferent spectra, and a predict() method for new
spectra. With degree = 0 it equals msc().step_baseline() (arPLS, ALS or LSP),
step_snv(), step_msc(),
step_emsc(), step_pareto_scale() and
step_poisson_scale(). MSC and EMSC estimate the reference
spectrum, and the scaling steps the column scales, on the training data
only. The baseline lambda or degree and the
EMSC degree are tunable; the new dials parameter
baseline_lambda() covers lambda.fourrage: LIBS spectra of 365 forage
samples with reference contents of 12 elements, and a vignette based on
it, “Removing unwanted variation: a comparison of orthogonalization
methods” (what EPO, GLSW, the OSC family, DO/NAS, DOSC, POSC/OPLS and
y-gradient GLSW remove, whether it improves potassium predictions, and
which methods are equivalent).This release fixes a large number of bugs so that every exported function now runs and returns numerically correct results. It adds a test suite (400+ expectations), and several results are checked against independent reference implementations.
Functions were renamed to a consistent snake_case scheme. The old
names still work but give a deprecation warning (see
?"specProc-deprecated"): whittaker() ->
baseline_als(), lorentzian() ->
lorentzian_profile(), pseudo_voigt() ->
pseudo_voigt_profile(), peakfit() ->
peak_fit(), multipeakfit() ->
multipeak_fit(), plotfit() ->
plot_fit(), plotSpec() ->
plot_spectra(), outlierplot() ->
plot_outliers(), directOutlyingness() ->
directional_outlyingness(), iqrMethod() ->
iqr_outliers(), robustBCYJ() ->
robust_bcyj(), rousseeuwCroux() ->
rousseeuw_croux(), summaryStats() ->
summary_stats(), tukeyGH() ->
tukey_gh(), yGradientglsw() ->
y_gradient_glsw(), pareto() ->
pareto_scale().
gaussian() is renamed to
gaussian_profile() without an alias,
because exporting gaussian() masked
stats::gaussian(), which breaks
glm(family = gaussian) when specProc is loaded.
baseline_arpls(), baseline_lsp() and
whittaker() return tibbles (they previously returned plain
lists of columns). They no longer replace negative corrected values with
1.
average() with .group_by now returns
the group labels as the first column.
biweight_location() and
biweight_scale() use the unscaled MAD, following the
standard definition (Beers et al., 1990), so that
biweight_scale()^2 equals
biweight_midvariance(). biweight_scale() loses
its unused tol and max_iter arguments.
loc = NULL now means “median of x”.
direct_orthogonal() loses its unused
max_iter and tol arguments, and
direct_osc() loses max_iter. tol
now sets the pseudo-inverse tolerance.
epo() gains a clutter argument (the
matrix describing the external variation). Its default
ncomp is now 2, as documented. It now removes the
dominant singular directions; before, it removed the smallest
ones.
o2pls() has a new interface:
o2pls(x, y, ncomp, nx, ny, center, scale).
pds() returns transfer_matrix (was
transfert_matrix), alpha now blends the local
and global models, and progress is no longer printed.
projected_osc() always returns the corrected
training data. New data are preprocessed with the training centers and
scales.
quantile_weight() uses whole-distribution quantiles,
as in Brys et al. (2006), and defaults to p = 0.125,
q = 0.875.
summaryStats(robust = TRUE) drops the
rsd column. It applied the 1.4826 consistency factor twice,
and mad already holds the correct value.
opls() now requires the Bioconductor package ropls
to be installed (it moved to Suggests). It no longer prints output or
draws plots.
R >= 4.1.0 is required.
baseline_arpls(),
whittaker(), baseline_lsp() (data frames),
correlation(method = "bicor"),
directOutlyingness(maxRatio = ), direct_osc(),
msc(), multipeakfit(), nas(),
o2pls(), osc() ("wold",
"fearn"), pds(), peakfit(),
projected_osc() and yGradientglsw().glsw() and yGradientglsw() squared the
eigenvalues of the covariance matrix when building the filter.msc() regressed each wavelength instead of each
spectrum on the reference.generalized_boxplot() transformed the ranks instead of
the data. This made the g-and-h estimates meaningless. It also failed
when the two tails had different numbers of outliers.pseudo_voigt() counted the area and the offset
twice.biweight_scale() used a wrong exponent in the
denominator. It also ignored reduced = TRUE.biweight_location() did not iterate.average() did not ignore missing values. It also read
out of bounds for factors with unused levels or missing group
values.normalize(method = "background") rejected data frames
for bkg.plotSpec() connected all spectra into a single line
when id was not given.outlierplot() no longer changes the user’s random
number generator state.iqrMethod() no longer changes global options. Missing
values no longer cause errors.Three vignettes based on specLIBS: “Preprocessing
LIBS spectra” (baseline, normalization judged by replicate RSD and
intraclass correlation, robust screening of shots), “Fitting emission
lines” (profile choice, sources of uncertainty, identifiability) and
“Predicting soil clay content from LIBS spectra” (a compositional
log-ratio PLS model with nested, repeated cross-validation by sample,
and the optimism of common shortcuts).
plot_fit() draws the fitted profiles on a fine
wavelength grid.
Functions taking a response now accept 1-d arrays, such as the
output of tapply().
New voigt_profile(): the exact Voigt profile,
computed in C++ from the Faddeeva function with Weideman’s (1994)
rational approximation (relative error below 1e-10).
peak_fit() and multipeak_fit() now use it for
profile = "voigt"; the pseudo-Voigt approximation is
available as profile = "pseudo_voigt".
New data set specLIBS: 400 LIBS spectra (50 soil
samples x 8 locations, 7152 channels, 199-822 nm) with the clay, sand
and silt content of each sample.
generalized_boxplot() whiskers now end at the most
extreme observations within the fences, as in standard boxplots. The
fences are returned as lower_fence and
upper_fence.
The README has been rewritten around a complete worked example on
specLIBS.
peakfit() and multipeakfit() estimate
starting values from the data when wL, wG or
A are not given. multipeakfit() fits all lines
simultaneously and returns each line’s contribution. A spectrum that
fails to fit produces a warning instead of stopping the whole
batch.NEWS.md file to track changes to the
package.