# 1 Overview

Spectral preprocessing is a critical step in near-infrared (NIR) calibration workflows. Raw spectral data often contains systematic variations, noise, and artifacts that can obscure the true relationship between spectra and reference properties. The `proximetricsR` package provides a flexible, composable system for building and applying preprocessing pipelines.

## 1.1 Key concepts

- Preprocessing constructors (`prep_*()` functions): Create specifications for individual preprocessing steps
- Preprocessing recipes (`preprocess_recipe()`): Assemble multiple constructors into an ordered sequence
- Recipe execution (`process()`): Apply the recipe to spectral data matrices
- Device compatibility: Different devices (ProxiMate, ProxiScout) support different preprocessing algorithms

This separation of specification and execution enables reproducible, device-aware preprocessing pipelines that can be stored, shared, and applied consistently.

# 2 Setup

    library("proximetricsR")

    data("proximateCannabis")
    X <- proximateCannabis$spc

# 3 Preprocessing constructors

The `prep_*()` functions create preprocessing step objects. Each constructor validates its parameters and encodes algorithm-specific information. The order in which constructors are passed to `preprocess_recipe()` defines the execution order.

## 3.1 Resampling: `prep_resample()`

Resampling interpolates spectra to a new wavelength or wavenumber grid.

ProxiMate mode (user-defined grid):

    prep_resample(grid = c(1001, 1700, 2))

    - prep_resample
        min_wav: 1001; max_wav: 1700; resolution: 2

ProxiScout mode (NeoSpectra standard grid):

    prep_resample(grid = "proxiscout")

    - prep_resample

Resampling is often the first step to standardize wavelength grids across different instruments.

## 3.2 Smoothing: `prep_smooth()`

Smoothing reduces high-frequency noise while preserving spectral features.

Savitzky-Golay (ProxiScout compatible):

    prep_smooth(w = 11, p = 3, algorithm = "savitzky-golay")

    - prep_smooth
        w: 11; p: 3; algorithm: 'savitzky-golay'

Moving average (ProxiMate compatible):

    prep_smooth(w = 7, algorithm = "moving-average")

    - prep_smooth
        w: 7; algorithm: 'moving-average'

- `w`: window size (must be a positive odd integer)
- `p`: polynomial order for Savitzky-Golay (must be < w)

## 3.3 Standard Normal Variate: `prep_snv()`

SNV (Standard Normal Variate) normalizes each spectrum independently by centering and scaling:

\[SNV_i = \frac{x_i - \bar{x}_i}{s_i}\]

where \(\bar{x}_i\) and \(s_i\) are the mean and standard deviation of the \(i\)-th spectrum.

    prep_snv()

    - prep_snv

SNV corrects for multiplicative effects (e.g., baseline offsets, path length variations) and is device-agnostic.

## 3.4 Derivatives: `prep_derivative()`

Derivatives enhance spectral differences and reduce baseline effects.

Savitzky-Golay (ProxiScout):

    prep_derivative(m = 1, w = 11, p = 3, algorithm = "savitzky-golay")

    - prep_derivative
        m: 1; w: 11; p: 3; algorithm: 'savitzky-golay'

Gap-Segment (ProxiScout):

    prep_derivative(m = 2, w = 9, p = 3, algorithm = "gap-segment")

    - prep_derivative
        m: 2; w: 9; p: 3; algorithm: 'gap-segment'

NIRWise PLUS compatible (ProxiMate):

    prep_derivative(m = 1, w = 5, p = 11, algorithm = "nwp")

    - prep_derivative
        m: 1; w: 5; p: 11; algorithm: 'nwp'

Parameters: - `m`: derivative order (1 or 2) - `w`: window/gap size (positive odd integer) - `p`: polynomial order (Savitzky-Golay) or smoothing window (gap-segment, nwp) - `algorithm`: choice of method

## 3.5 Detrending: `prep_detrend()`

Detrending removes wavelength-dependent baseline effects by fitting and removing a polynomial trend (ProxiScout only):

    prep_detrend(p = 2)

    - prep_detrend
        p: 2

- `p`: polynomial order (default 2)

For the full Barnes et al. (1989) procedure (SNV + detrending), chain `prep_snv()` before `prep_detrend()`.

## 3.6 Reflectance/Absorbance conversion: `prep_transform()`

Convert between reflectance and absorbance using Beer’s Law (ProxiScout only):

\[A = -\log_{10}(R)\]

    prep_transform(to = "absorbance")

    - prep_transform
        to: 'absorbance'

- `to`: target unit (`"absorbance"` or `"reflectance"`)

## 3.7 Wavelength trimming: `prep_wav_trim()`

Retain only a specified wavelength band and/or remove constant-valued edge columns (ProxiScout only):

    prep_wav_trim(band = c(1000, 1800), trim_constant_edges = TRUE)

    - prep_wav_trim
        band: 1000, 1800; trim_constant_edges: TRUE

- `band`: wavelength range to retain (or `c()` to skip)
- `trim_constant_edges`: remove zero or constant-valued columns at edges

# 4 Building preprocessing recipes

The `preprocess_recipe()` function combines constructors into an ordered pipeline. Order matters: preprocessing steps are applied in the order specified.

## 4.1 Device compatibility

Different BUCHI devices support different preprocessing steps:

ProxiMate supports: - `prep_resample()` with user-defined grids - `prep_smooth()` with moving-average algorithm - `prep_snv()` - `prep_derivative()` with nwp algorithm

ProxiScout supports: - `prep_resample()` with “proxiscout” grid - `prep_smooth()` with savitzky-golay algorithm - `prep_snv()` - `prep_derivative()` with savitzky-golay or gap-segment algorithms - `prep_detrend()` - `prep_transform()` - `prep_wav_trim()`

## 4.2 Building recipes

Single preprocessing step (SNV only):

SNV is device-agnostic, so `device` is optional:

    recipe_snv <- preprocess_recipe(prep_snv())
    recipe_snv

    Spectral preprocessing recipe (device: "unspecified"):
     - Step 1: prep_snv

Multiple steps (requires device):

    recipe_ps <- preprocess_recipe(
      prep_smooth(w = 7, p = 1, algorithm = "savitzky-golay"),
      prep_snv(),
      prep_derivative(m = 1, w = 5, p = 2, algorithm = "savitzky-golay"),
      device = "proxiscout"
    )
    recipe_ps

    Spectral preprocessing recipe (device: "proxiscout"):
     - Step 1: prep_smooth
        w: 7; p: 1; algorithm: 'savitzky-golay'
     - Step 2: prep_snv
     - Step 3: prep_derivative
        m: 1; w: 5; p: 2; algorithm: 'savitzky-golay'

ProxiMate-specific recipe:

    recipe_pm <- preprocess_recipe(
      prep_smooth(w = 7, algorithm = "moving-average"),
      prep_snv(),
      prep_derivative(m = 1, w = 5, p = 11, algorithm = "nwp"),
      device = "proximate"
    )
    recipe_pm

    Spectral preprocessing recipe (device: "proximate"):
     - Step 1: prep_smooth
        w: 7; algorithm: 'moving-average'
     - Step 2: prep_snv
     - Step 3: prep_derivative
        m: 1; w: 5; p: 11; algorithm: 'nwp'

Recipes validate that all steps are compatible with the specified device and raise informative errors if not.

# 5 Applying recipes with `process()`

The `process()` function executes a recipe on spectral data:

    X_snv <- process(X, recipe_snv)
    dim(X_snv)

    [1]  80 234

    X_ps <- process(X, recipe_ps)
    dim(X_ps)

    [1]  80 224

The applied recipe is stored as an attribute and can be retrieved:

    applied_recipe <- attr(X_ps, "preprocess_recipe")
    applied_recipe

    Spectral preprocessing recipe (device: "proxiscout"):
     - Step 1: prep_smooth
        w: 7; p: 1; algorithm: 'savitzky-golay'
     - Step 2: prep_snv
     - Step 3: prep_derivative
        m: 1; w: 5; p: 2; algorithm: 'savitzky-golay'

# 6 Practical examples

## 6.1 Example 1: ProxiMate workflow

A typical ProxiMate workflow for fat/protein prediction:

    recipe_pm_fat <- preprocess_recipe(
      prep_smooth(w = 7, algorithm = "moving-average"),
      prep_snv(),
      prep_derivative(m = 1, w = 5, p = 11, algorithm = "nwp"),
      device = "proximate"
    )

    X_fat_prep <- process(X, recipe_pm_fat)
    head(X_fat_prep[, 1:5])

            1025    1028    1031    1034    1037
    [1,] -0.0137 -0.0138 -0.0138 -0.0135 -0.0131
    [2,] -0.0208 -0.0213 -0.0215 -0.0215 -0.0212
    [3,] -0.0184 -0.0186 -0.0187 -0.0185 -0.0181
    [4,] -0.0122 -0.0124 -0.0124 -0.0122 -0.0119
    [5,] -0.0170 -0.0171 -0.0170 -0.0167 -0.0161
    [6,] -0.0134 -0.0135 -0.0135 -0.0132 -0.0128

## 6.2 Example 2: ProxiScout workflow with detrending

ProxiScout instruments benefit from additional preprocessing steps:

    recipe_ps_full <- preprocess_recipe(
      prep_resample(grid = "proxiscout"),
      prep_smooth(w = 7, p = 1, algorithm = "savitzky-golay"),
      prep_snv(),
      prep_detrend(p = 2),
      prep_derivative(m = 1, w = 5, p = 2, algorithm = "savitzky-golay"),
      device = "proxiscout"
    )

    X_ps_full <- process(X, recipe_ps_full)
    dim(X_ps_full)

    [1]  80 102

## 6.3 Example 3: Minimal preprocessing

Sometimes less is more. A minimal recipe with only SNV:

    recipe_minimal <- preprocess_recipe(prep_snv())
    X_minimal <- process(X, recipe_minimal)

## 6.4 Example 4: Wavelength band selection

Select a specific wavelength range for ProxiScout:

    recipe_band <- preprocess_recipe(
      prep_wav_trim(band = c(1100, 1600)),
      prep_smooth(w = 5, p = 1, algorithm = "savitzky-golay"),
      prep_snv(),
      device = "proxiscout"
    )

    X_band <- process(X, recipe_band)
    colnames(X_band)[c(1, ncol(X_band))]

    [1] "1106" "1592"

# 7 Best practices

## 7.1 Order matters

Preprocessing steps affect each other. Common orderings:

1. Smoothing → SNV → Derivative (standard for noise reduction + normalization + enhancement)
2. Resampling → Smoothing → Derivative → SNV (device-specific resampling first)
3. SNV → Detrend (full de-trending procedure)

## 7.2 Device-aware development

Always specify `device = "proximate"` or `device = "proxiscout"` when building recipes (except for SNV-only recipes). This ensures recipes are portable and the preprocessing is compatible with the target device.

## 7.3 Reproducibility

Store recipes alongside calibration models to ensure preprocessing is applied identically during prediction. The `process()` function attaches the recipe as an attribute for downstream tracking.

## 7.4 Parameter tuning

Preprocessing parameters are typically tuned during model development:

- Window sizes should be small relative to spectral features of interest
- Derivative orders depend on spectral complexity (first-order for baseline removal, second-order for sharper features)
- SNV is almost universally beneficial for multiplicative interference

# 8 Summary

The preprocessing recipe system in `proximetricsR` provides a structured, reproducible approach to spectral preprocessing:

- Constructors define individual steps with validation
- Recipes compose steps into device-aware pipelines
- Process applies recipes consistently
- Attributes preserve recipe information for downstream use

This design enables seamless integration with calibration workflows and ensures preprocessing is applied consistently from model development through deployment on BUCHI devices.