---
title: "Common pitfalls / FAQ"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Common pitfalls / FAQ}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment  = "#>",
  fig.width  = 6,
  fig.height = 4
)
```

This vignette collects the questions early users most often hit when running
pigauto on their own data. Each section follows the same template:

- **Symptom** — the surprising output, in user voice.
- **Why this happens** — the mechanism in 2–3 sentences.
- **Diagnose** — 1–3 R commands that confirm the pattern.
- **Fix** — concrete code change with a short explanation.
- **See also** — links to the relevant `?function`, design memo, or
  Methodology bench under the Methodology navbar dropdown.

If you hit something that isn't here and feels surprising, please
[open an issue](https://github.com/itchyshin/pigauto/issues/new) — most
of the items below were added because real users tripped on them.

## 1. "`impute()` stopped because my input is fully observed"

**Symptom.** You call `impute(df, tree)` on a fully-observed dataset and
it stops before training.

**Why this happens.** `impute()` only *imputes* cells that are `NA`
in matched target rows. A fully observed input now stops before training
and directs you to `cross_validate()` for predictive assessment. Supply a
real missing target cell to use the imputation path. A complete data set is
for predictive assessment, not an imputation result.

**Diagnose.**

```{r section1-diag, eval = FALSE}
library(pigauto)
data(avonet300, tree300)
df <- avonet300
rownames(df) <- df$Species_Key
df$Species_Key <- NULL
sum(is.na(df))                  # if 0, there's nothing for impute() to do
```

**Fix.** Mask some cells before calling `impute()`, then evaluate
predictions only on the held-out cells:

```{r section1-fix, eval = FALSE}
set.seed(1L)
hide   <- sample(which(!is.na(df$Mass)), 30L)
df_obs <- df
df_obs$Mass[hide] <- NA          # hide 30 mass values

result <- impute(df_obs, tree300)

result$completed$Mass[hide]      # pigauto's imputations
df$Mass[hide]                    # held-out truth, for comparison
sum(result$imputed_mask[, "Mass"])  # 30
```

For your own data with real `NA`s, the imputed values you actually
care about are `result$completed[result$imputed_mask]`, not
`result$prediction$imputed`.

**See also.** `?impute` ("What gets imputed (read this first)"),
[issue #67](https://github.com/itchyshin/pigauto/issues/67).

## 2. "My ordinal trait predicted 100 % majority class"

**Symptom.** You impute an ordinal trait and the prediction is the
majority class for every species. For example, on `avonet300$Migration`
(K = 3 ordinal: Resident / Partial / Full), 300/0/0.

**Why this happens.** Two things compound:

1. If your input has no `NA`s in that column, `impute()` stops before fitting;
   use `cross_validate()` instead (see Pitfall 1).
2. At default settings (`n_imputations = 1L`, `pool_method = "median"`),
   a small ordinal trait whose marginal distribution is heavily skewed
   (AVONET `Migration` is ~78 % Resident / 14 % Partial / 8 % Full at
   n = 300) can have its calibrated gate snap to a corner that returns
   the majority class for every species.

**Diagnose.**

```{r section2-diag, eval = FALSE}
library(pigauto)
data(avonet300, tree300)
df <- avonet300; rownames(df) <- df$Species_Key; df$Species_Key <- NULL

table(df$Migration)              # check the marginal distribution
hide <- sample(which(!is.na(df$Migration)), 30L)
df$Migration[hide] <- NA
result <- impute(df, tree300, verbose = FALSE)
table(result$completed$Migration[hide])
```

**Fix.** Compare candidate settings with `cross_validate()` on your own
trait table and report the held-out result. For an imbalanced ordinal trait,
one candidate is more draws with `pool_method = "mode"`; it is not a
package-wide recommendation.

```{r section2-fix, eval = FALSE}
set.seed(1L)
hide   <- sample(which(!is.na(df$Migration)), 30L)
df_obs <- df
df_obs$Migration[hide] <- NA

# Default settings: prone to majority-class collapse on imbalanced K = 3
result <- impute(df_obs, tree300, verbose = FALSE)
table(result$completed$Migration[hide], df$Migration[hide])

# Recommended for K = 3 ordinal: more draws + mode pooling
result_mode <- impute(df_obs, tree300, n_imputations = 20L,
                      pool_method = "mode", verbose = FALSE)
table(result_mode$completed$Migration[hide], df$Migration[hide])
```

**See also.** `?impute` ("Imbalanced K-class traits"),
[Phase H memo](https://github.com/itchyshin/pigauto/blob/main/useful/MEMO_2026-05-01_phase_h_results.md),
[issue #68](https://github.com/itchyshin/pigauto/issues/68).

## 3. "The gate stays closed and the GNN seems to do nothing"

This section applies when you fit with `gnn = TRUE`. With the default
`gnn = FALSE` no GNN is trained, so `r_cal_gnn` is 0 by construction and
predictions come from the calibrated baseline.

**Symptom.** You expected the GNN to dominate, but inspecting the
fitted model shows the calibrated gate is fully or near-fully closed
(`r_cal_gnn` ≈ 0) — predictions equal the BM baseline.

**Why this happens.** This is the gate's design behaviour, not a
bug (with the default `safety_floor = FALSE` the mean term is absent and
`r_cal_mean` is 0; the next sentence describes `safety_floor = TRUE`). After training, pigauto picks the per-latent-column gate that
minimises validation loss across the simplex
$r_\text{BM} \cdot \text{BM} + r_\text{GNN} \cdot \text{GNN} + r_\text{MEAN} \cdot \text{MEAN}$.
When the GNN cannot beat BM on the held-out validation set, the
optimum can be `r_cal_gnn = 0`. In that case the calibrated prediction
stays on the validation-supported baseline or mean corner instead of
forcing a GNN contribution. This is what the package was designed to do
on high-phylogenetic-signal traits where BM is already hard to beat.

**Diagnose.**

```{r section3-diag, eval = FALSE}
library(pigauto)
data(avonet300, tree300)
df <- avonet300; rownames(df) <- df$Species_Key; df$Species_Key <- NULL
fit <- impute(df, tree300, verbose = FALSE)$fit

# Per-latent-column calibrated gates (since v0.9.1.9002):
fit$r_cal_bm        # r assigned to the BM baseline
fit$r_cal_gnn       # r assigned to the GNN delta
fit$r_cal_mean      # r assigned to the grand mean
```

A row where `r_cal_gnn` is small (< 0.1) means the gate has effectively
closed for that latent column.

**Fix.** Often there is nothing to fix — the closed gate is
**evidence of high phylogenetic signal**, not a problem. If you suspect
the GNN should be helping (e.g. you've added covariates, or the trait
has known cross-trait structure) but the gate is closed:

- Check the validation set is not pathologically small (the
  "small validation set" warning during fitting is a red flag — see
  `?fit_pigauto` "Calibration at small n").
- Verify covariates are not all-NA or constant after preprocessing.
- For ordinal / categorical traits, see Pitfall 2 — the gate may be
  closing onto a majority-class corner that mode pooling resolves.

**See also.** `?fit_pigauto` (`phylo_signal_gate`, "Safety floor"),
[design spec](https://github.com/itchyshin/pigauto/blob/main/specs/2026-04-23-safety-floor-mean-gate-design.md).

## 4. "How do I know if my dataset has enough phylogenetic signal?"

**Symptom.** You aren't sure whether phylogenetic information is useful for
your dataset.

**Why this matters.** A lambda estimate describes fitted phylogenetic
structure; it does not determine predictive performance on a new dataset.
Use held-out evaluation rather than a generic threshold to compare a
phylogenetic workflow with an appropriate alternative.

**Diagnose.** The fitted object stores the per-trait λ values used by
`phylo_signal_gate` (they are computed only when the gate is switched on,
which is no longer the default):

```{r section4-diag, eval = FALSE}
library(pigauto)
data(avonet300, tree300)
df <- avonet300; rownames(df) <- df$Species_Key; df$Species_Key <- NULL

fit <- impute(df, tree300, phylo_signal_gate = TRUE)$fit
fit$phylo_signal_per_trait
```

The output reports λ on the observed cells where it can be estimated. Discrete
traits use the package's internal continuous proxy on the latent/liability
scale.

**Fix.** Use the lambda estimate as a diagnostic, then run
`cross_validate()` on the input and compare the candidate workflows under the
same held-out design. If phylogenetic structure is weak, consider whether a
non-phylogenetic analysis is more appropriate for your scientific question.

**See also.** `?fit_pigauto` (`phylo_signal_gate`),
[Phase 8 signal sweep memo](https://github.com/itchyshin/pigauto/blob/main/script/bench_signal_sweep.md).

## 5. "Predictions are way bigger than anything I observed"

**Symptom.** A masked log-transformed continuous trait (body mass,
seed mass, fish weight) predicts a value 50–100× larger than anything
observed. On AVONET, the canonical case is the cassowary: truth ≈ 35 kg,
predicted up to ~540 kg.

**Why this happens.** For log-transformed traits, the GNN's
MC-dropout draws are on the log scale. A latent ~+3-4 σ above the
training distribution survives as a ~50-100× value error after
`expm1()` back-transformation. With `n_imputations = 1`, a single
unlucky dropout pattern can produce this; with `pool_method = "median"`
(default) the median of M draws is robust to one bad draw, but a small
M (≤ 5) on the long tail of the latent distribution can still mis-pool.

**Diagnose.**

```{r section5-diag, eval = FALSE}
# After running impute(), check whether any imputation exceeds the
# observed maximum by an unrealistic factor:
predicted_mass <- result$completed$Mass[result$imputed_mask[, "Mass"]]
obs_max <- max(df$Mass, na.rm = TRUE)
sum(predicted_mass > 5 * obs_max)
```

A non-zero count is a signal of tail extrapolation.

**Fix.** Phase G `clamp_outliers = TRUE` caps post-back-transform
predictions for log-transformed continuous, count, and `zi_count`
magnitude traits at `obs_max * clamp_factor` (default 5). This is
opt-in because for legitimate growth-curve datasets where 5× the
observed maximum is plausible, you don't want it.

```{r section5-fix, eval = FALSE}
result <- impute(df_obs, tree300,
                 clamp_outliers = TRUE,
                 clamp_factor   = 5,    # Tukey-style outlier cap
                 verbose = FALSE)
```

**See also.** `?impute` (`clamp_outliers`, `clamp_factor` arguments),
[AVONET Mass diagnosis memo](https://github.com/itchyshin/pigauto/blob/main/useful/MEMO_2026-05-01_avonet_mass_diag.md),
[Phase G results](https://github.com/itchyshin/pigauto/blob/main/useful/MEMO_2026-05-01_phase_g_results.md).