---
title: "Lag sequential analysis"
author: "Written by [Mohammed Saqr](https://saqr.me) and [Sonsoles López-Pernas](https://sonsoles.me)"
output:
  rmarkdown::html_vignette:
    toc: true
    toc_depth: 2
vignette: >
  %\VignetteIndexEntry{Lag sequential analysis}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r options, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>", fig.width = 6.5,
                      fig.height = 4.5, fig.align = "center", dpi = 72)
old_options <- options(digits = 3, width = 90)
```

Lag sequential analysis (LSA) studies the order in which categorical events
occur (Sackett, 1979; Bakeman & Gottman, 1997). A process is coded as a
sequence of states, such as weekly engagement levels or the moves of a
collaborative discussion, and the analysis asks whether one state is followed
by another more often, or less often, than the base rates of the two states
alone would predict. A transition that occurs more often than expected is a
sequential regularity of the process; a transition that occurs less often is
an avoidance. Neither can be read from frequencies alone: a frequent
transition may only reflect two busy states, and a rare transition may still
occur far more often than chance allows.

The method reduces the data to a lagged contingency table and tests every
cell of it against sequential independence, the model in which the next state
carries no memory of the current one. The result is a set of tested
transitions, each with a direction, a size, and a p-value, together with an
omnibus test of the whole table. `lagdynamics` fits the model in one call,
`lsa()`, and returns every part of it as a tidy data frame.

## Data

`engagement` records the weekly engagement state of 138 students over 15
weeks, one row per student and one column per week.

- **Sequence:** each row is one student's sequence.
- **Order:** the column order (weeks 1 to 15) orders the states.
- **State:** each cell is `Active`, `Average`, or `Disengaged`; `NA` marks
  a missing week and is skipped rather than counted as a state.

Two rows are missing in every week, so the model is fitted to 136 sequences.

```{r setup}
library(lagdynamics)
head(engagement)
```

## The lagged contingency table

Let the process visit $K$ states. At lag $\ell$, every pair of positions
$\ell$ steps apart within a sequence is one transition, and the transitions
are tabulated in a $K \times K$ table whose cell $O_{ij}$ counts the
occasions on which state $i$ was followed, $\ell$ positions later, by state
$j$. At lag 1 this is the table of immediate successors. Pairs never cross
the boundary between two sequences, so the last week of one student and the
first week of the next are not counted.

Two descriptive quantities come directly from the table. The transitional
probability $P(j \mid i) = O_{ij} / R_i$, where $R_i$ is the row total,
is the share of moves out of state $i$ that reach state $j$; it describes
where the process goes. The expected count under independence,

$$E_{ij} = \frac{R_i \, C_j}{N},$$

with $C_j$ the column total and $N$ the number of transitions, is the
count the cell would reach if the next state were drawn from the base rates
regardless of the current one.

## Adjusted residuals and Yule's Q

The adjusted residual standardises the gap between the observed and the
expected count (Haberman, 1973),

$$z_{ij} = \frac{O_{ij} - E_{ij}}
  {\sqrt{E_{ij}\,(1 - R_i/N)\,(1 - C_j/N)}},$$

and is approximately standard normal under independence. A residual above
1.96 marks a transition that is over-represented at the 5% level, and a
residual below -1.96 one that is under-represented. The residual is the
evidential unit of LSA: it states how far a transition departs from chance
on a common scale, whatever the base rates of its states.

The residual is a test statistic, so it grows with the number of
transitions. Yule's Q is the matching effect size. For each cell the table
is collapsed to a 2-by-2 table (from $i$ or not, to $j$ or not) with cells
$a = O_{ij}$, $b$, $c$, and $d$, and

$$Q_{ij} = \frac{ad - bc}{ad + bc},$$

which lies between -1 (the transition never occurs) and 1 (state $i$ is
always followed by state $j$), with 0 at independence. Because it does not
grow with sample size, Q is the scale on which transitions of different data
sets or groups are compared (Bakeman & Quera, 2011).

## Fitting the model

`lsa()` takes the wide data frame directly: rows are sequences and columns
are ordered positions. The default is the classical engine at lag 1.

```{r fit}
fit <- lsa(engagement)
fit
```

The model counts 1734 transitions among 1870 events in 136 sequences. The
omnibus likelihood-ratio test of independence is $G^2 = 618.3$ on 4 degrees
of freedom, so the next week's state depends strongly on the current one.
Seven of the nine transitions depart from independence, and the three
strongest over-represented transitions are the three self-transitions.

## Which transitions depart from chance

`transitions()` returns one row per ordered pair with the observed and
expected counts, the transitional probability (`prob`), the adjusted
residual (`adj_res`), its p-value, Yule's Q, and further association
measures. The argument `significant = TRUE` keeps the transitions whose
p-value is below `alpha`, and `sort = "strength"` orders them by the
absolute residual.

```{r significant}
transitions(fit, significant = TRUE, sort = "strength")
```

`Active -> Active` is the strongest regularity: it occurred 459 times
against 247 expected, an adjusted residual of 21.7 and a Yule's Q of 0.828.
Active students therefore stay active from one week to the next far more
often than the base rate of activity predicts. `Disengaged -> Disengaged`
(z = 15.4, Q = 0.754) and `Average -> Average` (z = 12.5, Q = 0.553) are
the other two persistences, so every state tends to repeat. The avoidances
are as informative. `Active -> Disengaged` occurred 23 times against 121
expected (z = -12.6, Q = -0.826): an active student almost never drops to
disengagement in a single week. `Disengaged -> Active` is also avoided
(z = -10.5, Q = -0.698), so recovery from disengagement to full activity is
rare. Moves between `Average` and `Disengaged`, by contrast, occur at the
rate the base rates predict.

The two transitions missing from this table, `Average -> Disengaged` and
`Disengaged -> Average`, are the ones that occur at the rate chance predicts. `direction = "over"` or `direction = "under"` keeps
only one sign, and `min_count` drops pairs with few observations.

```{r over}
transitions(fit, direction = "under", sort = "strength")
```

Four transitions are significantly avoided, and all four connect `Active`
with another state, in either direction. Active engagement is therefore a
closed state: students who hold it keep it, and students outside it rarely
reach it in one week (`Average -> Active`, z = -12.9).

## Is the table as a whole dependent

`tests()` returns the omnibus tests of independence for the whole table: the
likelihood-ratio statistic (`lrx2`) and Pearson's chi-squared (`x2`).

```{r tests}
tests(fit)
```

Both statistics reject independence ($G^2 = 618$ and $\chi^2 = 629$ on 4
degrees of freedom, p far below 0.001). A non-significant omnibus test would
give no basis for interpreting individual residuals, so this test is read
before the transitions.

## States and starting points

`nodes()` returns one row per state with the number of transitions it sends
and receives, and `initial()` the share of sequences that start in each
state.

```{r nodes}
nodes(fit)
initial(fit)
```

`Average` is the busiest state (751 outgoing, 763 incoming transitions) and
`Disengaged` the least visited (325 and 320). These totals are what the
expected counts control for: `Average` is not credited with regularities on
the strength of its volume. Sequences start in `Active` (0.382) and
`Average` (0.368) more often than in `Disengaged` (0.250).

## A summary of the model

`summary()` returns a summary object. Printed, it gives the model in a few
lines; its tables are read with `as.data.frame()`.

```{r summary}
fit_summary <- summary(fit)
fit_summary
as.data.frame(fit_summary)
as.data.frame(fit_summary, what = "nodes")
```

The overview table is one row with the dimensions and omnibus tests of the
model, the form to collect when several models are fitted. The node table
adds the shares of outgoing and incoming transitions and the initial
probability for each state: `Average` carries 43.3% of the outgoing
transitions and `Disengaged` 18.7%.

## Transition probabilities

`transition_probabilities()` returns the transitional probabilities
$P(\text{to} \mid \text{from})$ as a tidy table, one row per ordered pair.
They are the edge weights of a transition network analysis model and the
input that probability-based tools expect.

```{r probabilities}
transition_probabilities(fit)
```

Each state is most often followed by itself: 0.698 for `Active`, 0.610 for
`Average`, and 0.483 for `Disengaged`. The probabilities and the residuals
answer different questions. `Disengaged -> Average` has a probability of
0.397, the second largest move out of `Disengaged`, yet its residual
(-1.7) is not significant: the move is frequent because `Average` is a busy
state, not because disengaged students are drawn to it.

## Residual heatmap

`plot()` draws a residual heatmap by default. Rows are the current states,
columns the next states, and the fill is the adjusted residual on a
diverging scale.

```{r heatmap}
plot(fit)
```

The heatmap shows the structure of the model at a glance: the diagonal is
uniformly over-represented, the four off-diagonal cells in the `Active` row
and column are avoided, and the two cells linking `Average` and
`Disengaged` sit near zero. The process is one of persistence, with
movement mostly confined to the two lower states.

## Where next

- [Lag transition networks](https://pak.dynasite.org/lagdynamics/articles/lag-transition-networks.html):
  the fitted model drawn as a residual, Yule's Q, or probability network.
- [Evaluating transitions](https://pak.dynasite.org/lagdynamics/articles/confirmatory.html):
  edge uncertainty, case-drop stability, split-half reliability, and a
  permutation null.
- [Comparing groups](https://pak.dynasite.org/lagdynamics/articles/group-comparison.html):
  grouped fits and permutation and Bayesian tests of group differences.
- [Workflow](https://pak.dynasite.org/lagdynamics/articles/workflow.html),
  [event logs](https://pak.dynasite.org/lagdynamics/articles/event-logs.html),
  [engines](https://pak.dynasite.org/lagdynamics/articles/engines.html),
  [lags and structural zeros](https://pak.dynasite.org/lagdynamics/articles/lags-structural-zeros.html),
  and [plotting](https://pak.dynasite.org/lagdynamics/articles/plotting.html):
  the guides to each part of the package.

## References

Bakeman, R., & Gottman, J. M. (1997). *Observing interaction: An
introduction to sequential analysis* (2nd ed.). Cambridge University Press.

Bakeman, R., & Quera, V. (2011). *Sequential analysis and observational
methods for the behavioral sciences*. Cambridge University Press.

Haberman, S. J. (1973). The analysis of residuals in cross-classified tables.
*Biometrics, 29*, 205-220.

Sackett, G. P. (1979). The lag sequential analysis of contingency and
cyclicity in behavioral interaction research. In J. D. Osofsky (Ed.),
*Handbook of infant development* (pp. 623-649). Wiley.

```{r cleanup, include = FALSE}
options(old_options)
```
