---
title: "Pre-Linkage Data Quality: preflight(), check_medicare(), and flock()"
subtitle: "Preparing your data before murmuration()"
author: "Dr Nicolas Smoll, SCPHU, Sunshine Coast Hospital and Health Service"
date: "`r Sys.Date()`"
output:
  html_document:
    toc: true
    toc_depth: 3
    toc_float: true
    theme: flatly
  pdf_document:
    toc: true
    toc_depth: 3
vignette: >
  %\VignetteIndexEntry{Pre-Linkage Data Quality}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment  = "#>",
  message  = FALSE,
  warning  = FALSE
)
```

## Why pre-linkage quality matters

The Fellegi-Sunter EM algorithm scores pairs by comparing field values. Poor
data quality in any of those fields degrades the score distribution in ways that
no threshold choice can fully recover:

- **High missingness** in a comparison variable reduces its discriminating power,
  flattening the valley between match and non-match clusters.
- **Invalid Medicare numbers** that happen to match a wrong AIR record add
  spurious positive weight to false pairs.
- **Name typos** are expected (the algorithm handles them via fuzzy string
  comparison) but systematic truncation or encoding differences are not.
- **Duplicate IDs** in the primary dataset produce multiple output rows per
  original record, inflating counts and biasing VE estimates.

`starling` provides three functions specifically for catching these issues before
`murmuration()` is called.

---

## `preflight()` — structured pre-linkage audit

`preflight()` runs a battery of checks across both datasets and returns a
structured report. It is the recommended first step before any linkage call.

```{r preflight}
library(starling)
data(cases_notifiable)
data(vax_air)

audit <- preflight(
  data1        = cases_notifiable,
  data2        = vax_air,
  linkage_vars = c("lettername1", "lettername2", "dob", "medicare10"),
  id_col1      = "id_var",
  id_col2      = "id_var",
  date_cols    = c("dob", "onset_date"),
  medicare_col = "medicare10",
  verbose      = TRUE
)
```

The `audit` list contains structured results for programmatic access:

```{r audit-structure}
# Completeness table
head(audit$completeness)

# Duplicate ID counts
audit$duplicates

# Flags raised (empty = all clear)
audit$flags
```

---

## `check_medicare()` — Modulus 10 checksum validation

Australian Medicare numbers contain a check digit at position 9 calculated
from positions 1–8 using the weights 1, 3, 7, 9, 1, 3, 7, 9. 
`check_medicare()` verifies this and returns a three-state flag:

| Value | Meaning |
|---|---|
| `1L` | Checksum passes — number is internally consistent |
| `0L` | Checksum fails — at least one digit is wrong |
| `NA` | Missing, blank, or non-numeric — not verifiable |

```{r medicare}
cases_checked <- check_medicare(
  cases_notifiable,
  medicare_col = "medicare10",
  output_col   = "medicare_valid",
  verbose      = TRUE
)

# Distribution of flags
table(cases_checked$medicare_valid, useNA = "always")
```

### What to do with invalid Medicare numbers

Do not discard the record — it may still link correctly on name and DOB. Instead,
replace the invalid number with `NA` before linkage so it does not contribute
negatively to the score:

```{r fix-medicare}
cases_checked$medicare10 <- ifelse(
  cases_checked$medicare_valid == 1L,
  cases_checked$medicare10,
  NA_character_
)
```

---

## `flock()` — blocking variable construction

`flock()` creates one to three blocking keys from demographic fields. Blocking
restricts `murmuration()` to comparing only pairs that share a blocking key,
making the search tractable for large datasets without sacrificing recall on
the key variables.

```{r flock}
# Single-field block (gender only — broadest, lowest specificity)
cases_blocked <- flock(
  cases_checked,
  block1_vars    = "gender",         # block1: gender only
  block2_vars    = "gender",         # block2: same here; use composite in production
  block3_vars    = "postcode",       # block3: postcode (finer)
  birth_year_col = "dob"             # derives birth_year column for composite use
)

vax_blocked <- flock(
  vax_air,
  block1_vars    = "gender",
  block3_vars    = "postcode",
  birth_year_col = "dob"
)

# Inspect blocking key distributions
table(cases_blocked$block1)
head(sort(table(cases_blocked$block3), decreasing = TRUE))
```

### Choosing blocking variables for Australian datasets

| Dataset size | Recommended block | Rationale |
|---|---|---|
| < 10 000 records | `gender` | Broadest; halves candidate space immediately |
| 10 000–500 000 | `gender` + `birth_year` composite | Balances sensitivity and specificity |
| > 500 000 | `postcode` + `birth_year` | Fine-grained; requires high postcode completeness |

For maximum recall, run `murmuration()` twice — once with a broad block, once
with a fine block — and union the results:

```{r multi-pass, eval = FALSE}
linked_broad <- murmuration(cases_blocked, vax_blocked,
  blocking_var = "block1", ...)
linked_fine  <- murmuration(cases_blocked, vax_blocked,
  blocking_var = "block3", ...)

# Union, keeping the highest-scoring link per case
linked_all <- dplyr::bind_rows(linked_broad, linked_fine) |>
  dplyr::group_by(id_var.x) |>
  dplyr::slice_max(weights, n = 1, with_ties = FALSE) |>
  dplyr::ungroup()
```

---

## Putting it together

```{r summary, eval = FALSE}
library(starling)
data(cases_notifiable); data(vax_air)

# 1. Audit
preflight(cases_notifiable, vax_air,
          linkage_vars = c("lettername1", "lettername2", "dob", "medicare10"),
          medicare_col = "medicare10")

# 2. Fix Medicare
cases <- check_medicare(cases_notifiable)
cases$medicare10 <- ifelse(cases$medicare_valid == 1L,
                            cases$medicare10, NA_character_)

# 3. Block
cases <- flock(cases, block1_vars = "gender", birth_year_col = "dob")
vax   <- flock(vax_air, block1_vars = "gender", birth_year_col = "dob")

# 4. Link
linked <- murmuration(cases, vax,
  linkage_type    = "v2c",
  event_date      = "onset_date",
  id_var          = "id_var",
  blocking_var    = "block1",
  compare_vars    = c("lettername1", "lettername2", "dob", "medicare10"),
  threshold_value = 17)
```

---

## Session information

```{r session}
sessionInfo()
```