Tracing Data Lineage

datom records two complementary lineage fields in every table’s metadata:

Together they answer two different audit questions without walking a DAG:

Question Field Tool
“What exact versions did my script read?” parents datom_get_lineage(depth = "parents")
“What raw data is ultimately in this table?” source_lineage datom_get_lineage(depth = "source")

This article shows how lineage is recorded, how to query it, and how to recompute it from existing reads to check consistency.

Setup

The walkthrough below is illustrative (eval = FALSE) – it shows the calls without needing live credentials. As in the other articles, keep machine-specific values up front:

library(datom)

project_name <- "my-study"
bucket       <- "my-datom-bucket"
prefix       <- "lineage-demo"
region       <- "us-east-1"
access_key   <- keyring::key_get("AWS_ACCESS_KEY_ID")
secret_key   <- keyring::key_get("AWS_SECRET_ACCESS_KEY")
github_pat   <- keyring::key_get("GITHUB_PAT")

Writing derived tables (below) needs a developer connection. The lineage queries – datom_get_parents() and datom_get_lineage() – are reads, and the consistency recipe near the end is composed entirely of those reads plus datom_lineage_union(), so it works just as well from a reader connection (build the same store with github_pat omitted). This setup assumes the project already exists:

store <- datom_store(
  governance = NULL,
  data       = datom_store_s3(
    bucket     = bucket,
    prefix     = prefix,
    region     = region,
    access_key = access_key,
    secret_key = secret_key
  ),
  github_pat = github_pat            # omit for a read-only reader connection
)

conn <- datom_get_conn(path = "path/to/my-study-dev", store = store)

A simple study: raw imports to derived analysis

Imagine a clinical study with two raw source tables:

raw_dm <- datom_example_data("dm")      # Demographics
raw_lb <- datom_example_data("lb")      # Lab results

They are onboarded via datom_sync(), the file-based workflow shown in the Getting Started article: stage the raw tables as files in the clone’s gitignored input_files/ inbox, scan them into a manifest with datom_sync_manifest(), then datom_sync(). datom automatically records each imported table’s own content SHA as its source_lineage – a single self-entry.

# input_files/ lives inside the git clone and is the sync inbox.
input_dir <- file.path("path/to/my-study-dev", "input_files")

write.csv(raw_dm, file.path(input_dir, "dm.csv"), row.names = FALSE)
write.csv(raw_lb, file.path(input_dir, "lb.csv"), row.names = FALSE)

manifest <- datom_sync_manifest(conn)   # scans input_files/
datom_sync(conn, manifest)              # onboards; self-lineage recorded

After syncing, the imported tables carry their own lineage:

datom_get_lineage(conn, "dm", depth = "source")
#> [[1]]
#> [[1]]$project
#> [1] "my-study"
#>
#> [[1]]$table
#> [1] "dm"
#>
#> [[1]]$version_sha
#> [1] "abc123..."

The self-entry means: “the raw content of dm is itself a source.”

Deriving a table

A downstream script reads both raw tables and produces a cleaned demographics table:

# Read the current versions
raw_dm_data <- datom_read(conn, "dm")
raw_lb_data <- datom_read(conn, "lb")

# ... cleaning logic ...
dm_clean <- raw_dm_data  # simplified

# Retrieve the metadata shas to identify exactly which versions were read
dm_version  <- datom_history(conn, "dm")[1, "version"]
lb_version  <- datom_history(conn, "lb")[1, "version"]

When writing the derived table, declare each parent with datom_parent(). Each record reads the parent’s authoritative data_sha and its source_lineage from the parent’s own versioned snapshot. datom_write() then derives the derived table’s source_lineage itself, as the deduplicated union of the parents’ lineages – there is no public source_lineage argument to supply or keep in sync:

datom_write(
  conn,
  data    = dm_clean,
  name    = "dm_clean",
  parents = list(
    datom_parent(conn, "dm", dm_version),
    datom_parent(conn, "lb", lb_version)
  )
)

Now dm_clean knows which raw tables it came from:

datom_get_lineage(conn, "dm_clean", depth = "source")
#> [[1]]
#> $project
#> [1] "my-study"
#> $table
#> [1] "dm"
#> $version_sha
#> [1] "abc123..."
#>
#> [[2]]
#> $project
#> [1] "my-study"
#> $table
#> [1] "lb"
#> $version_sha
#> [1] "def456..."

And which immediate versions it was derived from:

datom_get_lineage(conn, "dm_clean", depth = "parents")
#> [[1]]
#> $source
#> [1] "my-study"
#> $table
#> [1] "dm"
#> $version
#> [1] "..."   # metadata_sha of dm at derivation time

Propagating lineage further downstream

An analysis table derived from dm_clean propagates the lineage automatically. Because dm_clean already encodes both dm and lb, datom_parent(conn, "dm_clean", clean_version) captures that transitive lineage, and datom_write() unions it into analysis_pop without re-reading the original files:

# Identify the dm_clean version this analysis was derived from.
clean_version <- datom_history(conn, "dm_clean")[1, "version"]

datom_write(
  conn,
  data    = analysis_pop,
  name    = "analysis_pop",
  parents = list(
    datom_parent(conn, "dm_clean", clean_version)
  )
)

Recomputing lineage consistency

datom_write() derives a derived table’s source_lineage from the union of its parents’ lineages at write time, and lineage is version-pinned. So a recompute equals the recorded value in normal operation – a difference flags drift or corruption worth investigating. There is no dedicated validator; instead you compose the existing reads with datom_lineage_union().

The recipe has four steps:

# 1. Read the derived table's recorded parents. Each entry carries
#    source, table, version, and data_sha -- enough to pick the parent's
#    project connection and its pinned version.
parents <- datom_get_parents(conn, "dm_clean")

# 2. Read each parent's source_lineage through a connection scoped to that
#    parent's project. For same-project parents this is the same `conn`.
parent_lineages <- lapply(parents, function(p) {
  datom_get_lineage(conn, p$table, version = p$version, depth = "source")
})

# 3. Union the parents' lineages (dedup by {project, table, version_sha}).
recomputed <- datom_lineage_union(parent_lineages)

# 4. Compare against the derived table's recorded source_lineage.
recorded <- datom_get_lineage(conn, "dm_clean", depth = "source")
identical(recomputed, recorded)
#> [1] TRUE

Because the recipe reads each parent through its own connection, it extends to cross-project parents without any change: open a connection scoped to each parent’s project (p$source) and read that parent through it. No single connection is ever expected to reach across project stores.

# Cross-project variant: resolve a connection per parent project.
parent_lineages <- lapply(parents, function(p) {
  parent_conn <- conn_for_project(p$source)   # your connection resolver
  datom_get_lineage(parent_conn, p$table, version = p$version,
                    depth = "source")
})
recomputed <- datom_lineage_union(parent_lineages)

Key points