Goal: Stand up a versioned datom project using a
local filesystem store and onboard data using datom_sync()
– the primary workflow for bringing files into datom. By the end of this
article the project holds multiple versioned tables, and you have seen:
syncing one file, updating it, the idempotent no-op, and syncing a
batch. No AWS account needed. The same workflow extends unchanged to S3
(see Starting on S3).
Already done the two-minute tour in the README? The tour and this article cover the same ground. If your project initialized cleanly and
datom_read()returnedTRUE, you can jump straight to Starting on S3 or Tracing Data Lineage.
You are the data engineer for STUDY-001, a Phase II clinical trial. EDC extracts land as files in a staging folder; your job is to onboard them into a shared, versioned data space – multiple engineers writing new extracts, multiple analysts reading any version, all coordinated through a single git history. Every sync is a git commit; every read resolves to an exact version SHA. No one can silently overwrite history, and anyone with access to the repo can reproduce any past analysis by pinning to a SHA.
This first article walks the local-only path. The same workflow – same functions, same commands – works for a shared S3 space: you build an S3 store instead of a local one (see Starting on S3).
Two locations, two roles:
datom_sync() onboards a file, storage holds the versioned,
content-addressed copy. The original input file is no longer needed; the
onboarded data stands on its own.datom keeps metadata in git (diff-able, auditable) and data wherever you tell it to live (S3 or a local directory). Even when data lives on a local filesystem, metadata still goes to a git remote – that is how version history stays reproducible across machines.
You need two things, both one-time:
repo. Store it in your OS keychain once with
keyring::key_set("GITHUB_PAT"); every article after this
picks it up automatically. Using keyring keeps the PAT out
of your code and command history.gh CLI is not required – datom
creates GitHub repos through the GitHub REST API directly using your
PAT.No AWS, no cloud account, no governance repo for this article.
Everything machine-specific lives in one block. Set these values once for your environment; every code chunk after this is copy/paste as-is.
library(datom)
library(fs)
# --- Settings you control --------------------------------------------------
project_name <- "STUDY_001" # logical project name (recorded in metadata)
repo_name <- "study-001-data" # GitHub repo name for the metadata repo
# dev_dir -- your local clone of the metadata git repo (stays on your machine)
# data_dir -- where parquet bytes are written; point at a shared location
# (network mount, or an S3 store) for a real team space. Temp dirs
# are used here so the article leaves nothing behind.
dev_dir <- path(tempdir(), "study_001_dev")
data_dir <- path(tempdir(), "study_001_data")
# Your GitHub personal access token (PAT), scoped to `repo`. Stored once in your
# OS keychain with keyring::key_set("GITHUB_PAT"); read here by name so the
# token never appears in your code or command history.
github_pat <- keyring::key_get("GITHUB_PAT")
# ---------------------------------------------------------------------------
dir_create(data_dir)A store bundles the addresses datom needs: where
parquet bytes go and the GitHub PAT that lets datom push metadata.
Governance is not attached (governance = NULL); it is
opt-in and added later through the governance companion package when
sharing across a portfolio matters.
datom_init_repo(
path = dev_dir,
project_name = project_name,
store = store,
create_repo = TRUE,
repo_name = repo_name
)This creates a GitHub repo, clones it into dev_dir, and
commits a project.yaml that records the project’s data
store address. The git repo is now live on GitHub. No parquet data is
pushed to GitHub – only the metadata commits travel over the wire; the
parquet bytes stay in data_dir.
datom_init_repo() also creates an
input_files/ directory inside the clone. It is gitignored –
files placed there are never committed. It is the inbox for
datom_sync().
datom_sync() onboards flat tabular files:
csv, tsv, txt, psv,
parquet, sas7bdat, xpt,
sav, zsav, por, dta,
xls, xlsx. Anything else in the inbox (an
.rds, a .json) is reported as
unsupported_format and skipped without blocking its
neighbours – see the ingestion
allowlist for why, and for the one-line escape hatch.
Take a moment to inspect the repo structure before moving on:
The month-1 extract has just landed – a single demographics CSV. Drop it in the input folder:
# The input folder lives inside the git clone but is gitignored.
# Files placed here are the raw material for datom_sync().
input_dir <- path(dev_dir, "input_files")
write.csv(
datom_example_data("dm", cutoff_date = "2026-01-28"),
path(input_dir, "dm.csv"),
row.names = FALSE
)Scan the folder to build a sync manifest – datom detects what is new, changed, or unchanged relative to what has already been onboarded:
manifest <- datom_sync_manifest(conn)
#> i Scanned 1 file: 1 new, 0 changed, 0 unchanged.
manifest
#> name file format original_file_sha status
#> 1 dm .../dm.csv csv 7a3b... newNow onboard it:
datom_sync(conn, manifest)
#> i Syncing 1 table...
#> v dm synced (new).
#> i Sync complete: 1 succeeded, 0 failed, 0 skipped.Three things just happened, in this order:
data_dir.
No data was pushed to the GitHub repo – parquet bytes
never leave your local store.metadata.json and version_history.json
were updated in the git clone and committed.Confirm with datom_list(), datom_history(),
and a round-trip read:
datom_list(conn)
#> name current_version current_data_sha last_updated
#> 1 dm a8ee7a31 4b6d0a7e 2026-01-28T...
datom_history(conn, "dm")
#> version data_sha timestamp message
#> 1 a8ee7a31 4b6d0a7e 2026-01-28T09:02:11Z dm synced from dm.csv
datom_read(conn, "dm")Key point: the input file is now disposable. Delete it and the data is still accessible from storage – storage is the source of truth, not the input folder:
A month passes. The month-2 extract arrives with new subjects. Overwrite the input file and sync again:
write.csv(
datom_example_data("dm", cutoff_date = "2026-02-28"),
path(input_dir, "dm.csv"),
row.names = FALSE
)
manifest <- datom_sync_manifest(conn)
#> i Scanned 1 file: 0 new, 1 changed, 0 unchanged.
datom_sync(conn, manifest)
#> i Syncing 1 table...
#> v dm synced (changed).
#> i Sync complete: 1 succeeded, 0 failed, 0 skipped.The table now has two versions. The previous version is still intact – nothing was overwritten:
datom_history(conn, "dm")
#> version data_sha timestamp message
#> 1 5c1a3f7b 9e8f1c2d 2026-02-28T10:14:02Z dm synced from dm.csv
#> 2 a8ee7a31 4b6d0a7e 2026-01-28T09:02:11Z dm synced from dm.csv
# Read the current version (month 2)
nrow(datom_read(conn, "dm"))
#> [1] 16
# Read the prior version (month 1) by its SHA
hist <- datom_history(conn, "dm")
m1_ver <- hist$version[nrow(hist)] # oldest row is the month-1 version
nrow(datom_read(conn, "dm", version = m1_ver))
#> [1] 4Both versions coexist; any historical snapshot is retrievable by its SHA. This is the property that downstream statisticians, regulators, and auditors rely on.
Run datom_sync() again with nothing changed:
manifest <- datom_sync_manifest(conn)
#> i Scanned 1 file: 0 new, 0 changed, 1 unchanged.
datom_sync(conn, manifest)
#> i No new or changed files. Nothing to sync.Re-syncing identical content is a no-op. This is what makes sync safe
to run in a scheduled job or pipeline – accidental re-runs cost nothing
and pollute no history. The same idempotency applies to
datom_write(): writing an identical data frame to the same
table name detects the duplicate and skips.
Idempotency here is stronger than “the file’s bytes are unchanged”. If a source system re-exports the same data – new export timestamp in the header, same rows – the file’s bytes change but its content does not. datom records the new file provenance as a version and does not store the data a second time. Version SHAs explains the three hashes that make that distinction.
It is now month 3. The data management team has switched from emailing single files to dropping a folder of monthly extracts. Today’s drop contains four domains:
cutoff <- "2026-03-28"
write.csv(datom_example_data("dm", cutoff_date = cutoff),
path(input_dir, "dm.csv"), row.names = FALSE)
write.csv(datom_example_data("ex", cutoff_date = cutoff),
path(input_dir, "ex.csv"), row.names = FALSE)
write.csv(datom_example_data("lb", cutoff_date = cutoff),
path(input_dir, "lb.csv"), row.names = FALSE)
write.csv(datom_example_data("ae", cutoff_date = cutoff),
path(input_dir, "ae.csv"), row.names = FALSE)Scan and sync in one pass:
manifest <- datom_sync_manifest(conn)
#> i Scanned 4 files: 3 new, 1 changed, 0 unchanged.
manifest
#> name file format original_file_sha status
#> 1 dm .../dm.csv csv c41a... changed
#> 2 ex .../ex.csv csv 9b08... new
#> 3 lb .../lb.csv csv 72d3... new
#> 4 ae .../ae.csv csv 1e4a... new
datom_sync(conn, manifest)
#> i Syncing 4 tables...
#> v dm synced (changed).
#> v ex synced (new).
#> v lb synced (new).
#> v ae synced (new).
#> i Sync complete: 4 succeeded, 0 failed, 0 skipped.dm is changed (month-3 update);
ex, lb, ae are brand new. All
four are now versioned:
datom_status() summarizes the project state at a
glance:
datom_status(conn)
#> -- datom status: STUDY_001
#> v Git: clean, in sync with origin
#> i Tables on local: 4
#> i Last commit: <sha> "Update ae"datom_validate() cross-checks that every table in the
manifest has its parquet file in the data store and its metadata in git
history:
datom_validate(conn)
#> v 4 tables validated.
#> v Manifest <-> data store: consistent.
#> v Manifest <-> git history: consistent.If you ever see a discrepancy from datom_validate(),
that is the moment to stop and investigate before doing more work – it
means the project’s state has drifted from one of git or storage.
Writing is a developer action. Reading is what analysts and
downstream pipelines do, and they use the reader role –
a connection built from the data store alone, with no GitHub PAT and no
local clone. You build a reader store by leaving github_pat
unset, then connect by project name:
reader_store <- datom_store(
governance = NULL,
data = datom_store_local(path = data_dir) # same data location
) # no PAT -> reader role
reader_conn <- datom_get_conn(store = reader_store, project_name = project_name)
print(reader_conn)
#> -- datom connection
#> * Project: "STUDY_001"
#> * Role: "reader"
#> * Backend: "local"Read any table through the reader connection:
datom stores data in Apache
Parquet format and reads it back as a tibble. The read
does not go through GitHub – the reader resolves the
parquet file directly from the data store. A teammate on another machine
takes the same path: they need access to the data store, not to the git
repo.
datom_write()datom_sync() is the primary workflow for onboarding
files from a staging folder. For tables derived in code – where the data
frame is computed rather than imported from a file – use
datom_write() directly:
# Derive a summary table in code
lb <- datom_read(conn, "lb")
lb_summary <- dplyr::summarise(
dplyr::group_by(lb, LBTESTCD),
n = dplyr::n(),
.groups = "drop"
)
# Write it as a versioned table, declaring its parent for lineage.
# datom_parent() resolves the parent's data_sha and source_lineage from
# the stored metadata snapshot -- always use it instead of a raw list.
datom_write(
conn,
data = lb_summary,
name = "lb_summary",
message = "Lab test counts from month-3 LB",
parents = list(
datom_parent(conn, table = "lb", version = datom_history(conn, "lb")$version[1])
)
)
#> v Wrote "lb_summary" (full): "b9c4e21a"datom_write() takes the same connection and produces the
same versioned, content-addressed result. The parents
argument declares lineage – this table was derived from lb
– making the provenance chain auditable (see Tracing Data Lineage).
A table version is identified by a canonical hash of its
values (data_sha), which means every
column has to be a type datom knows how to encode: logical,
integer, double, character,
factor, Date, POSIXct,
difftime/hms,
data.table::ITime/IDate,
bit64::integer64, or a labelled vector over one of
those.
Files onboarded through datom_sync() land inside that
set by construction. It is derived tables – the
datom_write() path above – where a column can fall outside
it: a tidyr::nest() result carries a list column,
strptime() returns POSIXlt, an sf
table carries geometry, a database round-trip can bring back a blob.
datom_check_hashable() tells you before you write. It
needs no connection and no store, so it is safe to run anywhere:
library(datom)
lb_summary <- data.frame(LBTESTCD = c("ALT", "AST"), n = c(12L, 12L))
datom_check_hashable(lb_summary)
#> ✔ All 2 columns are hashable. This table is ready for `datom_write()`.When a column is outside the contract, the report names it and says what to do about it:
messy <- data.frame(id = 1:2)
messy$measurements <- list(c(1, 2), c(3, 4))
report <- datom_check_hashable(messy)
#> ✖ 1 of 2 columns are not hashable.
#> ✖ Column measurements (<list>): List and blob columns are not hashable. Flatten
#> to one value per row with tidyr::unnest(), or serialize each element to
#> character (for example with jsonlite::toJSON() per element), before writing.
#> ℹ Fix these before `datom_write()`, which refuses the whole table until they are resolved.
report$recourse[report$status == "unsupported"]
#> [1] "List and blob columns are not hashable. Flatten to one value per row with tidyr::unnest(), or serialize each element to character (for example with jsonlite::toJSON() per element), before writing."datom_write() runs the same check itself and refuses the
whole table until every column is inside the contract, reporting all
offenders at once and leaving no partial state behind. The full
type-by-type recourse table and the reasoning live in Version SHAs.
datom_status() and datom_validate()
provide consistency checks.datom_check_hashable() is the pre-flight check for the
table contract.From here, Starting on S3 shows the
same workflow with data in object storage, Tracing Data Lineage shows how derived
tables record where they came from, and Version SHAs explains the three
hashes behind version, data_sha, and
stored-object integrity.
When done, remove the project. Pick the one option that fits, and copy only that chunk:
# Option A -- full scripted teardown (deletes local files AND the GitHub repo).
# Do this BEFORE any manual unlink().
datom_repo_delete(conn, confirm = "STUDY_001")# Option B -- local only (the GitHub repo stays; delete it from the UI later).
unlink(c(dev_dir, data_dir), recursive = TRUE)Do not call unlink() before
datom_repo_delete() – removing the local clone first strips
the GitHub remote reference and the remote repo will not be deleted.