--- title: "The datom Model: Code in Git, Data in Cloud" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{The datom Model: Code in Git, Data in Cloud} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r, include = FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>", eval = FALSE ) ``` > **Companion to**: [Getting Started](getting-started.html). Read this when you > want to understand *why* datom split storage in two. The single design decision that shapes everything else in datom is this: **metadata and data live in different places, and that's intentional.** This article explains the split, the consequences that fall out of it, and the properties you get for free when you accept it. ## The split ``` +------------------------+ +-------------------------+ | Git repository | | Object store | | | | | | manifest.json | | dm/ | | metadata.json | | a8ee7a31.parquet | | version_history.json | ----> | 5c1a3f7b.parquet | | project.yaml | refers | ex/ | | | to | f44910b5.parquet | | (everything text, | | | | diffable, reviewable)| | (parquet bytes only, | | | | keyed by SHA) | +------------------------+ +-------------------------+ ``` **Metadata** -- the catalog of what tables exist, what their current and historical versions are, and where the bytes live -- goes in a git repository. It's small, text-based, diffable, reviewable, and protected by all the access controls and audit machinery your organization already has around git. **Data** -- the parquet bytes themselves -- goes in an object store. Each parquet file is named by a SHA of its content. The git repo never holds data; it only holds *pointers* to data. This is not a storage optimization. It's an information-architecture choice with downstream consequences. ## Property 1: Immutability is automatic Parquet files are addressed by **SHA of their content**. Two consequences fall out: - **Writing the same data twice produces the same filename.** The second upload is either a no-op or a harmless overwrite of identical bytes. - **Different data produces a different filename.** A new version never overwrites an old one; both files coexist in the store. You don't have to remember to copy old files before overwriting them. The file *system* makes overwriting impossible by construction. The history your `datom_history()` call shows is real -- those parquet files are still in the store. ## Property 2: The metadata is reviewable like code Because metadata lives in git, every change to a project's catalog is a commit. That means: - `git log` shows the history of every table. - `git blame` tells you which `datom_write()` introduced a specific version. - Pull requests, branch protection, and code review apply to data changes the same way they apply to code changes. - Restoring a project to "the way it was on March 15" is `git checkout`, no special tooling. If you've ever debugged a missing column by reading commit messages, you already understand why this matters. ## Property 3: Readers don't need write access The split lets datom support two roles cleanly: - **Data developers** have a local git clone of the project, push to the remote, and write to the data store. - **Data readers** connect to the data store directly -- no git clone, no write credentials required. `datom_get_conn(store = ..., project_name = "...")` (reader form, no `path`) resolves everything it needs from the metadata mirrored to the data store at write time: the manifest, per-table `metadata.json`, and versioned snapshots. The data store can be an object store (S3) or a local filesystem store (`datom_store_local()`); the reader path is the same either way. What readers never need is access to the git repository (the code store) or write credentials for the data store. A statistician with read-only access to the data store can read any table and reproduce any historical analysis -- the metadata they need is already there, mirrored when the developer wrote. ## Property 4: Storage is swappable Because datom only knows the *address* of data through the metadata, not the bytes themselves, the storage backend is a substitution point. `datom_store_local()` puts parquet on a directory; `datom_store_s3()` puts parquet on S3; future backends (GCS, Azure Blob, etc.) drop in without touching how versions are computed or how history is recorded. What does *not* swap is git. The metadata still goes to a git remote, always. ## What stays in the metadata A datom project's git repository holds, per table: - `metadata.json` -- the current state (current data SHA, table type, size, parents). - `version_history.json` -- an append-only log of every version ever written. - A `manifest.json` at the project root summarizing all tables. It does **not** hold the data, and it never will. The `.gitignore` in a new project explicitly excludes parquet, csv, and other data formats so nobody can commit them by accident. ## What this isn't datom is not a content-addressed *blob store* like git-LFS or DVC. The key difference: datom's metadata is a structured, queryable catalog -- not opaque pointer files in your repo. You ask `datom_history(conn, "dm")` and you get a data frame, not a `git log` of `.dvc` files. The metadata is designed to be read by code, not just by humans. datom is also not a database. It has no query engine, no ACID transactions across tables, no joins. It is a content-addressed catalog of immutable parquet files, and it's deliberately not more than that. ## Where this leads Once you accept "metadata in git, data in data store," several other datom design choices stop looking arbitrary: - The two-repo split (governance vs. project) -- the organization-scope governance repo is managed by the governance companion package. - The `ref.json` indirection layer for portable, migration-ready storage. - The two-flavor SHA scheme (data SHA + metadata SHA) -- see [Version SHAs: Data SHA vs. Metadata SHA](design-version-shas.html). Each is a direct consequence of the split, applied to a specific problem. None of them stand on their own.