Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
145 lines (109 loc) · 4.84 KB

ARCHITECTURE.md

File metadata and controls

145 lines (109 loc) · 4.84 KB
Raw

GitDB Architecture

GitDB treats a GitHub repository as database storage. Application code talks to GitDB tables and SQL; GitDB persists committed state into repository files and advances Git history with non-force ref updates.

The design is intentionally small: no DB server, no ORM compatibility layer, no wire protocol service. The repository is the durable store.

Runtime Layers

  1. App API

    • GitDb.open
    • openGitDb
    • defineTable
    • insert, insertMany, upsertMany, select, first, deleteWhere
    • raw SQL through execute and query

  2. Client runtime

    • Local SQL execution after open/replay
    • Serialized mutation queue
    • Isolated transaction copies
    • Rebuildable equality indexes

  3. Store contract

    • Manifest read/write
    • Mutation append/replay
    • Additive batch commit boundary
    • Optional local snapshots, locks, compaction, and tree commits

  4. GitHub database repository

    • Plaintext manifest/log files for reviewable repositories
    • Encrypted manifest/log files for opaque repositories
    • Git commits as durable history

Commit Boundary

The manifest is the committed boundary. A mutation becomes database state only when the manifest references its segment. Unreferenced files may exist after a crash or failed commit, but they are not replayed.

For GitHub stores, a batch write creates a tree containing the new log segments and manifest, creates a commit, and updates the branch ref with force: false. If the branch moved, GitDB retries from the latest ref.

read ref -> read base commit -> create tree -> create commit -> update ref

Browser Store

The browser entrypoint avoids Node-only imports. It uses:

  • fetch for GitHub REST calls
  • GitHub contents API for reads
  • GitHub Git Database API for tree commits
  • Web Crypto AES-GCM for encrypted mode

This lets extension service workers, static apps, and other serverless clients use GitHub as the backing database. The client still needs a user-scoped GitHub token or OAuth flow with repository contents access.

Recovery

On open:

  1. Read the manifest from the configured store.
  2. Read a visible snapshot only if the store supports it.
  3. Use the snapshot only when its checkpoint sequence equals the manifest sequence.
  4. Otherwise replay manifest-listed mutation segments in order.
  5. Rebuild equality indexes from recovered rows.

This is why stale page files, stale indexes.json, and orphan mutation files do not corrupt query results.

Guarantee Matrix

Area What GitDB guarantees What it does not claim
GitHub storage Manifest and log files are committed into a repository through Git history. Low-latency server database behavior
Transactions Successful mutations advance one manifest sequence boundary. Failed mutations restore previous committed state. Cross-device distributed transactions
Concurrency One opened runtime serializes its own writes; GitHub ref conflicts trigger retry. Mature multi-writer OLTP coordination
Durability Manifest-listed mutation logs replay on open. Matching snapshots can replace replay. Trusting uncheckpointed snapshot files
Indexes Equality indexes rebuild from committed rows and accelerate safe equality lookups. Full SQL optimizer or range indexes
Compaction Local plaintext compaction writes checkpoint first, manifest second, deletes old logs last. Browser or encrypted compaction

Plaintext Layout

gitdb/v1/
  manifest.json
  log/
    00000000000000000001.json

Node plaintext stores may also write visible review snapshots:

gitdb/v1/
  snapshot.json
  todos/
    schema.json
    pages.json
    pages/
      000000.json
    indexes.json

indexes.json is derived data. It is useful for inspection and future tooling, but the runtime does not trust it over committed rows.

Encrypted Layout

gitdb/v1/
  manifest.enc
  log/
    00000000000000000001.enc

Node encrypted stores use the Node crypto implementation. Browser encrypted stores use Web Crypto while preserving the same packed AES-GCM payload shape.

Compaction

GitDbEngine.compact() currently targets local plaintext stores:

  1. Write a visible page snapshot at the current manifest sequence.
  2. Advance the manifest to an empty logSegments list.
  3. Delete obsolete mutation segment files.

Crash behavior:

  • Before checkpoint: old manifest and old logs remain usable.
  • After checkpoint before manifest: old manifest can still reopen safely.
  • After manifest before deletion: new manifest reopens from the checkpoint while old logs are harmless.

Limits

GitDB is a good fit when slow GitHub writes are acceptable and a repository is the desired durable record. It is not a replacement for Postgres, SQLite, or a hosted transactional database when latency, throughput, large query planning, or multi-writer coordination matters.