dl — a deduplicating download cache

A command-line download tool that never downloads the same thing twice. It uses content-addressable storage (the same idea behind Git, Docker, and pnpm) to save both bandwidth and disk space.

The idea

There are two independent things worth not repeating, and they're caught at two different moments:

• URL check (before download) — if you've fetched this exact URL before, the file is served instantly from the cache. Saves the download.
• Hash check (after download) — files are identified by their SHA-256 hash, computed as the bytes stream in. If a download turns out to be byte-for-byte identical to something already stored (e.g. the same file from a mirror), the duplicate is discarded and the new URL just points at the existing copy. Saves the disk.

Files are stored once, named by their hash, under ~/.dl/store/. A SQLite index at ~/.dl/cache.db maps each URL to its hash and metadata.

Usage

npm install

# download (or serve from cache if seen before)
npm run dl -- get https://example.com/file.zip
npm run dl -- get https://example.com/file.zip -o ~/Downloads
npm run dl -- get https://example.com/file.zip --open   # ...and open it afterwards

# open a file in its default app (served from cache if already downloaded)
npm run dl -- open https://example.com/file.zip

npm run dl -- list     # show everything cached ( ! = file missing on disk )
npm run dl -- stats    # bandwidth + disk saved
npm run dl -- scrub    # remove stale index rows, delete unreferenced files

How it stays consistent

The index can drift from the disk (you might delete a stored file). Two mechanisms keep it honest:

• Verify-on-access — before serving a cached file, it checks the file still exists; if not, it drops the stale row and re-downloads. The index self-heals.
• scrub — an explicit sweep that removes rows for missing files and garbage-collects blobs no URL references anymore (reference counting, like git gc).

Always-on daemon

The same engine can run as a background service so it's always ready (and so a browser extension can call it). It's a small localhost-only HTTP server.

npm run dl -- daemon      # run it in the foreground (for testing)
npm run dl -- install     # register with launchd: auto-start at login + auto-restart on crash
npm run dl -- status      # is it running?
npm run dl -- uninstall   # stop and remove it

HTTP API (bound to 127.0.0.1 only):

GET  /health
GET  /list
GET  /stats
POST /get      body: {"url": "...", "outDir": "..."}

How it stays correct even when it's down

A background service is never up 100% of the time (crashes, reboots, updates). Rather than chasing perfect uptime, the daemon is self-correcting — four layers, each covering the previous one's blind spot:

1. launchd (RunAtLoad + KeepAlive) — starts it at login and relaunches it on crash.
2. Startup scrub — on every boot it reconciles the index with disk, healing anything that changed while it was down.
3. Verify-on-access — before serving any file it checks the file still exists; the per-request correctness guarantee.
4. File-watcher — while running, reacts to store changes in real time (the speed layer).

Architecture

CLI (src/cli.ts) ─┐
Daemon (src/daemon.ts) ─┤  both call the same engine
                        └─ engine (src/engine.ts)  URL check → download+hash → dedup → store
                             ├─ db (src/db.ts)          SQLite index: url → hash, size, etag, hits
                             ├─ store (src/store.ts)     content-addressable blob store
                             └─ maintenance (src/maintenance.ts)  scrub (reconcile + GC) + stats

Roadmap

• Engine + CLI
• Always-on daemon (launchd + startup scrub + verify-on-access + file-watcher)
• Browser extension that routes downloads through the daemon automatically (extension/)
• HTTP revalidation with ETag / If-None-Match (304 = serve cache)
• Local proxy to catch downloads outside the browser (stretch)

Stack

TypeScript · Node.js (streaming fetch + crypto) · better-sqlite3 · commander