Migrate db and fix retention sweep#12
Merged
Merged
Conversation
The SQLite state DB used to live at $RECORDINGS/.viofosync.db, putting
every per-write fcntl() syscall through the NAS-backed recordings
mount. On Synology DSM with NFS, that costs up to 30 s per write while
NFS write-back flushes the freshly-downloaded MP4 — observed
empirically.
Two new public helpers in web/db.py:
* default_db_path() — resolves the new location, $CONFIG_DIR/viofosync.db
(default /config/viofosync.db, a local volume).
* migrate_legacy_db_path(new_path) — on first boot, copies any DB
at the legacy $RECORDINGS/.viofosync.db location to the new path
and renames the legacy file to .viofosync.db.migrated so an
operator has a recoverable copy. Idempotent.
The migration is called explicitly from web/app.py's lifespan, NOT
from Database.__init__. This keeps test isolation tight: test code
constructing Database(tmp_path) never reads the host's RECORDINGS env
var.
Implementation details:
* Atomic main-file copy via tmp = new_path + ".part" + os.replace,
so a crash mid-copy leaves the new path absent and the next boot
retries cleanly.
* -wal / -shm sidecar copy failures log WARNING and continue
(SQLite reconstructs from the main file alone if these are missing
or stale).
* Legacy-rename failure logs WARNING and continues — harmless,
leaves the operator with the original file still in place.
* Main-copy failure propagates: loud startup failure is preferable
to silently opening an empty fresh DB.
Tests: 7 new in tests/test_db_migration.py covering happy path,
no-overwrite, legacy-missing, sidecar-failure-non-fatal, the
Database.__init__ isolation guarantee, and both default_db_path
branches.
docker-compose.yml comment updated to direct operators to /config
for the state DB.
The startup retention sweep used to run synchronously inside the
FastAPI lifespan, before `yield`. On a deployment with a large
backlog of >max_days clips, this blocked the UI for tens of minutes
(1126 clips ≈ 210 GB observed on Synology). uvicorn stayed at
"Waiting for application startup" the entire time, with no progress
signal because retention.sweep() only logged totals at the end.
Changes:
* Move the startup sweep into asyncio.create_task(_background_retention())
after the existing initial-scan task. The UI is reachable within
seconds of "Started server process" regardless of backlog size.
* Add an INFO header log when the time-phase has work, so the operator
can see the sweep started:
"retention sweep: N clip(s) older than D days — examining"
Says "examining" rather than "deleting" because the count is the
pre-filter row total; RO-protected rows are excluded inside the
loop. The end-of-sweep summary reports the real deleted count.
* Add an every-10-deletions progress log:
"retention sweep: N/M clip(s) deleted (X.X MB freed so far)"
so a long sweep is visibly progressing rather than silent.
* Promote `from .services import retention as _ret_mod` to module
scope in web/app.py (was inline).
* Extend the shutdown finally block to cancel app.state.retention_task
alongside the existing initial_scan_task.
Tests: 3 new in tests/test_retention.py (header logs when work,
silence when no work, every-10 progress). 2 new in
tests/test_lifespan_retention.py (background task created during
lifespan, source-inspection that shutdown cancels retention_task).
No change to retention eligibility rules. The sync-worker's
post-cycle piggyback retention call (web/services/sync_worker.py)
is intentionally untouched — current cadence (runs after each
productive sync cycle) is adequate for the always-on dashcam use
case.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixed
/configvolume rather than the recordingsmount for better performance when the recordings are on
slower storage.
blocking the UI when there's a large delete backlog.
Changed
work, then a line every 10 deletions. Previously silent until the
end-of-sweep summary.
Migration
${RECORDINGS}/.viofosync.dbis copied to${CONFIG_DIR}/viofosync.dbon first boot under v2.1. The legacyfile is renamed to
.viofosync.db.migratedon the recordingsvolume as a recoverable fallback.