Conversation
shutil.disk_usage reports the underlying volume, which on Synology shares, ZFS datasets, NFS shares, and other quota-bound mounts can be far larger than the slice viofosync is actually allowed to consume. The disk-pressure rule then never fired: a 3 TB share inside a 42 TB volume sat silently at 100% of its quota while the threshold saw 86% of the volume. Adds RECORDINGS_QUOTA_GB. When set, used % is measured against bytes-under-recordings ÷ declared quota; a 60-second tree-walk cache amortises the cost across both the retention sweep and any external consumer that wants the same number. Bookkeeping decrement on each delete keeps the inner sweep loop from triggering a fresh walk per file. Leaving the quota at 0 preserves the legacy filesystem path. The percentage and quota rules now trigger independently: each setting works standalone, both-set uses OR semantics so cleanup runs whenever either is breached.
A small card sitting between the section heading and the retention
fields now shows current usage as a percentage with absolute bytes,
and a coloured bar that turns red once usage crosses the cleanup
threshold. A vertical marker on the bar shows where the threshold
sits.
The card reports the same number the retention sweep evaluates
against — quota mode when RECORDINGS_QUOTA_GB is set, filesystem
mode otherwise. Both consumers share retention.py's 60-second
tree-walk cache, so opening the Settings page costs at most one
walk per minute.
* GET /api/storage/usage returns {mode, used_bytes, total_bytes,
used_pct, threshold_pct, max_days} — session-authed, read-only.
* retention.disk_used_pct() is a new public helper that returns
Optional[float] (None on error) so external consumers can show
"unknown" rather than a misleading 0%.
* renderArchiveSection appends the card and kicks an initial
refresh.
* CSS namespaced under .storage-usage* — no overlap with existing
selectors.
Publish viofosync state and accept actions over MQTT, with full HA
discovery so sensors and buttons appear automatically with zero
HA-side YAML.
Surface:
- One device card "Viofosync" with 11 entities
- binary_sensor.viofosync_dashcam (connectivity)
- sensor.viofosync_sync_status (stopped/idle/paused/downloading)
- sensor.viofosync_queue_pending
- sensor.viofosync_last_downloaded_clip (timestamp)
- sensor.viofosync_disk_used (% of the tighter of quota or
filesystem — same number the Settings card shows)
- disabled-by-default: queue_failed, queue_downloading,
current_filename, current_progress, total_clips
- 6 action buttons: start_sync, pause_sync, skip_current,
refresh_queue, retry_failed, rescan_archive
- 1 parameterised command topic: <node>/cmd/prioritize_recent
{"hours": N}
Architecture:
- Single MqttService managed by the FastAPI lifespan; aiomqtt v2
client in one asyncio task with exponential backoff reconnect
- Pure-data EntityDef catalog in mqtt_topology.py is the single
source of truth for discovery payloads, state extractors, and
command handlers
- PublishCoalescer suppresses unchanged payloads and rate-limits
per topic so idle traffic is zero
- LWT + retained availability "online"/"offline" so HA marks all
entities Unavailable within ~45s of an unclean disconnect
- Hot-reload of broker settings; node-rename publishes empty-payload
discovery deletes so HA forgets the old topology cleanly
- Settings page gains an MQTT panel with broker host/port/auth,
TLS, topic prefix, discovery prefix, QoS, a Test connection probe,
and a live status dot
Tests: 60+ unit tests plus an end-to-end test against an in-process
amqtt broker covering connect, discovery republish, state
publication, and command dispatch.
Docs: README section + CHANGELOG entry.
Replaces the previous web UI "Dashcam online/offline" badge and the
Home Assistant `sync_status` sensor's `stopped`/`paused`/`downloading`/
`idle` vocabulary with a single four-state status that means the same
thing everywhere. Fixes two bugs from the old behaviour:
- HA `sync_status` sensor stuck at `downloading` when the dashcam went
offline (it only checked sync_state.running, never dashcam_online).
- Web UI sync icon kept spinning while the dashcam was offline even
though the badge correctly flipped to "Dashcam offline".
State model:
downloading green spinning arrows active sync in progress
waiting orange static arrows dashcam offline or queue empty
paused red pause icon user paused
error red+ warning icon sticky condition needing action
Error conditions (with human-readable reason):
- Camera address not configured (ADDRESS unset)
- Recordings path not writable
- Camera authentication failure (HTTP 401/403)
- Filesystem usage above DISK_CRITICAL_PCT (new setting, default 95%)
The critical-disk trigger measures *filesystem* percentage, not the
retention-aware quota percentage — quota retention is designed to
keep the recordings dir at the quota, so the quota metric reads
~100% during normal operation.
Architecture:
- `web/services/sync_status.py` — pure `compute_sync_status()` is the
single source of truth, called by both the MQTT sensor and the Hub.
- Hub stores `sync_status` and `sync_status_reason` in `last_state`,
broadcasts a `sync_status` event on change so the web UI updates.
- Sync worker broadcasts `disk_pct` and stateful `sync_error` events
that feed the compute function.
- MQTT `sync_status` sensor exposes a `reason` JSON attribute; new
`attrs_fn` infrastructure on `EntityDef` adds `json_attributes_topic`
to discovery payloads.
- Web UI badge shows "Error: <reason>" inline so users see the cause
without having to hover.
Breaking change: HA `sensor.viofosync_sync_status` published values
change from {stopped, paused, downloading, idle} to
{downloading, waiting, paused, error}. Automations matching the old
strings need updating — see CHANGELOG.
Two production bugs both observable as the HA entity briefly going Unavailable when the MqttService trips its reconnect cycle. 1. PublishCoalescer race (KeyError: 'viofosync/current_progress/state') flush_due() snapshots _pending, then awaits sink() per topic, then does `del self._pending[topic]`. During the sink await, a concurrent consider() from _drain_publishes can run; if its payload equals the not-yet-updated _last_payload[topic] it takes the first branch and pops the same entry. The trailing del then raises KeyError, crashes the tick task, and trips the reconnect cycle. A concurrent consider could also replace the entry with a NEWER pending which the bare del silently dropped — a latent data-loss bug paired with the KeyError. Fix: identity check after the await in both flush_due() and consider()'s immediate-publish branch. Only clear _pending[topic] if it still references the same object we just published; otherwise leave whatever a concurrent task installed in place. 2. aiomqtt publish timeout on high-frequency current_progress topic The `current_progress` entity fires on every `item_progress` event during a download. QoS=1 PUBACK waits stall the publisher under broker latency and grow the in-flight queue; aiomqtt eventually times out the publish and the connection drops. The retained value in HA means losing a single progress update is acceptable. Fix: optional per-entity qos override on EntityDef (defaults to the global MQTT_QOS setting). Set qos=0 on current_progress only. All four publish sites in MqttService now resolve to entity.qos when set, falling back to cfg["qos"]. Other entities (sync_status, dashcam, last_downloaded_clip, disk_used, queue_*) keep QoS=1 for reliable state delivery. Tests: - test_flush_due_survives_concurrent_consider_popping_pending — deterministic reproduction of the KeyError race using a blocking sink to interleave a payload-matches-last_payload consider(). - test_flush_due_preserves_newer_pending_added_during_await — deterministic reproduction of the data-loss companion bug. - test_current_progress_uses_qos_0 / test_state_entities_default_to_global_qos structural assertions on the topology change.
When a sync session is downloading, the download manager now shows a session-wide moving-average download speed and an estimated time to complete, and a throttled Home Assistant sensor reports the speed. - DownloadSession tracker (web/services/download_session.py): monotonic wire-byte accounting (clamps retry/file-boundary resets), a 30s windowed moving-average speed, and an ETA computed from the pending queue bytes plus the in-flight file remainder. Pure/injectable. - queue.pending_bytes(): sums remote_size of pending rows for the ETA. - Hub feeds the tracker from broadcast() and emits a deduped session_stats follow-up (whole-second elapsed in the key gives a ~1/s UI heartbeat); last_state["session"] rehydrates reconnecting clients. - HA download_speed sensor (data_rate, MB/s, enabled by default): state_download_speed gates publishing until 30s into a session then the 60s min-interval caps it to once a minute; reports 0 when idle. Triggered by the source broadcast events (item_progress, etc.) so it reaches the MQTT bridge; the session idles on dashcam_offline. - Web UI: live "Session avg X/s, ETA Y, Z this session" line. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add an optional second IP/host for the SAME dashcam, used only when the primary is unreachable (e.g. reaching the camera over a VPN — a Pi hotspot or a site-to-site link to a second parking spot). Not for a second camera. - New ADDRESS_FALLBACK setting (validated like the primary), threaded through the Snapshot and the settings API. - Sync worker selects the active address once per cycle, primary first, alternative only when the primary is down; held for the whole drain (no mid-download switching) and auto-returns to the primary next cycle. - Hub retains the live source/address; new Home Assistant dashcam_connection sensor reports primary/alternative/offline with the live address as an attribute; web UI shows a "via alternative" chip. - Settings → Dashcam gains the alternative-address field (with Test button and field-grouped help note); README + CHANGELOG updated. Co-Authored-By: Claude Opus 4.8 <[email protected]>
DISK_CRITICAL_PCT was in EDITABLE_KEYS (so PUT accepted and persisted it) but missing from the _editable_values() projection that GET returns, so the Settings UI could neither display nor edit it. Add it to the projection and surface the field in the Archive Retention pane. Pre-existing bug, independent of the alternative-camera-address feature. Co-Authored-By: Claude Opus 4.8 <[email protected]>
…download Retention only ran once at startup (unconditional) and at the end of a download cycle that actually downloaded something (gated behind did_any, after the offline early-return). The quota/disk caps therefore went unenforced whenever the camera was offline or had nothing new to download, so the archive stayed over quota until the next restart. Add a standalone retention loop to SyncWorker that sweeps on its own cadence (RETENTION_INTERVAL_SECONDS = 5 min) independent of download activity and camera reachability, tied to the worker lifecycle so it runs while scheduled sync is enabled. The post-drain sweep still provides immediate cleanup after downloads. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Joined/PiP export downloads now get friendly names derived from the
selected clips' date range, camera, and clip count
(e.g. 2024-03-15_1430-1502_front_4clips.mp4), falling back to the
legacy viofosync_export_{id}.mp4 when the source clips were pruned.
Originals can be downloaded directly as individual files (no ZIP, no
joining) via new Originals F/R buttons. The archive action bar is
regrouped into Originals / Joined / PIP sections, and a new rear-main
picture-in-picture variant (pip_rear) joins the existing front-main one.
- web/services/naming.py: build_basename / export_download_name / parse_clip_ids
- web/routers/exports.py: derive download filename; accept pip_rear
- web/services/exporter.py: parametrised PiP main lens; pip_rear job type
- web/static/: regrouped action bar, downloadOriginals helper, wiring
- tests: naming, pip_rear filter, end-to-end download filename + fallback
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Break each day's download list into a collapsible hour tier. Opening a day now shows a list of hours (newest first), each with its clip count, size, and state pills (downloading/pending/done/failed/gone); drilling into an hour reveals the file table. A tri-state select-all cascade spans day -> hour -> file, all backed by the existing flat filename selection set. This keeps a busy day (hundreds of clips) scannable instead of painting one giant table on expand. Pure client-side change — rendering and selection only; the backend, API, and SQL are untouched. Also ignores the local .superpowers/ brainstorm scratch dir. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Rework the export jobs table to be clearer and tidier: - Drop the ID and Created columns. - Show export type as a human-readable badge (Join Front / Join Rear / PiP Fr / PiP Rf), echoing the toolbar buttons. - Merge State + Progress into a single Status cell with a slim inline progress bar for running jobs. - Add a Footage column showing the source clips' date range and clip count, captured onto the export_jobs row at enqueue time (new clip_start/clip_end columns) so it survives retention pruning of the underlying clips. - Replace the text Download/Delete links with right-aligned icon buttons; the delete bin sits in a fixed rightmost column. Co-Authored-By: Claude Opus 4.8 <[email protected]>
ffmpeg's concat demuxer resolves relative entries in the list file against the directory of the list file itself — which lives in the system temp dir. When clip_index stores relative paths (a dev box launched with a relative RECORDINGS), a join export fed ffmpeg a relative path and it went looking under /tmp/.../recordings/..., failing with "No such file or directory". Resolve each clip path to an absolute path before writing it, so the concat works regardless of how the path was stored or where the temp list file happens to live. PiP exports were unaffected (they list absolute mkdtemp segment paths). Co-Authored-By: Claude Opus 4.8 <[email protected]>
Bring Viofo clips into the archive without Wi-Fi sync, via browser folder upload (with folder drag-and-drop) or a configurable folder/USB drop path. Imported clips land in the same recordings/<date>/ layout with GPX, thumbnails, indexing, RO/parking classification and retention, reusing the post-sync pipeline. - IMPORT_PATH advanced setting (default <recordings>/import) - importer service: scan_source, ingest_clip (dedup, cross-volume copy+verify keeping the source, same-volume restore-on-failure), run_folder_ingest with hub progress - smart quota eviction that never deletes anything newer than the clip being imported; import staging excluded from the quota walk - /api/import scan|ingest|upload endpoints (auth + CSRF, path-safe) - RO/locked status survives rescans via a download_queue origin row - Import modal (Upload + Folder tabs) with folder drag-and-drop, launched from the Download manager - README + CHANGELOG Squashed from the feature/sd-card-import branch. Also folds in the v2.2 README/CHANGELOG release-notes edits that were in progress on the branch. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Replace the ephemeral in-DOM "Event log" on the downloads page with a
durable, filterable log backed by a new app_log SQLite table.
- DBLogHandler on the root logger captures INFO+ from viofosync* loggers
(WARNING+ from everything else). emit() only enqueues; an async drain
task batch-inserts off-loop, prunes to a 50k-row cap, and broadcasts
each row over the existing WebSocket hub as {"type":"log"}.
- GET /api/logs serves level/logger/search-filtered, before-cursor
paginated history (auth-gated, fully parameterised queries).
- New "Logs" tab live-tails new rows and loads history with expandable
tracebacks, level filter (default Warning), and a soft row cap; the
old event-log panel is removed.
- Also folds in a small MQTT settings nav-label clarification.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Pausing mid-download set the cancel flag, which surfaced as a generic UserWarning and was swallowed by the retry handler — so a deliberate abort retried 3x and logged a misleading "Failed to download … after 3 attempts", while each pause also burned a queue attempt (mark_downloading increments on every pickup) until the clip was marked failed. - Add a dedicated DownloadCancelled exception; download_file now stops immediately on cancel, logs once at INFO, and consumes no retries. - Add queue.mark_cancelled to requeue an interrupted download and refund the attempt mark_downloading consumed (mirrors reconcile_orphan_downloads). - _download_one distinguishes cancellation from failure so a paused or stopped clip is never marked failed. Logging audit (targeted): - INFO logs for sync paused/resumed/started/stopped and skip/abort. - dashcam online<->offline logged on transition only, not every cycle. - a paused worker now skips the cycle entirely (no probe, no listing), ending the "Found N recordings" log spam while paused. Co-Authored-By: Claude Opus 4.8 <[email protected]>
- Constrain header content to the same 1400px column as <main> and centre it; the bar keeps its full-bleed background and bottom border, with the inset tracking main's padding at each breakpoint. - Route the MQTT status dots and the storage-usage card through the OKLCH palette tokens. The card previously referenced undefined --bg-elev / --bar-track vars and a black threshold marker, all of which rendered near-invisible on the dark theme. - Colour the sync button amber (--warn) in the waiting state so it matches the sync-status badge; fold the waiting badge's raw hex into the --warn token. - Tidy labels: page title "Download manager" -> "Downloads", settings nav "Archives" -> "Storage", "New password (min 8)" -> "(min 8 characters)", and "skipped(quota)" -> "skipped (over quota)". Co-Authored-By: Claude Opus 4.8 <[email protected]>
- Sort/hoist module-level imports to top of file (I001, E402) - Remove unused imports and the unused `hub` local (F401, F841) - Split semicolon statements onto separate lines (E702) - Assert pydantic.ValidationError instead of blind Exception (B017) - Pin starlette<1.0: its TestClient deprecates httpx for the httpx2 fork, which breaks the test suite under filterwarnings=error Co-Authored-By: Claude Opus 4.8 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Added
Home Assistant MQTT Support
Auto-discovered sensors and action buttons over MQTT, set up from a new Settings panel.
Manual Import
Add clips to the archive without Wi-Fi sync — by browser upload or a folder/USB drop path.
Alternative Camera Address
An optional second address for the same camera, used automatically when the primary is unreachable (e.g. reaching the dashcam over a mobile VPN).
Quota-Bound Retention
Measure retention and disk thresholds against a declared quota (
RECORDINGS_QUOTA_GB), for recordings on a NAS share or ZFS dataset.Sync Error Reporting
Sync now surfaces a sticky
errorstate — missing config, unwritable path, camera auth failure, or disk full — in both the UI and Home Assistant.Download Manager Improvements
Session speed and ETA while syncing, one-click retry of failed downloads, and live disk usage in Settings.
Export Improvements
Meaningful download filenames, direct download of the original front/rear clips, and a new rear-main picture-in-picture variant.
Changed
downloading/waiting/paused/error); update any Home Assistant automations that matched the oldidle/stoppedstrings.Fixed