diff --git a/.env.example b/.env.example index 3ee9589..bfaf2b2 100644 --- a/.env.example +++ b/.env.example @@ -80,6 +80,15 @@ APRS_CALLSIGN=N0CALL APRS_PASSCODE=-1 APRS_FILTER_RADIUS_KM=80 +# MeshCore node gating — a pyMC-Repeater advertises every node it has ever +# heard, which can span an entire regional mesh. When enabled (default), +# nodes whose advertised position falls outside the configured region +# bbox(es) are dropped, like the ADS-B/AIS bbox gating. The pad widens the +# bbox by N degrees so nearby ridge-top repeaters just outside the box are +# still kept. Nodes with no advertised position always pass. +MESH_BBOX_FILTER=true +MESH_BBOX_PAD_DEG=0.25 + # ADS-B ingest strategy # Mode A — BEAST TCP (set ADSB_ENABLE_BEAST=true + configure sources.yml) # Mode B — ultrafeeder HTTP only (BEAST disabled, sources.yml has adsb entries) diff --git a/TASK_LOG.md b/TASK_LOG.md index df39b3c..5e95a65 100644 --- a/TASK_LOG.md +++ b/TASK_LOG.md @@ -5,6 +5,40 @@ Format: `## YYYY-MM-DD — ` with bullet points for details. --- +## 2026-07-05 — Gated mesh nodes to region bbox and made entity filters persistent + +- **Mesh node bbox gating** ([meshcore.py](poller/pollers/meshcore.py), [config.py](poller/config.py)): + - A pyMC-Repeater's advert table covers every node it has ever heard (the whole Cascade mesh), so the poller was publishing hundreds of far-away nodes that cluttered the map and dragged down the render pipeline. + - Added `_should_publish_node()` gate that drops `mesh_node` entities whose advertised position falls outside the configured region bbox(es), mirroring the ADS-B/AIS/Amtrak gating. Applied to all three publish paths: REST advert sync, SSE `advert_received`/`contact_path_updated`, and stats neighbors. + - New settings `MESH_BBOX_FILTER` (default true) and `MESH_BBOX_PAD_DEG` (default 0.25°, keeps nearby ridge-top repeaters just outside the box). Documented in `.env.example`. + - Nodes with no advertised position always pass — they can't clutter the map and direct RF neighbors often advertise without GPS. Gated nodes age out of Redis (120s entity TTL) within ~2 minutes of a poller restart. +- **Stateful entity filters** ([store.ts](frontend/src/store.ts)): + - Added `entityFilter` to the Zustand `persist` partialize so layer toggles (e.g. hiding mesh nodes) survive PWA close/reopen — previously the toggle reset to all-visible on every reload. + - Added a custom `merge` that deep-merges the persisted filter over defaults, so filter keys added in future versions default to visible instead of disappearing for users with older persisted state. + - Side benefit: the WebSocket subscription is built from `entityFilter` on connect, so a persisted `mesh_node=false` now also suppresses mesh entity updates server-side. +- **Tests**: New [test_meshcore_bbox.py](poller/tests/test_meshcore_bbox.py) — 6 tests covering in/out-of-bbox, pad behavior, unpositioned nodes, and the disabled-filter path. +- **Narrowed store subscriptions** ([store.ts](frontend/src/store.ts) + 26 components/hooks): + - Added `useCivicPick(...keys)` (shallow-equality picker) and converted all 29 bare `useCivicStore()` call sites — previously every WS entity update re-rendered the whole React tree from `Dashboard` down, which was the main-thread churn starving the P25 audio stream. +- **Batched WS entity updates** ([useWebSocket.ts](frontend/src/hooks/useWebSocket.ts), [store.ts](frontend/src/store.ts)): + - New `upsertEntities` action; `entity_update` messages buffer and flush in one store commit per 250ms (~4 notifications/sec instead of one per message). Cold-start REST seed now requests `limit=2000` (backend default of 200 gave a partial first paint). +- **Render-loop trims** ([MapOverlay.tsx](frontend/src/components/MapOverlay.tsx)): + - Layer-rebuild throttle 16ms→33ms (16 was a no-op at 60fps rAF); tooltip GPU picking throttled to 50ms; fixed tooltip-hide handler (`mouseout`, not the never-firing map-level `mouseleave`) and added its cleanup; removed write-only `layersRef`. +- **Bug fixes**: + - Mesh links now render: packet-derived links use `node_a="local"` which never resolved to an entity, so [MeshLinksLayer.tsx](frontend/src/components/layers/MeshLinksLayer.tsx) dropped every feature. `"local"` anchors at the region center; the layer also now subscribes to mesh nodes only instead of the whole entities map. + - Annotation draw preview no longer renders twice (removed the passive deck.gl copy in MapOverlay; the interactive MapLibre `AnnotationOverlay` owns it). + - Bounded the poller's `_entity_cache` in [bus.py](poller/bus.py) — entries carry a monotonic last-seen timestamp and stale ones (>10 min) are swept every 4096 publishes (was an unbounded slow leak). +- **Dead code removed**: `components/layers/MeshLayer.tsx`, `StreamGaugeLayer.tsx` (never imported), `ObservationRingLayer.tsx` (mounted no-op stub), `buildAnnotationDrawPreviewLayers`, unused `snr_to_quality` import, and uncalled `normalize_mesh_node`/`_bridge_status` in the poller. +- **Repeater self-node with GPS** ([meshcore.py](poller/pollers/meshcore.py), [MeshLinksLayer.tsx](frontend/src/components/layers/MeshLinksLayer.tsx), [sources.example.yml](config/sources.example.yml)): + - The repeater station now publishes itself as a `mesh_node` entity when its position is known: best-effort GPS extraction from `/api/stats` (top-level + nested containers), or an explicit `?lat=&lon=` pin on the source URL. Entity carries battery/noise-floor status for remote monitoring and bypasses the bbox gate (explicit operator config). + - Packet-derived links anchor on the repeater's entity id instead of the `"local"` placeholder; `mesh:status` carries `lat`/`lon` so the frontend anchors legacy `"local"` links at the actual station before falling back to the region center. + - Link metric fixes: `link_quality` now computed from SNR on the 0–100 scale the UI expects (was always NULL); WS `mesh_links` payload now includes `last_seen` (missing value rendered every live link at minimum opacity); `snrToColor` rescaled from RSSI-style −70/−90 thresholds (everything green) to LoRa SNR (green ≥ 5 dB, amber ≥ −10 dB). + - 16 new tests in [test_meshcore_self_node.py](poller/tests/test_meshcore_self_node.py). +- **Live neighbor list in EntityDetail** ([EntityDetail.tsx](frontend/src/components/panels/EntityDetail.tsx)): + - The Neighbors section now overlays the live `meshLinks` store array (WS-fed) on top of the one-shot REST fetch, per the 2026-05-10 mesh audit recommendation — fresh SNR readings appear immediately and the list survives REST failures. + - Neighbor rows show peer display names (raw id on hover) and click through to select known peers; `"local"` resolves to the repeater's site name from `mesh:status`. + - Fixed the panel's remaining RSSI-scale SNR thresholds (−70/−90 labeled "dBm") to the LoRa SNR scale with dB units. +- **Motivation**: User reported the entire Cascade mesh rendering (not just local nodes), render-pipeline jank severe enough to block the P25 audio stream from connecting, and mesh-node filter state not surviving PWA reloads; a follow-up audit request surfaced the re-render storm, batching gaps, bugs, and dead code fixed above. + ## 2026-06-15 — Enhanced Mesh Companion Connectivity & Streamlined Tab Layout - **Mesh Companion Status Sync**: diff --git a/config/sources.example.yml b/config/sources.example.yml index f956c23..d3766d6 100644 --- a/config/sources.example.yml +++ b/config/sources.example.yml @@ -93,6 +93,10 @@ poller_sources: # http://MY_API_KEY@192.168.1.x:8000 # Add a companion parameter if you want to lock the connection to a specific companion identity: # http://MY_API_KEY@192.168.1.x:8000?companion=MyCompanionNode + # Pin the repeater's own position with lat/lon parameters — used when the + # repeater API doesn't report GPS. The station then appears on the map as + # a mesh_node entity and anchors its packet-derived link lines: + # http://192.168.1.x:8000?lat=45.3842&lon=-122.7635 enabled: true source: config diff --git a/frontend/src/App.tsx b/frontend/src/App.tsx index 7b3b575..45aac2a 100644 --- a/frontend/src/App.tsx +++ b/frontend/src/App.tsx @@ -1,5 +1,5 @@ import { useState, useEffect } from 'react' -import { useCivicStore } from './store' +import { useCivicPick } from './store' import { useAlerts } from './hooks/useAlerts' import { useSystemHealth } from './hooks/useSystemHealth' import { useTrailHydration } from './hooks/useTrailHydration' @@ -43,7 +43,7 @@ function Dashboard() { usePreferences() useMeshHistory() - const { news, appendSystemEvent } = useCivicStore() + const { news, appendSystemEvent } = useCivicPick('news', 'appendSystemEvent') // Background Intelligence Processor // Elevates critical news headlines to system events (Incidents) @@ -56,7 +56,7 @@ function Dashboard() { }) }, [news, appendSystemEvent]) - const { activeTab, mode } = useCivicStore() + const { activeTab, mode } = useCivicPick('activeTab', 'mode') const isCritical = mode === 'critical' return ( diff --git a/frontend/src/components/DevInsetInspector.tsx b/frontend/src/components/DevInsetInspector.tsx index dfd4fa1..61d9d83 100644 --- a/frontend/src/components/DevInsetInspector.tsx +++ b/frontend/src/components/DevInsetInspector.tsx @@ -1,5 +1,5 @@ import { useEffect, useState, useCallback } from 'react' -import { useCivicStore } from '../store' +import { useCivicPick } from '../store' // ── Developer overlay: surfaces the exact iOS safe-area insets, viewport // geometry, and DOM-layer heights on-device, so we never have to guess at @@ -214,7 +214,7 @@ function LayerRow({ k, rect }: { k: LayerKey; rect: LayerRect }) { } export function DevInsetInspector() { - const { debugInsets, setDebugInsets } = useCivicStore() + const { debugInsets, setDebugInsets } = useCivicPick('debugInsets', 'setDebugInsets') const [metrics, setMetrics] = useState(() => readMetrics()) const [corner, setCorner] = useState('top-right') const [copied, setCopied] = useState(false) diff --git a/frontend/src/components/Map.tsx b/frontend/src/components/Map.tsx index a36c86b..71dbb8b 100644 --- a/frontend/src/components/Map.tsx +++ b/frontend/src/components/Map.tsx @@ -7,7 +7,6 @@ import { SmokeLayer } from './layers/SmokeLayer' import { GOESLayer } from './layers/GOESLayer' import { FirePerimeterLayer } from './layers/FirePerimeterLayer' import { GeofenceLayer } from './layers/GeofenceLayer' -import { ObservationRingLayer } from './layers/ObservationRingLayer' import { CustomLayersLayer } from './layers/CustomLayersLayer' import { AnnotationOverlay } from './layers/AnnotationOverlay' import { TerrainLayer } from './layers/TerrainLayer' @@ -208,7 +207,6 @@ export function Map() { - diff --git a/frontend/src/components/MapOverlay.tsx b/frontend/src/components/MapOverlay.tsx index e511b47..31cbe74 100644 --- a/frontend/src/components/MapOverlay.tsx +++ b/frontend/src/components/MapOverlay.tsx @@ -7,7 +7,7 @@ import { buildEntityLayers } from '../layers/buildEntityLayers' import { buildTrailLayers } from '../layers/buildTrailLayers' import { buildCameraLayer } from '../layers/buildCameraLayer' import { buildEventLayers } from '../layers/buildEventLayers' -import { buildAnnotationLayers, buildAnnotationDrawPreviewLayers } from '../layers/AnnotationLayer' +import { buildAnnotationLayers } from '../layers/AnnotationLayer' import { buildGeofenceLayers, type GeofenceItem } from '../layers/buildGeofenceLayers' import { buildObservationRingLayers } from '../layers/buildObservationRingLayer' import { buildCustomLayers } from '../layers/buildCustomLayers' @@ -88,7 +88,6 @@ function buildReplayTracks(data: ReplayData, atMs: number): Record(null) - const layersRef = useRef([]) // Per-group layer cache so static/slow-changing layers are rebuilt only when // their inputs change instead of on every animation frame. Reusing the same // Layer instances lets deck.gl skip re-diffing them entirely. @@ -139,14 +138,8 @@ export function MapOverlay({ map }: Props) { const annotationsVisible = useCivicStore((s) => s.annotationsVisible) const customLayers = useCivicStore((s) => s.customLayers) const annotationDrawMode = useCivicStore((s) => s.annotationDrawMode) - const annotationDrawPoints = useCivicStore((s) => s.annotationDrawPoints) - const annotationDrawCursor = useCivicStore((s) => s.annotationDrawCursor) const annotationDrawModeRef = useRef<'marker' | 'line' | 'polygon' | null>(null) - const annotationDrawPointsRef = useRef<[number, number][]>([]) - const annotationDrawCursorRef = useRef<[number, number] | null>(null) useEffect(() => { annotationDrawModeRef.current = annotationDrawMode }, [annotationDrawMode]) - useEffect(() => { annotationDrawPointsRef.current = annotationDrawPoints }, [annotationDrawPoints]) - useEffect(() => { annotationDrawCursorRef.current = annotationDrawCursor }, [annotationDrawCursor]) const annotationsRef = useRef(annotations) const annotationsVisibleRef = useRef(annotationsVisible) const customLayersRef = useRef(customLayers) @@ -295,6 +288,12 @@ export function MapOverlay({ map }: Props) { tooltip.className = 'absolute pointer-events-none z-[100] opacity-0 transition-opacity duration-150' container.appendChild(tooltip) + // Deck picking is a GPU readback — throttle it so fast mouse movement + // doesn't stall the render loop. Between picks the tooltip just tracks + // the cursor. + const PICK_INTERVAL_MS = 50 + let lastPickMs = 0 + const onMapMouseMove = (e: maplibregl.MapMouseEvent) => { const isMobileViewport = window.innerWidth < 1024 if (isMobileViewport) { @@ -303,6 +302,16 @@ export function MapOverlay({ map }: Props) { return } + const nowMs = performance.now() + if (nowMs - lastPickMs < PICK_INTERVAL_MS) { + if (tooltip.style.opacity === '1') { + tooltip.style.left = `${e.point.x + 15}px` + tooltip.style.top = `${e.point.y + 15}px` + } + return + } + lastPickMs = nowMs + // 1. Pick from Deck.gl (returns null until the overlay GL context is ready) const picked = overlay.pickObject({ x: e.point.x, y: e.point.y, radius: 5 }) @@ -473,8 +482,14 @@ export function MapOverlay({ map }: Props) { } } + // MapLibre's canvas-level leave event is 'mouseout' — 'mouseleave' only + // fires for the layer-id overload and would never trigger here. + const onMapMouseOut = () => { + tooltip.style.opacity = '0' + map.getCanvas().style.cursor = '' + } map.on('mousemove', onMapMouseMove) - map.on('mouseleave', () => { tooltip.style.opacity = '0' }) + map.on('mouseout', onMapMouseOut) // Allow selecting entities and cameras while preserving normal map interaction. const onMapClick = (e: maplibregl.MapMouseEvent) => { @@ -499,7 +514,11 @@ export function MapOverlay({ map }: Props) { let last = performance.now() let lastLayerBuild = 0 - const LAYER_BUILD_INTERVAL_MS = 16 + // rAF fires every ~16.7ms, so anything <= 16 here is a no-op throttle. + // 33ms rebuilds layers at ~30fps — half the filter/PVB/layer-construction + // work per second, and PVB keeps icon motion smooth between rebuilds while + // MapLibre still pans/zooms the drawn frame at full 60fps. + const LAYER_BUILD_INTERVAL_MS = 33 // When the tab is backgrounded the browser pauses/throttles rAF while the // WebSocket keeps delivering position updates. On return, re-anchor motion @@ -698,18 +717,13 @@ export function MapOverlay({ map }: Props) { () => (camerasVisibleRef.current ? [buildCameraLayer(camerasRef.current, selectedCamRef.current, zoom)] : [])), + // Draw preview intentionally omitted: AnnotationOverlay owns the + // interactive drawing UX and already renders the preview via its + // MapLibre source — rendering it here too drew it twice. ...memoGroup('annotation', [annotationsRef.current, annotationsVisibleRef.current], () => buildAnnotationLayers(annotationsRef.current, annotationsVisibleRef.current)), - ...memoGroup('annotationDraw', - [annotationDrawModeRef.current, annotationDrawPointsRef.current, annotationDrawCursorRef.current], - () => buildAnnotationDrawPreviewLayers({ - mode: annotationDrawModeRef.current, - points: annotationDrawPointsRef.current, - cursor: annotationDrawCursorRef.current, - })), ] - layersRef.current = layers overlay.setProps({ layers }) rafRef.current = requestAnimationFrame(tick) @@ -721,6 +735,7 @@ export function MapOverlay({ map }: Props) { document.removeEventListener('visibilitychange', onVisibility) map.off('click', onMapClick) map.off('mousemove', onMapMouseMove) + map.off('mouseout', onMapMouseOut) map.removeControl(overlay as unknown as maplibregl.IControl) tooltip.remove() layerMemoRef.current = {} diff --git a/frontend/src/components/layers/MeshLayer.tsx b/frontend/src/components/layers/MeshLayer.tsx deleted file mode 100644 index 5ec80e0..0000000 --- a/frontend/src/components/layers/MeshLayer.tsx +++ /dev/null @@ -1,84 +0,0 @@ -import { useEffect } from 'react' -import maplibregl from 'maplibre-gl' -import { useEntitiesByType } from '../../hooks/useEntities' -import { useCivicStore } from '../../store' - -interface Props { map: maplibregl.Map } - -const SRC = 'mesh-nodes' -const LAYER = 'mesh-node-points' -const STALE_MS = 10 * 60 * 1000 // 10 minutes - -export function MeshLayer({ map }: Props) { - const nodes = useEntitiesByType('mesh_node') - const selectEntity = useCivicStore((s) => s.selectEntity) - const meshVisible = useCivicStore((s) => s.entityFilter.mesh_node) - - useEffect(() => { - const handleClick = (e: maplibregl.MapMouseEvent & { features?: maplibregl.MapGeoJSONFeature[] }) => { - const f = e.features?.[0] - if (f?.properties?.id) selectEntity(f.properties.id as string) - } - const handleEnter = () => { map.getCanvas().style.cursor = 'pointer' } - const handleLeave = () => { map.getCanvas().style.cursor = '' } - - if (!map.getSource(SRC)) { - map.addSource(SRC, { type: 'geojson', data: { type: 'FeatureCollection', features: [] } }) - map.addLayer({ - id: `${LAYER}-ring`, - type: 'circle', - source: SRC, - paint: { - 'circle-radius': 9, - 'circle-color': ['case', ['get', 'stale'], '#555', '#1a9641'], - 'circle-opacity': 0.3, - }, - }) - map.addLayer({ - id: LAYER, - type: 'circle', - source: SRC, - paint: { - 'circle-radius': 5, - 'circle-color': ['case', ['get', 'stale'], '#888', '#4dac26'], - 'circle-stroke-width': 1, - 'circle-stroke-color': '#fff', - }, - }) - map.on('click', LAYER, handleClick) - map.on('mouseenter', LAYER, handleEnter) - map.on('mouseleave', LAYER, handleLeave) - } - - return () => { - map.off('click', LAYER, handleClick) - map.off('mouseenter', LAYER, handleEnter) - map.off('mouseleave', LAYER, handleLeave) - if (map.getLayer(`${LAYER}-ring`)) map.removeLayer(`${LAYER}-ring`) - if (map.getLayer(LAYER)) map.removeLayer(LAYER) - if (map.getSource(SRC)) map.removeSource(SRC) - } - }, [map, selectEntity]) - - useEffect(() => { - const src = map.getSource(SRC) as maplibregl.GeoJSONSource | undefined - if (!src) return - const now = Date.now() - const visibleNodes = meshVisible ? nodes : [] - const geojson: GeoJSON.FeatureCollection = { - type: 'FeatureCollection', - features: visibleNodes.map((n) => { - const lastMs = n.last_seen ? Date.parse(n.last_seen) : 0 - const isStale = now - lastMs > STALE_MS - return { - type: 'Feature', - geometry: { type: 'Point', coordinates: [n.lon!, n.lat!] }, - properties: { id: n.entity_id, name: n.display_name ?? n.entity_id, status: n.status ?? '', stale: isStale }, - } - }), - } - src.setData(geojson) - }, [nodes, map, meshVisible]) - - return null -} diff --git a/frontend/src/components/layers/MeshLinksLayer.tsx b/frontend/src/components/layers/MeshLinksLayer.tsx index cd7a392..ff26168 100644 --- a/frontend/src/components/layers/MeshLinksLayer.tsx +++ b/frontend/src/components/layers/MeshLinksLayer.tsx @@ -1,8 +1,9 @@ -import { useEffect, useRef } from 'react' +import { useEffect } from 'react' import type { Map as MapLibreMap } from 'maplibre-gl' import { useCivicStore } from '../../store' -import { API_BASE } from '../../config' -import { authHeaders } from '../../auth' +import type { Entity } from '../../store' +import { useEntitiesByType } from '../../hooks/useEntities' +import { DEFAULT_CENTER } from '../../config' interface MeshLink { node_a: string @@ -15,10 +16,12 @@ interface MeshLink { const SOURCE_ID = 'mesh-links' const LAYER_ID = 'mesh-links-line' +// MeshCore LoRa SNR spans roughly -20..+12 dB. (The old -70/-90 thresholds +// were RSSI-scale values, so every link always rendered green.) function snrToColor(snr: number | null): [number, number, number] { if (snr === null) return [150, 150, 150] - if (snr >= -70) return [68, 221, 136] // strong — green - if (snr >= -90) return [255, 184, 0] // medium — amber + if (snr >= 5) return [68, 221, 136] // strong — green + if (snr >= -10) return [255, 184, 0] // medium — amber return [255, 80, 80] // weak — red } @@ -32,8 +35,11 @@ interface Props { } export function MeshLinksLayer({ map }: Props) { - const entities = useCivicStore((s) => s.entities) + // Subscribe to mesh nodes only — depending on the whole entities map made + // this effect rebuild its GeoJSON on every aircraft/vessel update. + const meshNodes = useEntitiesByType('mesh_node') const meshLinks = useCivicStore((s) => s.meshLinks) + const meshStatus = useCivicStore((s) => s.meshStatus) // Initialize Layer and Source useEffect(() => { @@ -76,13 +82,34 @@ export function MeshLinksLayer({ map }: Props) { const now = Date.now() + const byId: Record = {} + for (const n of meshNodes) byId[n.entity_id] = n + + // Packet-derived links may be recorded with node_a = "local" (the + // repeater itself). Anchor those at the repeater's reported position + // from mesh:status (GPS from /api/stats or the ?lat=&lon= source pin), + // falling back to the configured region center. + const resolve = (id: string): [number, number] | null => { + if (id === 'local') { + const lat = Number(meshStatus?.lat) + const lon = Number(meshStatus?.lon) + if (Number.isFinite(lat) && Number.isFinite(lon)) return [lon, lat] + return DEFAULT_CENTER + } + const n = byId[id] + return n && n.lat != null && n.lon != null ? [n.lon, n.lat] : null + } + const features = meshLinks.flatMap((link) => { - const nodeA = entities[link.node_a] - const nodeB = entities[link.node_b] - if (!nodeA?.lat || !nodeA?.lon || !nodeB?.lat || !nodeB?.lon) return [] + const coordA = resolve(link.node_a) + const coordB = resolve(link.node_b) + if (!coordA || !coordB) return [] + // Missing/unparsable last_seen (NaN) counts as fresh rather than + // dimming the link to minimum opacity. const ageMinutes = (now - new Date(link.last_seen).getTime()) / 60_000 - const opacity = ageMinutes < 5 ? 0.9 : ageMinutes < 15 ? 0.6 : ageMinutes < 30 ? 0.35 : 0.15 + const opacity = !Number.isFinite(ageMinutes) || ageMinutes < 5 ? 0.9 + : ageMinutes < 15 ? 0.6 : ageMinutes < 30 ? 0.35 : 0.15 const quality = link.link_quality ?? 0 const width = 1.5 + (Math.min(Math.max(quality, 0), 100) / 100) * 2.5 @@ -90,10 +117,7 @@ export function MeshLinksLayer({ map }: Props) { type: 'Feature' as const, geometry: { type: 'LineString' as const, - coordinates: [ - [nodeA.lon, nodeA.lat], - [nodeB.lon, nodeB.lat], - ], + coordinates: [coordA, coordB], }, properties: { snr: link.snr, @@ -108,7 +132,7 @@ export function MeshLinksLayer({ map }: Props) { type: 'FeatureCollection', features }) - }, [map, entities, meshLinks]) + }, [map, meshNodes, meshLinks, meshStatus]) return null } diff --git a/frontend/src/components/layers/ObservationRingLayer.tsx b/frontend/src/components/layers/ObservationRingLayer.tsx deleted file mode 100644 index 8c1d2e2..0000000 --- a/frontend/src/components/layers/ObservationRingLayer.tsx +++ /dev/null @@ -1,9 +0,0 @@ -import maplibregl from 'maplibre-gl' - -interface Props { map: maplibregl.Map } - -export function ObservationRingLayer({ map }: Props) { - void map - - return null -} diff --git a/frontend/src/components/layers/StreamGaugeLayer.tsx b/frontend/src/components/layers/StreamGaugeLayer.tsx deleted file mode 100644 index d8a65da..0000000 --- a/frontend/src/components/layers/StreamGaugeLayer.tsx +++ /dev/null @@ -1,127 +0,0 @@ -import { useEffect, useMemo, useState } from 'react' -import maplibregl from 'maplibre-gl' -import { useEntitiesByType } from '../../hooks/useEntities' -import { useCivicStore } from '../../store' -import type { Entity } from '../../store' -import { API_BASE } from '../../config' -import { authHeaders } from '../../auth' - -interface Props { map: maplibregl.Map } - -const SRC = 'stream-gauges' -const LAYER = 'stream-gauge-dots' -const RING = 'stream-gauge-ring' -const LABEL = 'stream-gauge-label' - -// Flow stage → map color -const STAGE_COLOR: Record = { - normal: '#4fc3f7', // light blue - elevated: '#fff176', // yellow - 'minor flood': '#ffb74d', // orange - 'moderate flood':'#ef5350', // red - 'major flood': '#b71c1c', // dark red - unknown: '#90a4ae', // grey -} - -export function StreamGaugeLayer({ map }: Props) { - const gauges = useEntitiesByType('stream_gauge') - const gaugesVisible = useCivicStore((s) => s.gaugesVisible) - const selectEntity = useCivicStore((s) => s.selectEntity) - const [fallbackGauges, setFallbackGauges] = useState([]) - - // Defensive fallback: if websocket snapshot misses stream_gauge entities, - // fetch them from REST so operators still see hydrology overlays. - useEffect(() => { - let cancelled = false - - const loadFallback = async () => { - if (gauges.length > 0) { - setFallbackGauges([]) - return - } - try { - const res = await fetch(`${API_BASE}/entities?entity_type=stream_gauge`, { - headers: authHeaders(), - }) - if (!res.ok) return - const data = await res.json() - if (!cancelled && Array.isArray(data)) { - setFallbackGauges(data as Entity[]) - } - } catch { - // Non-fatal: map can still render websocket-fed entities. - } - } - - loadFallback() - const id = window.setInterval(loadFallback, 60_000) - return () => { - cancelled = true - window.clearInterval(id) - } - }, [gauges]) - - const displayGauges = useMemo( - () => (gauges.length > 0 ? gauges : fallbackGauges), - [gauges, fallbackGauges], - ) - - useEffect(() => { - const handleClick = (e: maplibregl.MapMouseEvent & { features?: maplibregl.MapGeoJSONFeature[] }) => { - const f = e.features?.[0] - if (f?.properties?.id) selectEntity(f.properties.id as string) - } - const handleEnter = () => { map.getCanvas().style.cursor = 'pointer' } - const handleLeave = () => { map.getCanvas().style.cursor = '' } - - if (!map.getSource(SRC)) { - map.addSource(SRC, { type: 'geojson', data: { type: 'FeatureCollection', features: [] } }) - map.addLayer({ id: RING, type: 'circle', source: SRC, paint: { 'circle-radius': 14, 'circle-color': ['get', 'color'], 'circle-opacity': 0.45 } }) - map.addLayer({ id: LAYER, type: 'circle', source: SRC, paint: { 'circle-radius': 7, 'circle-color': ['get', 'color'], 'circle-stroke-width': 2, 'circle-stroke-color': '#fff' } }) - map.addLayer({ - id: LABEL, type: 'symbol', source: SRC, - layout: { 'text-field': ['get', 'label'], 'text-font': ['Open Sans Regular', 'Arial Unicode MS Regular'], 'text-size': 10, 'text-offset': [0, 1.6], 'text-anchor': 'top', 'text-max-width': 8 }, - paint: { 'text-color': '#e0f7fa', 'text-halo-color': '#050505', 'text-halo-width': 1 }, - }) - map.on('click', LAYER, handleClick) - map.on('mouseenter', LAYER, handleEnter) - map.on('mouseleave', LAYER, handleLeave) - } - - return () => { - map.off('click', LAYER, handleClick) - map.off('mouseenter', LAYER, handleEnter) - map.off('mouseleave', LAYER, handleLeave) - if (map.getLayer(LABEL)) map.removeLayer(LABEL) - if (map.getLayer(LAYER)) map.removeLayer(LAYER) - if (map.getLayer(RING)) map.removeLayer(RING) - if (map.getSource(SRC)) map.removeSource(SRC) - } - }, [map, selectEntity]) - - useEffect(() => { - const src = map.getSource(SRC) as maplibregl.GeoJSONSource | undefined - if (!src) return - const visible = gaugesVisible ? displayGauges : [] - src.setData({ - type: 'FeatureCollection', - features: visible.map((g) => { - const ident = (g.identity ?? {}) as Record - const flow = typeof ident.flow_cfs === 'number' ? ident.flow_cfs : null - const height = typeof ident.height_ft === 'number' ? ident.height_ft : null - const stage = typeof ident.stage === 'string' ? ident.stage : 'unknown' - const color = STAGE_COLOR[stage] ?? STAGE_COLOR.unknown - let label = g.display_name ?? g.entity_id - if (flow !== null) label += `\n${Math.round(flow)} cfs` - else if (height !== null) label += `\n${height.toFixed(1)} ft` - return { - type: 'Feature', - geometry: { type: 'Point', coordinates: [g.lon!, g.lat!] }, - properties: { id: g.entity_id, name: g.display_name ?? g.entity_id, stage, color, label, flow: flow ?? '', height: height ?? '' }, - } - }), - }) - }, [displayGauges, map, gaugesVisible]) - - return null -} diff --git a/frontend/src/components/layout/AlertStatusBar.tsx b/frontend/src/components/layout/AlertStatusBar.tsx index e010701..c7ca12e 100644 --- a/frontend/src/components/layout/AlertStatusBar.tsx +++ b/frontend/src/components/layout/AlertStatusBar.tsx @@ -1,5 +1,5 @@ import { useMemo } from 'react' -import { useCivicStore } from '../../store' +import { useCivicPick } from '../../store' type Level = 'green' | 'yellow' | 'red' @@ -44,7 +44,7 @@ const LEVEL_ICONS: Record = { } export function AlertStatusBar() { - const { mode, alerts, weather, setActiveTab } = useCivicStore() + const { mode, alerts, weather, setActiveTab } = useCivicPick('mode', 'alerts', 'weather', 'setActiveTab') const hasEmergency = weather.alerts.some( (a) => a.severity === 'Extreme' || a.severity === 'Severe' diff --git a/frontend/src/components/layout/EnvBar.tsx b/frontend/src/components/layout/EnvBar.tsx index ddaea84..9f6691c 100644 --- a/frontend/src/components/layout/EnvBar.tsx +++ b/frontend/src/components/layout/EnvBar.tsx @@ -1,4 +1,4 @@ -import { useCivicStore } from '../../store' +import { useCivicPick } from '../../store' import { DEFAULT_CENTER } from '../../config' function aqiColor(aqi: number | undefined): string { @@ -24,7 +24,7 @@ function formatCoord(val: number, pos: string, neg: string): string { } export function EnvBar() { - const { weather, mode } = useCivicStore() + const { weather, mode } = useCivicPick('weather', 'mode') const [centerLon, centerLat] = DEFAULT_CENTER // In critical mode, highlight the entire bar if severe alerts exist diff --git a/frontend/src/components/layout/Header.tsx b/frontend/src/components/layout/Header.tsx index a916444..bf290e2 100644 --- a/frontend/src/components/layout/Header.tsx +++ b/frontend/src/components/layout/Header.tsx @@ -1,5 +1,5 @@ import { useState, useRef, useEffect } from 'react' -import { useCivicStore, SystemEvent, NavTab } from '../../store' +import { SystemEvent, NavTab, useCivicPick } from '../../store' import { exportDashboardSnapshot } from '../../snapshotExport' const TABS: { id: NavTab; label: string; icon: string }[] = [ @@ -50,7 +50,7 @@ function NotificationsDropdown({ events, onClose }: { events: SystemEvent[]; onC } export function Header() { - const { activeTab, setActiveTab, mode, setSettingsOpen, setHelpOpen, systemEvents } = useCivicStore() + const { activeTab, setActiveTab, mode, setSettingsOpen, setHelpOpen, systemEvents } = useCivicPick('activeTab', 'setActiveTab', 'mode', 'setSettingsOpen', 'setHelpOpen', 'systemEvents') const [notificationsOpen, setNotificationsOpen] = useState(false) const notifRef = useRef(null) diff --git a/frontend/src/components/layout/MobileNav.tsx b/frontend/src/components/layout/MobileNav.tsx index 5160522..69ddfe3 100644 --- a/frontend/src/components/layout/MobileNav.tsx +++ b/frontend/src/components/layout/MobileNav.tsx @@ -1,5 +1,5 @@ import { useState } from 'react' -import { useCivicStore, NavTab } from '../../store' +import { NavTab, useCivicPick } from '../../store' import { exportDashboardSnapshot } from '../../snapshotExport' const PRIMARY_TABS: { id: NavTab; label: string; icon: string }[] = [ @@ -17,7 +17,7 @@ const SECONDARY_TABS: { id: NavTab; label: string; icon: string }[] = [ ] export function MobileNav() { - const { activeTab, setActiveTab, setSettingsOpen, setHelpOpen } = useCivicStore() + const { activeTab, setActiveTab, setSettingsOpen, setHelpOpen } = useCivicPick('activeTab', 'setActiveTab', 'setSettingsOpen', 'setHelpOpen') const [showMore, setShowMore] = useState(false) const handleTabClick = (id: NavTab) => { diff --git a/frontend/src/components/layout/SettingsPanel.tsx b/frontend/src/components/layout/SettingsPanel.tsx index 1bb1c60..8530015 100644 --- a/frontend/src/components/layout/SettingsPanel.tsx +++ b/frontend/src/components/layout/SettingsPanel.tsx @@ -1,5 +1,5 @@ import { useEffect, useState, useCallback } from 'react' -import { useCivicStore } from '../../store' +import { useCivicPick } from '../../store' import { notificationPermission, requestNotificationPermission } from '../../notifications' import { getUserRole, clearToken, authHeaders } from '../../auth' import { API_BASE } from '../../config' @@ -8,26 +8,45 @@ import { AlertRulesSection } from './AlertRulesSection' export function SettingsPanel() { const { - settingsOpen, setSettingsOpen, - radarVisible, setRadarVisible, - radarOpacity, setRadarOpacity, - smokeVisible, setSmokeVisible, - goesVisible, setGoesVisible, - firePerimetersVisible, setFirePerimetersVisible, - camerasVisible, setCamerasVisible, - geofencesVisible, setGeofencesVisible, - trailsVisible, setTrailsVisible, - lightningVisible, setLightningVisible, - radarReflectivityVisible, setRadarReflectivityVisible, - nwsAlertsVisible, setNwsAlertsVisible, - lightningDensityVisible, setLightningDensityVisible, - railTracksVisible, setRailTracksVisible, - gaugesVisible, setGaugesVisible, - terrainEnabled, setTerrainEnabled, - terrainExaggeration, setTerrainExaggeration, - entityFilter, setEntityFilter, - debugInsets, setDebugInsets, - } = useCivicStore() + settingsOpen, + setSettingsOpen, + radarVisible, + setRadarVisible, + radarOpacity, + setRadarOpacity, + smokeVisible, + setSmokeVisible, + goesVisible, + setGoesVisible, + firePerimetersVisible, + setFirePerimetersVisible, + camerasVisible, + setCamerasVisible, + geofencesVisible, + setGeofencesVisible, + trailsVisible, + setTrailsVisible, + lightningVisible, + setLightningVisible, + radarReflectivityVisible, + setRadarReflectivityVisible, + nwsAlertsVisible, + setNwsAlertsVisible, + lightningDensityVisible, + setLightningDensityVisible, + railTracksVisible, + setRailTracksVisible, + gaugesVisible, + setGaugesVisible, + terrainEnabled, + setTerrainEnabled, + terrainExaggeration, + setTerrainExaggeration, + entityFilter, + setEntityFilter, + debugInsets, + setDebugInsets, + } = useCivicPick('settingsOpen', 'setSettingsOpen', 'radarVisible', 'setRadarVisible', 'radarOpacity', 'setRadarOpacity', 'smokeVisible', 'setSmokeVisible', 'goesVisible', 'setGoesVisible', 'firePerimetersVisible', 'setFirePerimetersVisible', 'camerasVisible', 'setCamerasVisible', 'geofencesVisible', 'setGeofencesVisible', 'trailsVisible', 'setTrailsVisible', 'lightningVisible', 'setLightningVisible', 'radarReflectivityVisible', 'setRadarReflectivityVisible', 'nwsAlertsVisible', 'setNwsAlertsVisible', 'lightningDensityVisible', 'setLightningDensityVisible', 'railTracksVisible', 'setRailTracksVisible', 'gaugesVisible', 'setGaugesVisible', 'terrainEnabled', 'setTerrainEnabled', 'terrainExaggeration', 'setTerrainExaggeration', 'entityFilter', 'setEntityFilter', 'debugInsets', 'setDebugInsets') const [notifPermission, setNotifPermission] = useState(() => notificationPermission()) const userRole = getUserRole() diff --git a/frontend/src/components/layout/Sidebar.tsx b/frontend/src/components/layout/Sidebar.tsx index 7f4d3f9..a1da256 100644 --- a/frontend/src/components/layout/Sidebar.tsx +++ b/frontend/src/components/layout/Sidebar.tsx @@ -1,5 +1,5 @@ import { useEffect, useState } from 'react' -import { useCivicStore, ALT_RANGE_DEFAULT, SPD_RANGE_DEFAULT, type EntityTypeFilter } from '../../store' +import { ALT_RANGE_DEFAULT, SPD_RANGE_DEFAULT, type EntityTypeFilter, useCivicPick } from '../../store' import { isMajorTrafficIncident } from '../../incidentUtils' const INCIDENTS_COLLAPSE_KEY = 'vertex.sidebar.incidentsCollapsed' @@ -122,7 +122,7 @@ export function Sidebar() { setGaugesVisible, lightningVisible, setLightningVisible, - } = useCivicStore() + } = useCivicPick('alerts', 'news', 'health', 'entities', 'connected', 'cameras', 'weather', 'trafficIncidents', 'lightningStrikes', 'setActiveTab', 'entityFilter', 'setEntityFilter', 'setEntitySearchQuery', 'setEntityAltRange', 'setEntitySpeedRange', 'camerasVisible', 'setCamerasVisible', 'gaugesVisible', 'setGaugesVisible', 'lightningVisible', 'setLightningVisible') const entityList = Object.values(entities) const aircraft = entityList.filter((e) => e.entity_type === 'aircraft').length diff --git a/frontend/src/components/panels/CameraModal.tsx b/frontend/src/components/panels/CameraModal.tsx index d41689a..9ba02b9 100644 --- a/frontend/src/components/panels/CameraModal.tsx +++ b/frontend/src/components/panels/CameraModal.tsx @@ -1,11 +1,14 @@ -import { useCivicStore, TrafficCamera } from '../../store' +import { TrafficCamera, useCivicPick } from '../../store' export function CameraModal() { - const { - selectedCamId, setSelectedCamId, - cameras, ldiMode, - favoriteCamIds, toggleFavoriteCam - } = useCivicStore() + const { + selectedCamId, + setSelectedCamId, + cameras, + ldiMode, + favoriteCamIds, + toggleFavoriteCam, + } = useCivicPick('selectedCamId', 'setSelectedCamId', 'cameras', 'ldiMode', 'favoriteCamIds', 'toggleFavoriteCam') if (!selectedCamId) return null diff --git a/frontend/src/components/panels/CommsPanel.tsx b/frontend/src/components/panels/CommsPanel.tsx index 2bc893f..d07b7a1 100644 --- a/frontend/src/components/panels/CommsPanel.tsx +++ b/frontend/src/components/panels/CommsPanel.tsx @@ -1,5 +1,5 @@ import { useState, useMemo } from 'react' -import { useCivicStore, MeshMessage, Entity, SystemEvent, Track, MeshLink } from '../../store' +import { MeshMessage, Entity, SystemEvent, Track, MeshLink, useCivicPick } from '../../store' import { getDistanceMeters } from '../../layers/geoUtils' import { DEFAULT_CENTER, API_BASE } from '../../config' import { MeshFleetPanel } from './MeshFleetPanel' @@ -161,7 +161,7 @@ function formatAge(iso: string | undefined): string { } function SpectralMonitor({ links, history, status }: { links: MeshLink[]; history: Record; status: any }) { - const { radio, entities } = useCivicStore() + const { radio, entities } = useCivicPick('radio', 'entities') // Calculate top 3 links by SNR const topLinks = useMemo(() => { @@ -371,7 +371,7 @@ function SpectralMonitor({ links, history, status }: { links: MeshLink[]; histor } export function CommsPanel() { - const { radio, meshMessages, entities, systemEvents, tracks, meshLinks, linkHistory, meshStatus } = useCivicStore() + const { radio, meshMessages, entities, systemEvents, tracks, meshLinks, linkHistory, meshStatus } = useCivicPick('radio', 'meshMessages', 'entities', 'systemEvents', 'tracks', 'meshLinks', 'linkHistory', 'meshStatus') const [msgFilter, setMsgFilter] = useState('') const [selectedConv, setSelectedConv] = useState('all') const [activeTab, setActiveTab] = useState<'chat' | 'fleet' | 'p25'>('chat') diff --git a/frontend/src/components/panels/CustomLayersTab.tsx b/frontend/src/components/panels/CustomLayersTab.tsx index e7632df..25cc5d5 100644 --- a/frontend/src/components/panels/CustomLayersTab.tsx +++ b/frontend/src/components/panels/CustomLayersTab.tsx @@ -1,5 +1,5 @@ import { useState, useCallback, useEffect, useRef } from 'react' -import { useCivicStore, CustomLayerItem } from '../../store' +import { CustomLayerItem, useCivicPick } from '../../store' import { API_BASE } from '../../config' import { authHeaders } from '../../auth' @@ -58,7 +58,7 @@ function kmlToGeoJson(kmlText: string): object { } export function CustomLayersTab() { - const { customLayers, setCustomLayers } = useCivicStore() + const { customLayers, setCustomLayers } = useCivicPick('customLayers', 'setCustomLayers') const fileInputRef = useRef(null) const [layerImportName, setLayerImportName] = useState('') diff --git a/frontend/src/components/panels/EntityDetail.tsx b/frontend/src/components/panels/EntityDetail.tsx index 1de2cf9..91a4abb 100644 --- a/frontend/src/components/panels/EntityDetail.tsx +++ b/frontend/src/components/panels/EntityDetail.tsx @@ -1,5 +1,5 @@ import { useEffect, useState, useCallback } from 'react' -import { useCivicStore } from '../../store' +import { useCivicPick } from '../../store' import type { EntityMissionTag } from '../../store' import { API_BASE } from '../../config' import { authHeaders } from '../../auth' @@ -45,13 +45,24 @@ const TYPE_ICONS: Record = { const TAG_PRESETS = ['#FF4444', '#FF8800', '#FFB800', '#44DD88', '#00BBFF', '#AA44FF', '#FF44AA'] +// MeshCore LoRa SNR spans roughly -20..+12 dB. +const snrColor = (snr: number | null): string => + snr === null ? '#999' : snr >= 5 ? '#44dd88' : snr >= -10 ? '#ffb800' : '#ff5050' + export function EntityDetail() { const { - entities, airports, selectedEntityId, selectEntity, - entityMissionTags, setEntityMissionTags, addEntityMissionTag, removeEntityMissionTag, - } = useCivicStore() + entities, + airports, + selectedEntityId, + selectEntity, + entityMissionTags, + setEntityMissionTags, + addEntityMissionTag, + removeEntityMissionTag, + } = useCivicPick('entities', 'airports', 'selectedEntityId', 'selectEntity', 'entityMissionTags', 'setEntityMissionTags', 'addEntityMissionTag', 'removeEntityMissionTag') + const { meshLinks, meshStatus } = useCivicPick('meshLinks', 'meshStatus') const entity = selectedEntityId ? entities[selectedEntityId] : null // Fetch trail for sparklines (aircraft only) @@ -71,11 +82,14 @@ export function EntityDetail() { return () => { cancelled = true } }, [selectedEntityId, entity?.entity_type]) - // Fetch mesh neighbors for mesh_node entities - const [meshNeighbors, setMeshNeighbors] = useState([]) + // Mesh neighbors for mesh_node entities: REST gives the persisted 60-min + // window; the live meshLinks store array (fed by WS mesh_links events) + // overlays it so fresh SNR readings appear without waiting on a refetch — + // and the list still populates when the REST call fails entirely. + const [restNeighbors, setRestNeighbors] = useState([]) useEffect(() => { if (!selectedEntityId || !entity || entity.entity_type !== 'mesh_node') { - setMeshNeighbors([]) + setRestNeighbors([]) return } let cancelled = false @@ -83,7 +97,7 @@ export function EntityDetail() { .then((r) => r.ok ? r.json() : []) .then((links: MeshNeighbor[]) => { if (!cancelled) { - setMeshNeighbors( + setRestNeighbors( links.filter( (l) => l.node_a === selectedEntityId || l.node_b === selectedEntityId ) @@ -94,6 +108,18 @@ export function EntityDetail() { return () => { cancelled = true } }, [selectedEntityId, entity?.entity_type]) + const meshNeighbors: MeshNeighbor[] = (() => { + if (!selectedEntityId || entity?.entity_type !== 'mesh_node') return [] + const byPair = new Map() + for (const l of restNeighbors) byPair.set(`${l.node_a}|${l.node_b}`, l) + for (const l of meshLinks) { + if (l.node_a === selectedEntityId || l.node_b === selectedEntityId) { + byPair.set(`${l.node_a}|${l.node_b}`, l) // live reading wins + } + } + return Array.from(byPair.values()) + })() + // Load mission tags when entity changes const loadTags = useCallback(async (entityId: string) => { try { @@ -285,10 +311,8 @@ export function EntityDetail() { {nodeSnr != null && (
Node SNR - = -70 ? '#44dd88' : nodeSnr >= -90 ? '#ffb800' : '#ff5050', - }}> - {nodeSnr} dBm + + {nodeSnr} dB
)} @@ -303,19 +327,31 @@ export function EntityDetail() {
    {meshNeighbors.map((lnk, i) => { const peerId = lnk.node_a === selectedEntityId ? lnk.node_b : lnk.node_a + const peer = entities[peerId] + const peerName = peerId === 'local' + ? (meshStatus?.site_name ?? 'Repeater (local)') + : (peer?.display_name ?? peerId) return (
  • - {peerId} + {peer ? ( + + ) : ( + + {peerName} + + )} = -70 ? '#44dd88' - : lnk.snr >= -90 ? '#ffb800' - : '#ff5050', - }} + style={{ color: snrColor(lnk.snr) }} > - {lnk.snr !== null ? `${lnk.snr} dBm` : '—'} + {lnk.snr !== null ? `${lnk.snr} dB` : '—'}
    • ) diff --git a/frontend/src/components/panels/EntitySearchPanel.tsx b/frontend/src/components/panels/EntitySearchPanel.tsx index ee3a142..be80a53 100644 --- a/frontend/src/components/panels/EntitySearchPanel.tsx +++ b/frontend/src/components/panels/EntitySearchPanel.tsx @@ -1,5 +1,5 @@ import { useState } from 'react' -import { useCivicStore, ALT_RANGE_DEFAULT, SPD_RANGE_DEFAULT } from '../../store' +import { ALT_RANGE_DEFAULT, SPD_RANGE_DEFAULT, useCivicPick } from '../../store' import { getDistanceMeters } from '../../layers/geoUtils' import { DEFAULT_CENTER } from '../../config' @@ -69,15 +69,22 @@ function RangeSlider({ export function EntitySearchPanel() { const { - tracks, entities, - entitySearchQuery, setEntitySearchQuery, - entityAltRange, setEntityAltRange, - entitySpeedRange, setEntitySpeedRange, - entityFilter, setEntityFilter, - trailsVisible, setTrailsVisible, - selectEntity, selectedEntityId, + tracks, + entities, + entitySearchQuery, + setEntitySearchQuery, + entityAltRange, + setEntityAltRange, + entitySpeedRange, + setEntitySpeedRange, + entityFilter, + setEntityFilter, + trailsVisible, + setTrailsVisible, + selectEntity, + selectedEntityId, entityMissionTags, - } = useCivicStore() + } = useCivicPick('tracks', 'entities', 'entitySearchQuery', 'setEntitySearchQuery', 'entityAltRange', 'setEntityAltRange', 'entitySpeedRange', 'setEntitySpeedRange', 'entityFilter', 'setEntityFilter', 'trailsVisible', 'setTrailsVisible', 'selectEntity', 'selectedEntityId', 'entityMissionTags') const [filtersOpen, setFiltersOpen] = useState(false) const [taggedOnly, setTaggedOnly] = useState(false) diff --git a/frontend/src/components/panels/EventLogPanel.tsx b/frontend/src/components/panels/EventLogPanel.tsx index 84348a3..7450248 100644 --- a/frontend/src/components/panels/EventLogPanel.tsx +++ b/frontend/src/components/panels/EventLogPanel.tsx @@ -1,5 +1,5 @@ import { useEffect, useState } from 'react' -import { useCivicStore, SystemEvent } from '../../store' +import { SystemEvent, useCivicPick } from '../../store' import { API_BASE } from '../../config' import { authHeaders, clearToken } from '../../auth' @@ -94,7 +94,7 @@ function EventRow({ event }: { event: SystemEvent }) { } export function EventLogPanel() { - const { systemEvents, setSystemEvents } = useCivicStore() + const { systemEvents, setSystemEvents } = useCivicPick('systemEvents', 'setSystemEvents') const [severityFilter, setSeverityFilter] = useState('all') const [search, setSearch] = useState('') const [sitrepHours, setSitrepHours] = useState(24) diff --git a/frontend/src/components/panels/FlightLogPanel.tsx b/frontend/src/components/panels/FlightLogPanel.tsx index 52a2ebb..0419d7f 100644 --- a/frontend/src/components/panels/FlightLogPanel.tsx +++ b/frontend/src/components/panels/FlightLogPanel.tsx @@ -1,6 +1,6 @@ import { useState, useMemo, useEffect, useCallback, useRef } from 'react' import maplibregl from 'maplibre-gl' -import { useCivicStore, Entity } from '../../store' +import { Entity, useCivicPick } from '../../store' import type { AcarsMessage } from '../../storeTypes' import { API_BASE, MAP_STYLE, DEFAULT_CENTER } from '../../config' import { authHeaders } from '../../auth' @@ -498,7 +498,7 @@ function FlightMiniMap({ trailPoints, entity }: { // ─── Main Panel ─────────────────────────────────────────────────────────────── export function FlightLogPanel() { - const { entities, selectedEntityId, selectEntity, acarsMessages } = useCivicStore() + const { entities, selectedEntityId, selectEntity, acarsMessages } = useCivicPick('entities', 'selectedEntityId', 'selectEntity', 'acarsMessages') const [timeWindow, setTimeWindow] = useState(60) const [search, setSearch] = useState('') diff --git a/frontend/src/components/panels/GeofencePanel.tsx b/frontend/src/components/panels/GeofencePanel.tsx index 2461cf9..fbe3ee5 100644 --- a/frontend/src/components/panels/GeofencePanel.tsx +++ b/frontend/src/components/panels/GeofencePanel.tsx @@ -1,5 +1,5 @@ import { useState, useEffect, useCallback } from 'react' -import { useCivicStore } from '../../store' +import { useCivicPick } from '../../store' import { API_BASE } from '../../config' import { authHeaders } from '../../auth' import { CustomLayersTab } from './CustomLayersTab' @@ -36,11 +36,14 @@ type PanelTab = 'geofences' | 'layers' export function GeofencePanel() { const { - geofenceDrawing, setGeofenceDrawing, - geofenceDrawMode, setGeofenceDrawMode, - geofenceDrawPoints, clearGeofenceDrawPoints, + geofenceDrawing, + setGeofenceDrawing, + geofenceDrawMode, + setGeofenceDrawMode, + geofenceDrawPoints, + clearGeofenceDrawPoints, customLayers, - } = useCivicStore() + } = useCivicPick('geofenceDrawing', 'setGeofenceDrawing', 'geofenceDrawMode', 'setGeofenceDrawMode', 'geofenceDrawPoints', 'clearGeofenceDrawPoints', 'customLayers') const [panelTab, setPanelTab] = useState('geofences') const [fences, setFences] = useState([]) diff --git a/frontend/src/components/panels/HelpPanel.tsx b/frontend/src/components/panels/HelpPanel.tsx index d41469b..8d8b20f 100644 --- a/frontend/src/components/panels/HelpPanel.tsx +++ b/frontend/src/components/panels/HelpPanel.tsx @@ -1,11 +1,11 @@ import { useEffect, useState } from 'react' -import { useCivicStore } from '../../store' +import { useCivicPick } from '../../store' import { DOC_PAGES, type DocPage } from '../../docs/content' const SECTIONS = ['Usage', 'Capabilities', 'Dashboards', 'Navigation'] as const export function HelpPanel() { - const { helpOpen, setHelpOpen } = useCivicStore() + const { helpOpen, setHelpOpen } = useCivicPick('helpOpen', 'setHelpOpen') const [activePage, setActivePage] = useState(DOC_PAGES[0]) const [mobileNavOpen, setMobileNavOpen] = useState(false) diff --git a/frontend/src/components/panels/IncidentsPanel.tsx b/frontend/src/components/panels/IncidentsPanel.tsx index aae5583..9708b13 100644 --- a/frontend/src/components/panels/IncidentsPanel.tsx +++ b/frontend/src/components/panels/IncidentsPanel.tsx @@ -1,5 +1,5 @@ import { useEffect } from 'react' -import { useCivicStore, WeatherAlert, SystemEvent } from '../../store' +import { WeatherAlert, SystemEvent, useCivicPick } from '../../store' import { isMajorTrafficIncident, isIncidentInRadius } from '../../incidentUtils' import ReactMarkdown from 'react-markdown' import { API_BASE } from '../../config' @@ -37,7 +37,7 @@ function deriveIncidentTitle(incident: { } function AiTrafficSummary() { - const { summary } = useCivicStore() + const { summary } = useCivicPick('summary') if (!summary.summary) return null return ( @@ -100,7 +100,7 @@ function sysSeverityColorClass(severity: string) { } export function IncidentsPanel() { - const { weather, trafficIncidents, systemEvents } = useCivicStore() + const { weather, trafficIncidents, systemEvents } = useCivicPick('weather', 'trafficIncidents', 'systemEvents') // Request an on-demand AI summary refresh whenever this panel is opened. // The updated result arrives via the existing WebSocket → store flow. diff --git a/frontend/src/components/panels/InfrastructureGrid.tsx b/frontend/src/components/panels/InfrastructureGrid.tsx index 8f1a759..fe5a93d 100644 --- a/frontend/src/components/panels/InfrastructureGrid.tsx +++ b/frontend/src/components/panels/InfrastructureGrid.tsx @@ -1,5 +1,5 @@ import { useState, useEffect } from 'react' -import { useCivicStore, TrafficCamera } from '../../store' +import { TrafficCamera, useCivicPick } from '../../store' import { isMajorTrafficIncident } from '../../incidentUtils' function CctvThumbnail({ @@ -141,10 +141,18 @@ const PLACEHOLDER_CAMERAS: TrafficCamera[] = [ export function InfrastructureGrid() { const { - cameras, trafficFlow, trafficIncidents, utilityStatus, oregonStatus, ldiMode, setLdiMode, - selectedCamId, setSelectedCamId, - favoriteCamIds, toggleFavoriteCam, - } = useCivicStore() + cameras, + trafficFlow, + trafficIncidents, + utilityStatus, + oregonStatus, + ldiMode, + setLdiMode, + selectedCamId, + setSelectedCamId, + favoriteCamIds, + toggleFavoriteCam, + } = useCivicPick('cameras', 'trafficFlow', 'trafficIncidents', 'utilityStatus', 'oregonStatus', 'ldiMode', 'setLdiMode', 'selectedCamId', 'setSelectedCamId', 'favoriteCamIds', 'toggleFavoriteCam') const [radiusKm, setRadiusKm] = useState(5) const [page, setPage] = useState(0) const PAGE_SIZE = 12 diff --git a/frontend/src/components/panels/IntelPanel.tsx b/frontend/src/components/panels/IntelPanel.tsx index d6aa305..9802e89 100644 --- a/frontend/src/components/panels/IntelPanel.tsx +++ b/frontend/src/components/panels/IntelPanel.tsx @@ -1,4 +1,4 @@ -import { useCivicStore, AlertItem, NewsItem } from '../../store' +import { AlertItem, NewsItem, useCivicPick } from '../../store' type FeedItem = { key: string @@ -106,7 +106,7 @@ function toFeedItem(item: AlertItem | NewsItem, i: number, isAlert: boolean): Fe } export function IntelPanel() { - const { alerts, news } = useCivicStore() + const { alerts, news } = useCivicPick('alerts', 'news') // Merge and sort by published date descending const allItems = [ diff --git a/frontend/src/components/panels/PlaybackController.tsx b/frontend/src/components/panels/PlaybackController.tsx index a78892d..d2ba2ef 100644 --- a/frontend/src/components/panels/PlaybackController.tsx +++ b/frontend/src/components/panels/PlaybackController.tsx @@ -1,5 +1,5 @@ import { useState, useEffect, useRef, useCallback } from 'react' -import { useCivicStore, ReplayEvent } from '../../store' +import { ReplayEvent, useCivicPick } from '../../store' import { API_BASE } from '../../config' import { authHeaders } from '../../auth' @@ -29,12 +29,17 @@ function toDatetimeLocal(d: Date): string { export function PlaybackController() { const { - replayMode, setReplayMode, - replayData, setReplayData, - replayCurrentTs, setReplayCurrentTs, - replayPlaying, setReplayPlaying, - replaySpeed, setReplaySpeed, - } = useCivicStore() + replayMode, + setReplayMode, + replayData, + setReplayData, + replayCurrentTs, + setReplayCurrentTs, + replayPlaying, + setReplayPlaying, + replaySpeed, + setReplaySpeed, + } = useCivicPick('replayMode', 'setReplayMode', 'replayData', 'setReplayData', 'replayCurrentTs', 'setReplayCurrentTs', 'replayPlaying', 'setReplayPlaying', 'replaySpeed', 'setReplaySpeed') const [windowHours, setWindowHours] = useState(2) const [useAbsolute, setUseAbsolute] = useState(false) diff --git a/frontend/src/hooks/useAlerts.ts b/frontend/src/hooks/useAlerts.ts index 946e560..e545827 100644 --- a/frontend/src/hooks/useAlerts.ts +++ b/frontend/src/hooks/useAlerts.ts @@ -1,6 +1,6 @@ import { useEffect, useRef } from 'react' import { API_BASE, ALERTS_POLL_MS, NEWS_POLL_MS, WEATHER_POLL_MS, CAMERAS_POLL_MS } from '../config' -import { useCivicStore } from '../store' +import { useCivicPick } from '../store' import { authHeaders, clearToken } from '../auth' import type { TrafficFlowSensor, UtilityStatus, OregonStatus } from '../storeTypes' @@ -26,7 +26,7 @@ export function useAlerts() { setUtilityStatus, setOregonStatus, setSummary, - } = useCivicStore() + } = useCivicPick('setAlerts', 'setNews', 'setWeather', 'setCameras', 'setTrafficFlow', 'setTrafficIncidents', 'setUtilityStatus', 'setOregonStatus', 'setSummary') const timers = useRef[]>([]) useEffect(() => { diff --git a/frontend/src/hooks/useMeshHistory.ts b/frontend/src/hooks/useMeshHistory.ts index bb85efd..1573f1b 100644 --- a/frontend/src/hooks/useMeshHistory.ts +++ b/frontend/src/hooks/useMeshHistory.ts @@ -1,10 +1,10 @@ import { useEffect } from 'react' -import { useCivicStore } from '../store' +import { useCivicPick } from '../store' import { API_BASE } from '../config' import { authHeaders } from '../auth' export function useMeshHistory() { - const { setMeshLinks, setMeshMessages, setMeshStatus, connected } = useCivicStore() + const { setMeshLinks, setMeshMessages, setMeshStatus, connected } = useCivicPick('setMeshLinks', 'setMeshMessages', 'setMeshStatus', 'connected') useEffect(() => { if (!connected) return diff --git a/frontend/src/hooks/useMeshLinks.ts b/frontend/src/hooks/useMeshLinks.ts index 5a8d8c4..4221142 100644 --- a/frontend/src/hooks/useMeshLinks.ts +++ b/frontend/src/hooks/useMeshLinks.ts @@ -1,10 +1,10 @@ import { useEffect } from 'react' -import { useCivicStore } from '../store' +import { useCivicPick } from '../store' import { API_BASE } from '../config' import { authHeaders } from '../auth' export function useMeshLinks() { - const { setMeshLinks, connected } = useCivicStore() + const { setMeshLinks, connected } = useCivicPick('setMeshLinks', 'connected') useEffect(() => { if (!connected) return diff --git a/frontend/src/hooks/useSystemHealth.ts b/frontend/src/hooks/useSystemHealth.ts index b48e7e7..4337c46 100644 --- a/frontend/src/hooks/useSystemHealth.ts +++ b/frontend/src/hooks/useSystemHealth.ts @@ -1,9 +1,9 @@ import { useEffect, useRef } from 'react' import { HEALTH_POLL_MS } from '../config' -import { useCivicStore } from '../store' +import { useCivicPick } from '../store' export function useSystemHealth() { - const { setHealth } = useCivicStore() + const { setHealth } = useCivicPick('setHealth') const timer = useRef | null>(null) useEffect(() => { diff --git a/frontend/src/hooks/useWebSocket.ts b/frontend/src/hooks/useWebSocket.ts index 9e1b898..90f6928 100644 --- a/frontend/src/hooks/useWebSocket.ts +++ b/frontend/src/hooks/useWebSocket.ts @@ -1,12 +1,19 @@ import { useEffect, useRef } from 'react' import { WS_URL, API_BASE } from '../config' -import { useCivicStore } from '../store' +import { useCivicStore, useCivicPick } from '../store' import type { Entity, EntityTypeFilter } from '../storeTypes' import { wsTokenParam, authHeaders } from '../auth' import { initNotifications, maybeNotify, notifyMeshMessage } from '../notifications' const RECONNECT_DELAY_INITIAL_MS = 1000 const RECONNECT_DELAY_MAX_MS = 60_000 +// Entity updates are buffered and applied in one store commit per flush so a +// busy feed produces ~4 store notifications/sec instead of one per WS message. +// PVB interpolates icon motion between flushes, so this adds no visible lag. +const ENTITY_FLUSH_MS = 250 +// Cold-start REST seed page size — the backend defaults to 200, which can be +// a partial snapshot on a busy feed. +const ENTITY_SEED_LIMIT = 2000 // All known entity types tracked by EntityTypeFilter keys. // adsbLocal / adsbSupplement are frontend sub-filters of 'aircraft'; the @@ -73,13 +80,14 @@ function buildSubscription( export function useWebSocket() { const wsRef = useRef(null) + const entityBufferRef = useRef([]) const reconnectDelayRef = useRef(RECONNECT_DELAY_INITIAL_MS) const emptyAircraftSnapshotStreakRef = useRef(0) const degradedAircraftSnapshotStreakRef = useRef(0) const { setEntities, setAircraftSnapshot, - upsertEntity, + upsertEntities, purgeStaleEntities, setConnected, setRadio, @@ -99,7 +107,7 @@ export function useWebSocket() { updateLinkHistory, setMeshStatus, appendAcarsMessage, - } = useCivicStore() + } = useCivicPick('setEntities', 'setAircraftSnapshot', 'upsertEntities', 'purgeStaleEntities', 'setConnected', 'setRadio', 'appendSystemEvent', 'setUtilityStatus', 'setOregonStatus', 'setAirports', 'setWeather', 'setAlerts', 'setNews', 'setCameras', 'setTrafficFlow', 'setTrafficIncidents', 'setSummary', 'appendLightningStrikes', 'appendMeshMessage', 'updateLinkHistory', 'setMeshStatus', 'appendAcarsMessage') useEffect(() => { let cancelled = false @@ -111,7 +119,7 @@ export function useWebSocket() { // seconds). Seed the store immediately over REST so the map renders right // away. Guarded on an empty store so it never clobbers live WS data that // may have already arrived. - fetch(`${API_BASE}/entities`, { headers: authHeaders() }) + fetch(`${API_BASE}/entities?limit=${ENTITY_SEED_LIMIT}`, { headers: authHeaders() }) .then((r) => (r.ok ? r.json() : null)) .then((list: Entity[] | null) => { if (cancelled || !Array.isArray(list) || list.length === 0) return @@ -124,6 +132,13 @@ export function useWebSocket() { purgeStaleEntities() }, 10000) + const flushInterval = setInterval(() => { + if (entityBufferRef.current.length === 0) return + const batch = entityBufferRef.current + entityBufferRef.current = [] + upsertEntities(batch) + }, ENTITY_FLUSH_MS) + const sendSubscription = (ws: WebSocket) => { if (ws.readyState !== WebSocket.OPEN) return const state = useCivicStore.getState() @@ -167,7 +182,7 @@ export function useWebSocket() { setEntities(msg.data as Parameters[0]) break case 'entity_update': - upsertEntity(msg.data as Parameters[0]) + entityBufferRef.current.push(msg.data as Entity) break case 'aircraft_snapshot': { if (msgData?.schema_version !== undefined && msgData.schema_version !== 1) { @@ -317,6 +332,7 @@ export function useWebSocket() { return () => { cancelled = true clearInterval(cleanupInterval) + clearInterval(flushInterval) unsubscribeStore() if (wsRef.current) { wsRef.current.onopen = null diff --git a/frontend/src/layers/AnnotationLayer.tsx b/frontend/src/layers/AnnotationLayer.tsx index 2e498dc..6bec4bb 100644 --- a/frontend/src/layers/AnnotationLayer.tsx +++ b/frontend/src/layers/AnnotationLayer.tsx @@ -2,12 +2,6 @@ import { ScatterplotLayer, LineLayer, PolygonLayer, TextLayer } from '@deck.gl/l import type { Layer } from '@deck.gl/core' import type { AnnotationItem } from '../storeTypes' -interface AnnotationDrawPreview { - mode: 'marker' | 'line' | 'polygon' | null - points: [number, number][] - cursor: [number, number] | null -} - function hexToRgb(hex: string): [number, number, number] { const cleaned = hex.replace('#', '') const r = parseInt(cleaned.slice(0, 2), 16) @@ -185,61 +179,3 @@ export function buildAnnotationLayers(annotations: AnnotationItem[], visible: bo return layers } - -export function buildAnnotationDrawPreviewLayers(preview: AnnotationDrawPreview): Layer[] { - const { mode, points, cursor } = preview - if (!mode || mode === 'marker' || points.length === 0) return [] - - const previewPoints = cursor ? [...points, cursor] : points - const lineData: Array<{ sourcePosition: [number, number]; targetPosition: [number, number] }> = [] - for (let i = 0; i < previewPoints.length - 1; i++) { - lineData.push({ sourcePosition: previewPoints[i], targetPosition: previewPoints[i + 1] }) - } - - const layers: Layer[] = [] - - if (lineData.length > 0) { - layers.push( - new LineLayer({ - id: 'annotation-draw-preview-line', - data: lineData, - getSourcePosition: (d: any) => d.sourcePosition, - getTargetPosition: (d: any) => d.targetPosition, - getColor: [255, 184, 0, 230] as [number, number, number, number], - getWidth: 2, - widthUnits: 'pixels', - pickable: false, - }) - ) - } - - if (mode === 'polygon' && previewPoints.length >= 3) { - layers.push( - new PolygonLayer({ - id: 'annotation-draw-preview-fill', - data: [{ polygon: [...previewPoints, previewPoints[0]] }], - getPolygon: (d: any) => d.polygon, - getFillColor: [255, 184, 0, 28] as [number, number, number, number], - stroked: false, - pickable: false, - }) - ) - } - - layers.push( - new ScatterplotLayer({ - id: 'annotation-draw-preview-points', - data: points, - getPosition: (d: [number, number]) => d, - getRadius: 4, - radiusUnits: 'pixels', - getFillColor: [255, 184, 0, 240] as [number, number, number, number], - getLineColor: [0, 0, 0, 220] as [number, number, number, number], - getLineWidth: 1, - lineWidthUnits: 'pixels', - pickable: false, - }) - ) - - return layers -} diff --git a/frontend/src/store.ts b/frontend/src/store.ts index 69a5d16..ed796b3 100644 --- a/frontend/src/store.ts +++ b/frontend/src/store.ts @@ -1,5 +1,6 @@ import { create } from 'zustand' import { persist } from 'zustand/middleware' +import { useShallow } from 'zustand/react/shallow' import { entityToTrack, mergeEntityState, loadFavoriteCamIds } from './entityUtils' // Re-export all types so existing imports from '../../store' keep working. @@ -15,7 +16,7 @@ import type { import { ALT_RANGE_DEFAULT, SPD_RANGE_DEFAULT } from './storeTypes' // ─── Store ──────────────────────────────────────────────────────────────────── -interface CivicStore { +export interface CivicStore { // Live data entities: Record tracks: Record @@ -62,6 +63,7 @@ interface CivicStore { setEntities: (entities: Entity[]) => void setAircraftSnapshot: (entities: Entity[]) => void upsertEntity: (entity: Entity) => void + upsertEntities: (entities: Entity[]) => void purgeStaleEntities: () => void appendSystemEvent: (event: SystemEvent) => void setSystemEvents: (events: SystemEvent[]) => void @@ -204,6 +206,25 @@ interface CivicStore { setTerrainExaggeration: (v: number) => void } +/** + * Subscribe to a named subset of store keys with shallow equality. + * + * Use this instead of a bare `useCivicStore()` — a selector-less hook + * re-renders the component on EVERY store change, including the dozens of + * entity updates per second arriving over the WebSocket. Picking only the + * keys a component reads limits its re-renders to changes of those keys. + * (Action functions are stable, so picking them never causes a re-render.) + */ +export function useCivicPick(...keys: K[]): Pick { + return useCivicStore( + useShallow((s: CivicStore) => { + const out = {} as Pick + for (const k of keys) out[k] = s[k] + return out + }), + ) +} + export interface LightningStrike { lat: number lon: number @@ -451,6 +472,21 @@ export const useCivicStore = create()( tracks: track ? { ...s.tracks, [entity.entity_id]: track } : s.tracks, } }), + // Batch variant — one store notification for a whole buffer of WS updates + // instead of one per message. Order within the batch is preserved. + upsertEntities: (list) => + set((s) => { + if (list.length === 0) return {} + const entities = { ...s.entities } + const tracks = { ...s.tracks } + for (const entity of list) { + const merged = mergeEntityState(entities[entity.entity_id], entity) + entities[entity.entity_id] = merged + const track = entityToTrack(merged, tracks[entity.entity_id]) + if (track) tracks[entity.entity_id] = track + } + return { entities, tracks } + }), purgeStaleEntities: () => set((s) => { const now = Date.now() @@ -655,7 +691,19 @@ export const useCivicStore = create()( }), { name: 'vertex.ui.prefs', + // Shallow merge except entityFilter, which is deep-merged so filter keys + // added in later versions default to visible instead of vanishing for + // users with older persisted state. + merge: (persisted, current) => { + const p = (persisted ?? {}) as Partial + return { + ...current, + ...p, + entityFilter: { ...current.entityFilter, ...(p.entityFilter ?? {}) }, + } + }, partialize: (state) => ({ + entityFilter: state.entityFilter, mode: state.mode, trailsVisible: state.trailsVisible, radarVisible: state.radarVisible, diff --git a/poller/bus.py b/poller/bus.py index b848b53..d875974 100644 --- a/poller/bus.py +++ b/poller/bus.py @@ -1,5 +1,6 @@ import json import logging +import time from redis.asyncio import Redis from config import settings from sanitize import sanitize_payload @@ -7,10 +8,18 @@ logger = logging.getLogger(__name__) _redis: Redis | None = None -# In-memory mirror of the last-published entity state per entity_id. -# Eliminates the Redis GET on every publish_entity call when adsb_publish_only_changes -# is enabled — the comparison is done locally instead of via a round-trip. -_entity_cache: dict[str, dict] = {} +# In-memory mirror of the last-published entity state per entity_id, stored as +# (monotonic_last_seen, entity). Eliminates the Redis GET on every +# publish_entity call when adsb_publish_only_changes is enabled — the +# comparison is done locally instead of via a round-trip. +# +# Entries for entities that stop reporting are swept periodically; without +# eviction the cache grows unboundedly with every unique aircraft ever seen, +# a slow leak on a Pi that runs for weeks. +_entity_cache: dict[str, tuple[float, dict]] = {} +_ENTITY_CACHE_MAX_AGE_S = 600.0 +_ENTITY_CACHE_SWEEP_EVERY = 4096 +_entity_cache_ops = 0 async def get_bus() -> Redis: @@ -48,11 +57,27 @@ async def publish_entity( except Exception: pass + global _entity_cache_ops + now = time.monotonic() + should_publish = True if settings.adsb_publish_only_changes: previous = _entity_cache.get(entity_id) if previous is not None: - should_publish = _entity_changed(previous, entity) + should_publish = _entity_changed(previous[1], entity) + # Refresh the timestamp on every call so active-but-static entities + # (e.g. mesh nodes) are not evicted while still reporting. + _entity_cache[entity_id] = (now, entity) + + _entity_cache_ops += 1 + if _entity_cache_ops >= _ENTITY_CACHE_SWEEP_EVERY: + _entity_cache_ops = 0 + cutoff = now - _ENTITY_CACHE_MAX_AGE_S + stale = [k for k, (ts, _) in _entity_cache.items() if ts < cutoff] + for k in stale: + del _entity_cache[k] + if stale: + logger.debug("entity cache: evicted %d stale entries", len(stale)) # Always refresh the Redis TTL so the key stays alive while the entity is active. # ⚡ Bolt Optimization: Cache json.dumps result to avoid re-serializing large payload for the pubsub wrapper @@ -60,7 +85,6 @@ async def publish_entity( await r.set(key, payload, ex=ttl) if should_publish: - _entity_cache[entity_id] = entity await r.publish("civic:updates", f'{{"type":"entity_update","data":{payload}}}') from db import write_entity_observation # lazy import — db imports geofence which imports bus diff --git a/poller/config.py b/poller/config.py index 3510e34..0a8b77e 100644 --- a/poller/config.py +++ b/poller/config.py @@ -114,6 +114,15 @@ def regions(self) -> list[RegionConfig]: aprs_passcode: str = "-1" aprs_filter_radius_km: int = 80 + # MeshCore node gating — a pyMC-Repeater advert table covers the whole + # regional mesh (every node it has ever heard an advert for), not just + # nearby RF neighbors. When enabled, nodes advertising a position outside + # the configured region bbox(es), padded by mesh_bbox_pad_deg degrees, + # are dropped — mirroring the ADS-B/AIS/Amtrak bbox gating. Nodes with no + # advertised position always pass (they cannot clutter the map). + mesh_bbox_filter: bool = True + mesh_bbox_pad_deg: float = 0.25 + # ADS-B ingest strategy adsb_enable_beast: bool = False adsb_beast_host: str = "localhost" diff --git a/poller/normalizers/mesh_node.py b/poller/normalizers/mesh_node.py index b5465d5..b5dedcb 100644 --- a/poller/normalizers/mesh_node.py +++ b/poller/normalizers/mesh_node.py @@ -72,44 +72,5 @@ def normalize_pymc_repeater_advert(data: dict, source_url: str) -> Optional[dict } -def normalize_mesh_node(data: dict) -> Optional[dict]: - """Normalize a MeshCore bridge node_update payload to canonical Entity.""" - node_id = data.get("entity_id", "").replace("mesh_node:", "") or data.get("identity", {}).get("node_id") - if not node_id: - return None - - if data.get("entity_type") == "mesh_node": - return data # already canonical - - return { - "entity_id": f"mesh_node:{node_id}", - "entity_type": "mesh_node", - "source": "meshcore", - "display_name": data.get("name") or data.get("short_name") or node_id, - "identity": { - "node_id": node_id, - "short_name": data.get("short_name", ""), - "hw_model": data.get("hw_model") or None, - }, - "lat": data.get("lat"), - "lon": data.get("lon"), - "altitude": data.get("altitude"), - "status": _bridge_status(data), - "last_seen": data.get("timestamp") or _now(), - "tags": ["mesh_node"], - } - - -def _bridge_status(node: dict) -> str: - parts = [] - if (v := node.get("battery_pct")) is not None: - parts.append(f"bat:{v}%") - if (v := node.get("snr")) is not None: - parts.append(f"snr:{v:.1f}") - if (v := node.get("rssi")) is not None: - parts.append(f"rssi:{v}") - return " ".join(parts) - - def _now() -> str: return datetime.now(timezone.utc).isoformat() diff --git a/poller/pollers/meshcore.py b/poller/pollers/meshcore.py index 128aa71..968a404 100644 --- a/poller/pollers/meshcore.py +++ b/poller/pollers/meshcore.py @@ -24,6 +24,7 @@ import httpx from bus import get_bus, publish_entity, set_feed +from config import settings from normalizers.mesh_node import normalize_pymc_repeater_advert, snr_to_quality from sanitize import sanitize_payload from .base import BasePoller @@ -116,32 +117,43 @@ async def _poll_once(self, src: dict): resp = await client.get(f"{base_url}/api/adverts_by_contact_type") if resp.status_code == 200: count = 0 + skipped = 0 for advert in _iter_items(resp.json()): entity = normalize_pymc_repeater_advert(advert, base_url) - if entity: - await publish_entity(entity) - count += 1 - logger.debug("[meshcore] synced %d adverts from %s", count, base_url) + if not entity: + continue + if not _should_publish_node(entity): + skipped += 1 + continue + await publish_entity(entity) + count += 1 + logger.debug( + "[meshcore] synced %d adverts from %s (%d outside region bbox)", + count, base_url, skipped, + ) except Exception as exc: logger.debug("[meshcore] advert fetch error: %s", exc) - # Recent packets → SNR / RSSI link metrics + # Recent packets → SNR / RSSI link metrics. Links anchor on the + # repeater's own entity once _publish_health has identified it. try: resp = await client.get( f"{base_url}/api/recent_packets", params={"limit": _PACKET_LIMIT} ) if resp.status_code == 200: - links = _extract_links_from_packets(resp.json(), base_url) + links = _extract_links_from_packets( + resp.json(), base_url, src.get("self_entity_id") or "local" + ) if links: await _upsert_mesh_links(links) except Exception as exc: logger.debug("[meshcore] packet fetch error: %s", exc) - # System health + # System health + the repeater's own entity try: resp = await client.get(f"{base_url}/api/stats") if resp.status_code == 200: - await _publish_health(resp.json(), base_url, src.get("companion")) + await _publish_health(resp.json(), src) except Exception as exc: logger.debug("[meshcore] stats fetch error: %s", exc) @@ -218,7 +230,7 @@ async def _handle_sse_event(self, event_type: str | None, data: dict, base_url: if event_type == "advert_received": entity = normalize_pymc_repeater_advert(payload, base_url) - if entity: + if entity and _should_publish_node(entity): await publish_entity(entity) elif event_type in ("message_received", "channel_message_received"): @@ -235,21 +247,60 @@ async def _handle_sse_event(self, event_type: str | None, data: dict, base_url: elif event_type == "contact_path_updated": entity = normalize_pymc_repeater_advert(payload, base_url) - if entity: + if entity and _should_publish_node(entity): await publish_entity(entity, merge=True, record_observation=False) # ─── helpers ────────────────────────────────────────────────────────────────── +def _in_region(lat: float, lon: float) -> bool: + """True when the coordinates fall inside any configured region bbox (padded).""" + from config import load_regions + pad = settings.mesh_bbox_pad_deg + for region in load_regions(): + b = region.bbox + if (b.min_lat - pad) <= lat <= (b.max_lat + pad) \ + and (b.min_lon - pad) <= lon <= (b.max_lon + pad): + return True + return False + + +def _should_publish_node(entity: dict) -> bool: + """Bbox gate for mesh_node entities, like the ADS-B/AIS/Amtrak pollers. + + Nodes with no advertised position always pass — they cannot clutter the + map, and direct RF neighbors often advertise without GPS. + """ + if not settings.mesh_bbox_filter: + return True + lat, lon = entity.get("lat"), entity.get("lon") + if lat is None or lon is None: + return True + return _in_region(lat, lon) + + def _parse_source(url: str) -> dict: - """Extract API key and companion from URL; return clean base_url, api_key, and companion.""" + """Extract API key, companion, and optional self-position pin from the URL. + + Query parameters: + companion= lock the SSE connection to a specific companion identity + lat=&lon= pin the repeater's own position (used when the + repeater API does not report its GPS location) + """ parsed = urlparse(url) api_key = None companion = None + self_lat = self_lon = None if parsed.query: qs = parse_qs(parsed.query) if "companion" in qs: companion = qs["companion"][0] + if "lat" in qs and "lon" in qs: + try: + self_lat = float(qs["lat"][0]) + self_lon = float(qs["lon"][0]) + except (TypeError, ValueError): + self_lat = self_lon = None if parsed.username: api_key = parsed.username @@ -257,7 +308,56 @@ def _parse_source(url: str) -> dict: url = urlunparse(parsed._replace(netloc=netloc, query="")) else: url = urlunparse(parsed._replace(query="")) - return {"base_url": url.rstrip("/"), "api_key": api_key, "companion": companion} + return { + "base_url": url.rstrip("/"), + "api_key": api_key, + "companion": companion, + "self_lat": self_lat, + "self_lon": self_lon, + } + + +def _extract_self_position(stats: dict) -> tuple[float, float] | None: + """Best-effort extraction of the repeater's own GPS position from /api/stats. + + pyMC-Repeater builds vary in where (and whether) they report the station's + coordinates, so check the top level plus the common nested containers. + """ + if not isinstance(stats, dict): + return None + containers = [stats] + [ + stats.get(k) for k in + ("self", "node_info", "position", "gps", "location", "radio_device_info") + ] + for obj in containers: + if not isinstance(obj, dict): + continue + lat = obj.get("gps_lat") or obj.get("lat") or obj.get("latitude") + lon = obj.get("gps_lon") or obj.get("lon") or obj.get("longitude") + try: + lat_f, lon_f = float(lat), float(lon) + except (TypeError, ValueError): + continue + if lat_f == 0.0 and lon_f == 0.0: + continue + if -90.0 <= lat_f <= 90.0 and -180.0 <= lon_f <= 180.0: + return lat_f, lon_f + return None + + +def _extract_self_pubkey(stats: dict) -> str | None: + """Best-effort extraction of the repeater's own public key from /api/stats.""" + if not isinstance(stats, dict): + return None + containers = [stats] + [stats.get(k) for k in ("self", "node_info")] + for obj in containers: + if not isinstance(obj, dict): + continue + for key in ("public_key", "self_public_key", "node_pubkey", "pubkey"): + v = obj.get(key) + if isinstance(v, str) and len(v) >= 8 and v != "0" * len(v): + return v + return None def _api_headers(api_key: str | None) -> dict[str, str]: @@ -296,8 +396,13 @@ async def _discover_all_companions(base_url: str, headers: dict) -> list[str]: return [] -def _extract_links_from_packets(payload, source_url: str) -> list[dict]: - """Deduplicate per-sender SNR/RSSI from the recent-packet list.""" +def _extract_links_from_packets(payload, source_url: str, self_id: str = "local") -> list[dict]: + """Deduplicate per-sender SNR/RSSI from the recent-packet list. + + self_id is the local end of every link — the repeater's own entity id when + known, else the legacy "local" placeholder (which the frontend anchors at + the repeater position from mesh:status, falling back to the region center). + """ packets = _iter_items(payload) now = time.time() links: list[dict] = [] @@ -335,7 +440,7 @@ def _extract_links_from_packets(payload, source_url: str) -> list[dict]: age_secs = max(0.0, now - float(ts)) if isinstance(ts, (int, float)) else 0.0 links.append({ "source_url": source_url, - "node_a": "local", + "node_a": self_id, "node_b": node_b, "snr": float(snr), "rssi": float(rssi) if rssi is not None else None, @@ -348,13 +453,20 @@ def _extract_links_from_packets(payload, source_url: str) -> list[dict]: async def _upsert_mesh_links(links: list[dict]) -> None: from db import get_pool now_utc = datetime.datetime.now(datetime.timezone.utc) + + # Normalise SNR to the 0-100 link_quality scale the frontend meters and + # link-width styling expect. + for lnk in links: + quality = snr_to_quality(lnk.get("snr")) + lnk["link_quality"] = round(quality * 100) if quality is not None else None + rows = [ ( lnk["source_url"], lnk["node_a"], lnk["node_b"], lnk.get("snr"), - None, + lnk.get("link_quality"), now_utc - datetime.timedelta(seconds=int(lnk.get("secs_ago", 0))), ) for lnk in links @@ -379,18 +491,39 @@ async def _upsert_mesh_links(links: list[dict]) -> None: "node_a": lnk["node_a"], "node_b": lnk["node_b"], "snr": lnk.get("snr"), - "link_quality": None, + "link_quality": lnk.get("link_quality"), + # Frontend line opacity is age-based; without this the WS + # variant of the payload rendered every link at minimum + # opacity (Date.parse(undefined) → NaN). + "last_seen": ( + now_utc - datetime.timedelta(seconds=int(lnk.get("secs_ago", 0))) + ).isoformat(), } for lnk in links ], })) -async def _publish_health(stats: dict, base_url: str, companion: str | None = None) -> None: +async def _publish_health(stats: dict, src: dict) -> None: if not isinstance(stats, dict): return + base_url = src["base_url"] + companion = src.get("companion") connected = stats.get("radio_connected", stats.get("connected", True)) - uptime = stats.get("uptime_seconds") or stats.get("uptime_secs") + radio_stats = stats.get("radio_stats") if isinstance(stats.get("radio_stats"), dict) else {} + uptime = ( + stats.get("uptime_seconds") + or stats.get("uptime_secs") + or radio_stats.get("uptime_secs") + ) + + # The repeater's own position: an explicit ?lat=&lon= pin on the source URL + # wins; otherwise try to read GPS coordinates from the stats payload. + if src.get("self_lat") is not None and src.get("self_lon") is not None: + self_pos = (src["self_lat"], src["self_lon"]) + else: + self_pos = _extract_self_position(stats) + payload = { "connected": connected, "url": base_url, @@ -398,9 +531,51 @@ async def _publish_health(stats: dict, base_url: str, companion: str | None = No "version": stats.get("version"), "site_name": stats.get("site_name"), "companion": companion, + "lat": self_pos[0] if self_pos else None, + "lon": self_pos[1] if self_pos else None, } await set_feed("mesh:status", payload) - + + # Publish the repeater itself as a mesh_node entity so it renders on the + # map, records observations, and anchors packet-derived links. Bypasses + # the bbox gate: the station's position is explicit operator config, and + # remotely monitoring a repeater outside the region is a supported case. + if self_pos: + pubkey = _extract_self_pubkey(stats) + entity_id = ( + f"mesh_node:{pubkey}" + if pubkey + else f"mesh_node:repeater:{urlparse(base_url).hostname}" + ) + src["self_entity_id"] = entity_id + name = stats.get("site_name") or "pyMC Repeater" + status_parts = [] + battery_mv = radio_stats.get("battery_mv") + if isinstance(battery_mv, (int, float)): + status_parts.append(f"bat:{battery_mv / 1000:.2f}V") + noise_floor = radio_stats.get("noise_floor") + if isinstance(noise_floor, (int, float)): + status_parts.append(f"nf:{noise_floor:.0f}dBm") + await publish_entity({ + "entity_id": entity_id, + "entity_type": "mesh_node", + "source": "meshcore", + "display_name": name, + "identity": { + "node_id": (pubkey or entity_id)[:12], + "short_name": str(name)[:12], + "contact_type": "repeater", + "source_url": base_url, + "is_self": True, + }, + "lat": self_pos[0], + "lon": self_pos[1], + "altitude": None, + "status": " ".join(status_parts), + "last_seen": datetime.datetime.now(datetime.timezone.utc).isoformat(), + "tags": ["mesh_node", "repeater", "self"], + }) + # Process neighbors from stats and publish as mesh_node entities neighbors = stats.get("neighbors", {}) if isinstance(neighbors, dict): @@ -411,7 +586,7 @@ async def _publish_health(stats: dict, base_url: str, companion: str | None = No node_copy["public_key"] = pub_key node_copy["name"] = node.get("node_name") entity = normalize_pymc_repeater_advert(node_copy, base_url) - if entity: + if entity and _should_publish_node(entity): await publish_entity(entity) r = await get_bus() diff --git a/poller/tests/test_meshcore_bbox.py b/poller/tests/test_meshcore_bbox.py new file mode 100644 index 0000000..0ec5528 --- /dev/null +++ b/poller/tests/test_meshcore_bbox.py @@ -0,0 +1,91 @@ +"""Tests for MeshCore node bbox gating. + +Run from poller/: + pytest tests/test_meshcore_bbox.py +""" +from __future__ import annotations + +import os +import sys +from unittest.mock import patch + +_POLLER_ROOT = os.path.join(os.path.dirname(__file__), "..") +if _POLLER_ROOT not in sys.path: + sys.path.insert(0, _POLLER_ROOT) + +from config import RegionConfig, RegionBbox, settings +from pollers.meshcore import _in_region, _should_publish_node + + +# Portland-metro-ish region +MOCK_REGIONS = [ + RegionConfig( + id="portland", + name="Portland Metro", + bbox=RegionBbox( + min_lat=44.8, + max_lat=45.9, + min_lon=-123.5, + max_lon=-121.8, + ), + enabled=True, + ), +] + + +def _node(lat, lon) -> dict: + return { + "entity_id": "mesh_node:abc123", + "entity_type": "mesh_node", + "lat": lat, + "lon": lon, + } + + +class TestInRegion: + @patch.object(settings, "mesh_bbox_pad_deg", 0.25) + @patch("config.load_regions") + def test_inside_bbox(self, mock_load_regions): + mock_load_regions.return_value = MOCK_REGIONS + assert _in_region(45.38, -122.76) is True # Tualatin + + @patch.object(settings, "mesh_bbox_pad_deg", 0.25) + @patch("config.load_regions") + def test_far_outside_bbox(self, mock_load_regions): + mock_load_regions.return_value = MOCK_REGIONS + assert _in_region(47.6, -122.3) is False # Seattle + assert _in_region(44.05, -123.09) is False # Eugene + + @patch.object(settings, "mesh_bbox_pad_deg", 0.25) + @patch("config.load_regions") + def test_pad_extends_bbox(self, mock_load_regions): + mock_load_regions.return_value = MOCK_REGIONS + # Just north of max_lat 45.9 but within the 0.25° pad + assert _in_region(46.1, -122.5) is True + # Beyond the pad + assert _in_region(46.2, -122.5) is False + + +class TestShouldPublishNode: + @patch.object(settings, "mesh_bbox_filter", True) + @patch.object(settings, "mesh_bbox_pad_deg", 0.25) + @patch("config.load_regions") + def test_gates_out_of_region_node(self, mock_load_regions): + mock_load_regions.return_value = MOCK_REGIONS + assert _should_publish_node(_node(45.38, -122.76)) is True + assert _should_publish_node(_node(47.6, -122.3)) is False + + @patch.object(settings, "mesh_bbox_filter", True) + @patch("config.load_regions") + def test_unpositioned_node_always_passes(self, mock_load_regions): + mock_load_regions.return_value = MOCK_REGIONS + assert _should_publish_node(_node(None, None)) is True + assert _should_publish_node(_node(45.38, None)) is True + mock_load_regions.assert_not_called() + + @patch.object(settings, "mesh_bbox_filter", False) + @patch("config.load_regions") + def test_disabled_filter_passes_everything(self, mock_load_regions): + mock_load_regions.return_value = MOCK_REGIONS + assert _should_publish_node(_node(47.6, -122.3)) is True + mock_load_regions.assert_not_called() diff --git a/poller/tests/test_meshcore_self_node.py b/poller/tests/test_meshcore_self_node.py new file mode 100644 index 0000000..52d049c --- /dev/null +++ b/poller/tests/test_meshcore_self_node.py @@ -0,0 +1,95 @@ +"""Tests for repeater self-node discovery and link normalization. + +Run from poller/: + pytest tests/test_meshcore_self_node.py +""" +from __future__ import annotations + +import os +import sys + +_POLLER_ROOT = os.path.join(os.path.dirname(__file__), "..") +if _POLLER_ROOT not in sys.path: + sys.path.insert(0, _POLLER_ROOT) + +from pollers.meshcore import ( + _extract_links_from_packets, + _extract_self_position, + _extract_self_pubkey, + _parse_source, +) + + +class TestParseSourceSelfPin: + def test_lat_lon_query_params(self): + src = _parse_source("http://KEY@192.168.1.10:8000?lat=45.38&lon=-122.76") + assert src["base_url"] == "http://192.168.1.10:8000" + assert src["api_key"] == "KEY" + assert src["self_lat"] == 45.38 + assert src["self_lon"] == -122.76 + + def test_no_pin_defaults_none(self): + src = _parse_source("http://192.168.1.10:8000") + assert src["self_lat"] is None + assert src["self_lon"] is None + + def test_invalid_pin_ignored(self): + src = _parse_source("http://192.168.1.10:8000?lat=abc&lon=-122.76") + assert src["self_lat"] is None + assert src["self_lon"] is None + + def test_companion_and_pin_together(self): + src = _parse_source("http://h:8000?companion=Base&lat=45.0&lon=-122.0") + assert src["companion"] == "Base" + assert src["self_lat"] == 45.0 + + +class TestExtractSelfPosition: + def test_top_level_gps_fields(self): + assert _extract_self_position({"gps_lat": 45.4, "gps_lon": -122.7}) == (45.4, -122.7) + + def test_nested_node_info(self): + stats = {"node_info": {"latitude": 45.4, "longitude": -122.7}} + assert _extract_self_position(stats) == (45.4, -122.7) + + def test_zero_zero_rejected(self): + assert _extract_self_position({"lat": 0.0, "lon": 0.0}) is None + + def test_out_of_range_rejected(self): + assert _extract_self_position({"lat": 91.0, "lon": -122.7}) is None + + def test_missing_position(self): + stats = {"radio_connected": True, "radio_stats": {"battery_mv": 3807}} + assert _extract_self_position(stats) is None + + def test_non_dict(self): + assert _extract_self_position(None) is None + + +class TestExtractSelfPubkey: + def test_top_level_public_key(self): + assert _extract_self_pubkey({"public_key": "abcdef123456"}) == "abcdef123456" + + def test_nested_self(self): + assert _extract_self_pubkey({"self": {"pubkey": "cafe" * 4}}) == "cafe" * 4 + + def test_all_zero_key_rejected(self): + assert _extract_self_pubkey({"public_key": "0" * 64}) is None + + def test_missing(self): + assert _extract_self_pubkey({"version": "1.15.0"}) is None + + +class TestExtractLinksSelfId: + PACKETS = [{"data": {"snr": 8.5, "rssi": -48, "sender_pubkey": "aa11"}}] + + def test_default_local(self): + links = _extract_links_from_packets(self.PACKETS, "http://h:8000") + assert links[0]["node_a"] == "local" + assert links[0]["node_b"] == "mesh_node:aa11" + + def test_self_entity_id(self): + links = _extract_links_from_packets( + self.PACKETS, "http://h:8000", "mesh_node:selfkey" + ) + assert links[0]["node_a"] == "mesh_node:selfkey"