Overhaul ADS-B pipeline: dead reckoning, signal-gap handling, stale-track rendering

.env.example

@@ -103,6 +103,11 @@ ADSB_BEAST_RECONNECT_MAX_SECONDS=30
 # Seconds without a BEAST frame before HTTP fallback activates (Mode C)
 ADSB_BEAST_STALE_THRESHOLD_SECONDS=30
 ADSB_PUBLISH_ONLY_CHANGES=true
+# Seconds since the last resolved CPR fix before a BEAST track is flagged stale
+ADSB_POSITION_STALE_SECONDS=10
+# Max seconds a stale position is dead-reckoned forward along the last known
+# velocity (keeps icons moving through short signal gaps; 0 disables)
+ADSB_DEAD_RECKON_MAX_SECONDS=60
 
 # Mode D — OpenSky supplement alongside local sources
 # Enabled by default. Local sources (beast/ultrafeeder) remain primary; OpenSky

TASK_LOG.md

@@ -5,6 +5,28 @@ Format: `## YYYY-MM-DD — <summary>` with bullet points for details.
 
 ---
 
+## 2026-07-06 — ADS-B pipeline overhaul: dead reckoning, signal-gap handling, stale-track rendering
+
+- **Server-side dead reckoning** ([beast_decoder.py](poller/normalizers/beast_decoder.py), [config.py](poller/config.py)):
+  - `_to_entity` now projects a stale position forward along the last known velocity (finally using the previously-unused `project_position` in `beast_math.py`) instead of freezing it. Velocity messages (DF17 TC19) usually keep decoding through CPR position gaps, so the track keeps moving where the aircraft actually is. Altitude is projected with the vertical rate.
+  - New `_AircraftState.last_velocity_ts` gates DR on velocity freshness; DR requires airborne status and is bounded by new `ADSB_DEAD_RECKON_MAX_SECONDS` (default 60). The stale threshold is now configurable via `ADSB_POSITION_STALE_SECONDS` (default 10).
+  - Entities carry `position_dr` and `position_age_s` alongside `position_stale`. DR positions never enter the trail ring buffer, are excluded from DB observations (`record_observation=False` in the frame worker + a hard guard in [db.py](poller/db.py)), and never trigger geofence entry/exit events.
+  - Redis-hydrated aircraft (position but no fix timestamp) now correctly report `position_stale=true` until a real fix arrives, matching the hydration docstring's intent.
+- **Cross-source CPR seeding** ([beast_decoder.py](poller/normalizers/beast_decoder.py), [adsb.py](poller/pollers/adsb.py)):
+  - New `BeastAircraftDecoder.seed_reference()` — OpenSky and ultrafeeder positions seed the decoder's Tier-2 local CPR reference, so a single odd/even frame resolves a position the moment an aircraft (re-)enters SDR range instead of waiting up to ~60s for a fresh even+odd pair (the main cause of cold-lock gaps). Local fixes stay authoritative: seeds only apply when the decoder's own fix is missing or stale, and never touch trails or snapshots.
+- **Signal-gap fixes** ([adsb.py](poller/pollers/adsb.py), [beast_decoder.py](poller/normalizers/beast_decoder.py)):
+  - The registry tick loop now refreshes decoder entities on every snapshot tick (not only when new BEAST frames arrived), so DR positions advance during a total feed outage instead of the whole snapshot freezing.
+  - The CPR heading-consistency guard is now bounded to gaps <30s — across a longer gap the aircraft may legitimately have turned, and rejecting the first good fix after a gap extended outages indefinitely.
+- **tar1090 normalizer fixes** ([aircraft.py](poller/normalizers/aircraft.py)):
+  - `alt_baro: "ground"` (readsb's actual ground signal — there is no `on_ground` key in aircraft.json) now sets `on_ground` status and no longer leaks the string `"ground"` into the numeric `altitude` field.
+  - `seen_pos` is honored: positions older than 60s are dropped, older than 10s are flagged `position_stale`; non-numeric track/gs/baro_rate values are nulled.
+- **Frontend stale/DR rendering** ([pvb.ts](frontend/src/layers/pvb.ts), [entityUtils.ts](frontend/src/entityUtils.ts), [buildEntityLayers.ts](frontend/src/layers/buildEntityLayers.ts), [storeTypes.ts](frontend/src/storeTypes.ts)):
+  - PVB keeps projecting server-dead-reckoned tracks between reports (previously stale tracks froze, then snapped to the next fix). `positionDr` is part of the report key so DR↔fix transitions re-blend smoothly.
+  - Stale/dead-reckoned local aircraft render dimmed (alpha 140) like OpenSky supplements, so estimated positions are visually distinct from live fixes.
+- **Hot-path optimization** ([beast_decoder.py](poller/normalizers/beast_decoder.py)): DF and frame length are checked on raw bytes (`byte >> 3`) before any hex-string allocation or pyModeS call — same technique as PR #115 (credit: Jules), folded in so the overhaul builds on it.
+- **Tests**: 13 new decoder tests (fast-path rejection, dead reckoning bounds, seed arbitration) in [test_beast_decoder.py](poller/tests/test_beast_decoder.py); 9 new tar1090 tests in [test_adsb_normalization.py](poller/tests/test_adsb_normalization.py). Full poller suite: 155 passed.
+- **Motivation**: User asked for an ADS-B pipeline overhaul targeting dead reckoning, odd signal gaps, and frontend rendering across both the local SDR BEAST feed and the OpenSky supplement.
+
 ## 2026-07-05 — Gated mesh nodes to region bbox and made entity filters persistent
 
 - **Mesh node bbox gating** ([meshcore.py](poller/pollers/meshcore.py), [config.py](poller/config.py)):

frontend/src/entityUtils.ts

@@ -28,7 +28,8 @@ export function entityToTrack(entity: Entity, existing?: Track): Track | null {
   const altMeters  = isAir ? (entity.altitude ?? 0) * ALT_FT_TO_M : 0
   const speedMs    = (entity.speed ?? 0) * SPD_KT_TO_MS
   const courseTrue = entity.heading ?? 0
-  const positionStale = Boolean((entity as Entity & { position_stale?: boolean }).position_stale)
+  const positionStale = Boolean(entity.position_stale)
+  const positionDr = Boolean(entity.position_dr)
 
   // ── Build raw trail ──────────────────────────────────────────────────────
   // Trail sources (merged in order, oldest → newest):
@@ -102,6 +103,7 @@ export function entityToTrack(entity: Entity, existing?: Track): Track | null {
     source:       entity.source,
     lastSeen:     entity.last_seen,
     positionStale,
+    positionDr,
     lat:          entity.lat,
     lon:          entity.lon,
     altMeters,

frontend/src/layers/buildEntityLayers.ts

@@ -160,11 +160,17 @@ export function buildEntityLayers(
       if (t.type === 'hazard')  return FIRE_ICON_COLOR
       if (t.type === 'rail')    return tagColorMap?.[t.uid] ?? TRAIN_ICON_COLOR
       if (t.type === 'sensor')  return RF_SENSOR_COLOR
-      // OpenSky-supplemented aircraft are lower-fidelity (coarser, delayed
-      // positions filling a local BEAST signal gap) — render them dimmed so it
-      // is visually clear the track is not from the local feed.
-      if (t.type === 'air' && (t.source ?? '').toLowerCase() === 'opensky') {
-        return tagColorMap?.[t.uid] ?? entityColor(t, 120)
+      // Lower-fidelity aircraft positions render dimmed so it is visually
+      // clear the track is not a live local fix: OpenSky supplements (coarse,
+      // delayed) and stale/dead-reckoned local tracks (estimated during a
+      // signal gap).
+      if (t.type === 'air') {
+        if ((t.source ?? '').toLowerCase() === 'opensky') {
+          return tagColorMap?.[t.uid] ?? entityColor(t, 120)
+        }
+        if (t.positionStale || t.positionDr) {
+          return tagColorMap?.[t.uid] ?? entityColor(t, 140)
+        }
       }
       return tagColorMap?.[t.uid] ?? entityColor(t)
     },
@@ -175,7 +181,7 @@ export function buildEntityLayers(
     updateTriggers: {
       getIcon:  zoom,
       getAngle: trackArr.map(t => t.courseTrue),
-      getColor: trackArr.map(t => tagColorMap?.[t.uid]?.join(',') ?? `${t.altMeters + t.speedMs}${t.stationType ?? ''}${t.source ?? ''}`),
+      getColor: trackArr.map(t => tagColorMap?.[t.uid]?.join(',') ?? `${t.altMeters + t.speedMs}${t.stationType ?? ''}${t.source ?? ''}${t.positionStale ? 's' : ''}${t.positionDr ? 'd' : ''}`),
       getSize:  [selectedUid, zoom],
     },
   })

frontend/src/layers/pvb.ts

@@ -36,6 +36,7 @@ function reportKey(track: Track, lastTs: string): string {
     track.lon.toFixed(5),
     track.lat.toFixed(5),
     track.positionStale ? '1' : '0',
+    track.positionDr ? '1' : '0',
   ].join('|')
 }
 
@@ -79,7 +80,11 @@ export function applyPVB(
 ): [number, number] {
   const lastTs = track.trail[track.trail.length - 1]?.[4] ?? ''
   const lastReportKey = reportKey(track, lastTs)
-  const projectedSpeed = track.positionStale ? 0 : track.speedMs
+  // Stale positions freeze — except server-side dead-reckoned ones, whose
+  // lat/lon are already advancing along the last known velocity. Keep
+  // projecting those between reports so motion stays continuous instead of
+  // stepping once per snapshot.
+  const projectedSpeed = track.positionStale && !track.positionDr ? 0 : track.speedMs
   const state  = pvb[track.uid]
 
   if (!state) {

frontend/src/storeTypes.ts

@@ -14,6 +14,12 @@ export interface Entity {
   signal_quality?: number
   status?:         string
   last_seen?:   string
+  // Position freshness (BEAST decoder / normalizers):
+  // stale = last real fix is older than the stale threshold;
+  // dr = lat/lon are dead-reckoned forward from that fix server-side.
+  position_stale?: boolean
+  position_dr?:    boolean
+  position_age_s?: number | null
   identity?:    Record<string, unknown>
   tags?:        string[]
   // Server-side position ring buffer emitted by the BEAST decoder.
@@ -57,6 +63,7 @@ export interface Track {
   source:        string
   lastSeen?:     string
   positionStale?: boolean
+  positionDr?:    boolean      // position is a server-side dead-reckoned estimate
   lat:           number
   lon:           number
   altMeters:     number        // metres MSL (0 for vessels)

poller/config.py

@@ -138,6 +138,14 @@ def regions(self) -> list[RegionConfig]:
     adsb_publish_only_changes: bool = True
     allow_private_ips: bool = False
 
+    # Seconds since the last resolved CPR fix before a BEAST track's position
+    # is flagged stale (freezes client-side extrapolation without dead reckoning).
+    adsb_position_stale_seconds: int = 10
+    # Maximum age of the last position fix (seconds) that the decoder will
+    # dead-reckon forward using the aircraft's last known velocity. Beyond
+    # this the position freezes at the last real fix. 0 disables dead reckoning.
+    adsb_dead_reckon_max_seconds: int = 60
+
     # Mode D — OpenSky supplement alongside local sources (beast or ultrafeeder).
     # When enabled, OpenSky polls on its own interval and fills in aircraft not
     # seen locally within adsb_opensky_stale_threshold seconds. On by default —

poller/db.py

@@ -135,6 +135,14 @@ async def write_entity_observation(entity: dict, record_observation: bool = True
     entity_id_key = entity["entity_id"]
     now_ts = time.time()
 
+    # Dead-reckoned positions are estimates projected from the last real fix —
+    # never let them trigger geofence entry/exit events or land in the
+    # observation history. The entity row update (identity/last_seen) still runs.
+    if entity.get("position_dr"):
+        lat = None
+        lon = None
+        record_observation = False
+
     # Determine if we should update the entity row
     should_write_entity = False
     last_write = _last_entity_write_ts.get(entity_id_key, 0.0)

poller/normalizers/aircraft.py

@@ -51,10 +51,34 @@ def normalize_opensky(state: list) -> Optional[dict]:
     }
 
 
+TAR1090_POSITION_STALE_SECONDS = 10
+TAR1090_POSITION_MAX_AGE_SECONDS = 60
+
+
+def _numeric_or_none(value) -> Optional[float]:
+    return value if isinstance(value, (int, float)) and not isinstance(value, bool) else None
+
+
 def normalize_tar1090(ac: dict) -> Optional[dict]:
     icao = ac.get("hex", "").lower()
     if not icao or ac.get("lat") is None or ac.get("lon") is None:
         return None
+
+    # tar1090/readsb signals ground via alt_baro == "ground" (there is no
+    # on_ground key in the aircraft.json schema); accept both for robustness.
+    on_ground = ac.get("alt_baro") == "ground" or bool(ac.get("on_ground"))
+    altitude = _numeric_or_none(ac.get("alt_baro"))
+    if altitude is None:
+        altitude = _numeric_or_none(ac.get("alt_geom"))
+
+    # seen_pos = seconds since the last position fix. Skip aircraft whose fix is
+    # ancient (readsb retains them for minutes) and flag moderately old ones so
+    # the frontend freezes extrapolation instead of projecting stale motion.
+    seen_pos = _numeric_or_none(ac.get("seen_pos"))
+    if seen_pos is not None and seen_pos > TAR1090_POSITION_MAX_AGE_SECONDS:
+        return None
+    position_stale = seen_pos is not None and seen_pos > TAR1090_POSITION_STALE_SECONDS
+
     return {
         "entity_id": f"aircraft:{icao}",
         "entity_type": "aircraft",
@@ -68,11 +92,13 @@ def normalize_tar1090(ac: dict) -> Optional[dict]:
         },
         "lat": ac.get("lat"),
         "lon": ac.get("lon"),
-        "altitude": ac.get("alt_baro") or ac.get("alt_geom"),
-        "heading": ac.get("track"),
-        "speed": ac.get("gs"),
-        "vertical_rate": ac.get("baro_rate"),
-        "status": "on_ground" if ac.get("on_ground") else "airborne",
+        "position_stale": position_stale,
+        "position_age_s": seen_pos,
+        "altitude": altitude,
+        "heading": _numeric_or_none(ac.get("track")),
+        "speed": _numeric_or_none(ac.get("gs")),
+        "vertical_rate": _numeric_or_none(ac.get("baro_rate")),
+        "status": "on_ground" if on_ground else "airborne",
         "last_seen": _now(),
         "tags": ["aircraft"],
     }

poller/normalizers/beast_decoder.py

@@ -40,6 +40,7 @@ class _AircraftState:
     even_ts: float | None = None
     odd_ts: float | None = None
     last_position_ts: float | None = None
+    last_velocity_ts: float | None = None
     last_mlat_ticks: int | None = None
     signal_quality: int | None = None
     msg_count: int = 0
@@ -100,18 +101,18 @@ def ingest(self, message_bytes: bytes, *, mlat_ticks: int | None = None, signal:
                 self._warned_missing_dep = True
             return None
 
-        hex_msg = message_bytes.hex().upper()
-        if len(hex_msg) not in (14, 28):
-            return None
-
-        try:
-            df = pms.df(hex_msg)
-        except Exception:
+        # Fast path: derive Downlink Format and length from the raw bytes so the
+        # flood of irrelevant Mode S traffic is discarded before any hex-string
+        # allocation or pyModeS call (~2x hot-path speedup, credit PR #115).
+        if len(message_bytes) not in (7, 14):
             return None
 
+        df = message_bytes[0] >> 3
         if df not in (4, 5, 11, 17, 18, 20, 21):
             return None
 
+        hex_msg = message_bytes.hex().upper()
+
         try:
             icao = (pms.icao(hex_msg) or "").lower()
         except Exception:
@@ -168,6 +169,8 @@ def ingest(self, message_bytes: bytes, *, mlat_ticks: int | None = None, signal:
                         ac.heading = float(trk)
                     if vr is not None:
                         ac.vertical_rate = float(vr)
+                    if spd is not None and trk is not None:
+                        ac.last_velocity_ts = now
 
         if df in (4, 20):
             alt = self._decode_altitude_reply(hex_msg)
@@ -188,6 +191,40 @@ def ingest(self, message_bytes: bytes, *, mlat_ticks: int | None = None, signal:
 
         return self._to_entity(ac)
 
+    def seed_reference(self, icao: str, lat: float, lon: float, ts: float | None = None) -> bool:
+        """Seed a position reference from another source (OpenSky / ultrafeeder).
+
+        Gives Tier-2 local CPR decode an immediate reference so a single odd or
+        even frame resolves a position without waiting up to ~60 s for a fresh
+        even+odd pair — the main cause of cold-lock gaps when an aircraft
+        re-enters SDR range. Never touches the trail or last_seen_ts, so seeded
+        aircraft do not appear in snapshots until real frames arrive.
+
+        Returns True if the reference was applied.
+        """
+        icao = (icao or "").lower()
+        if not icao or lat is None or lon is None:
+            return False
+
+        now = time.time()
+        seed_ts = float(ts) if ts is not None else now
+
+        ac = self._aircraft.get(icao)
+        if ac is None:
+            ac = _AircraftState(icao=icao)
+            self._aircraft[icao] = ac
+        elif ac.last_position_ts is not None:
+            # Keep the local fix unless it is older than the incoming reference
+            # and already past the stale threshold — local CPR beats a supplement.
+            stale_after = float(settings.adsb_position_stale_seconds)
+            if ac.last_position_ts >= seed_ts or (now - ac.last_position_ts) <= stale_after:
+                return False
+
+        ac.lat = float(lat)
+        ac.lon = float(lon)
+        ac.last_position_ts = seed_ts
+        return True
+
     def snapshot_entities(self, stale_seconds: int = 60) -> list[dict]:
         now = time.time()
         result: list[dict] = []
@@ -253,9 +290,13 @@ def _update_cpr(self, ac: _AircraftState, hex_msg: str, now: float):
                 return
 
             # Heading-consistency guard: reject bad Tier-2/3 CPR decodes where
-            # the candidate bearing differs >90° from the known track.
+            # the candidate bearing differs >90° from the known track. Only
+            # applied while the elapsed gap is short — across a long gap the
+            # aircraft may legitimately have turned, and rejecting the first
+            # good fix after a gap would extend the outage indefinitely.
             if (
                 distance_km > 0.1
+                and elapsed_seconds < 30.0
                 and ac.heading is not None
                 and ac.speed is not None
                 and ac.speed > 50  # knots — ignore heading at low taxi speed
@@ -287,10 +328,38 @@ def _to_entity(self, ac: _AircraftState, now: float | None = None) -> dict | Non
 
         display_lat = ac.lat
         display_lon = ac.lon
-        position_stale = (
-            ac.last_position_ts is not None
-            and (now_ts - ac.last_position_ts) > 10.0
+        display_alt = ac.altitude
+
+        stale_after = float(settings.adsb_position_stale_seconds)
+        pos_age = (
+            max(0.0, now_ts - ac.last_position_ts)
+            if ac.last_position_ts is not None else None
         )
+        # No fix timestamp (Redis-hydrated state) counts as stale until a real
+        # fix or cross-source seed arrives.
+        position_stale = pos_age is None or pos_age > stale_after
+
+        # Dead reckoning: velocity messages (TC19) usually keep decoding through
+        # CPR position gaps, so project the display position forward along the
+        # last known track instead of freezing it. Bounded by the configured
+        # window, never written to the trail, and flagged so downstream
+        # consumers (DB observations, geofences, UI) can treat it as estimated.
+        position_dr = False
+        dr_max = float(settings.adsb_dead_reckon_max_seconds)
+        if (
+            position_stale
+            and pos_age is not None
+            and pos_age <= dr_max
+            and ac.last_velocity_ts is not None
+            and (now_ts - ac.last_velocity_ts) <= dr_max
+            and not ac.on_ground
+        ):
+            projected = project_position(ac.lat, ac.lon, ac.heading, ac.speed, pos_age)
+            if projected is not None:
+                display_lat, display_lon = projected
+                position_dr = True
+                if display_alt is not None and ac.vertical_rate is not None:
+                    display_alt = max(0.0, display_alt + ac.vertical_rate * pos_age / 60.0)
 
         comm_b = self._build_comm_b_snapshot(ac, now_ts)
 
@@ -313,7 +382,9 @@ def _to_entity(self, ac: _AircraftState, now: float | None = None) -> dict | Non
             "lat": display_lat,
             "lon": display_lon,
             "position_stale": position_stale,
-            "altitude": ac.altitude,
+            "position_dr": position_dr,
+            "position_age_s": round(pos_age, 1) if pos_age is not None else None,
+            "altitude": display_alt,
             "heading": ac.heading,
             "speed": ac.speed,
             "vertical_rate": ac.vertical_rate,