qa: add manifest-check tool for #2225 (failed to download manifest)

.github/workflows/manifest-check.yml

@@ -0,0 +1,47 @@
+name: manifest-check
+
+# Catch issue #2225 ("Failed to download manifest") regressions. Probes the
+# public ESP Web Tools manifest endpoints served by functions/releases/[[path]].ts
+# and asserts they return well-formed JSON with reachable binaries.
+
+on:
+  push:
+    branches: [main]
+    paths:
+      - "tools/manifest_check/**"
+      - "functions/releases/**"
+      - ".github/workflows/manifest-check.yml"
+  pull_request:
+    paths:
+      - "tools/manifest_check/**"
+      - "functions/releases/**"
+      - ".github/workflows/manifest-check.yml"
+  schedule:
+    - cron: "17 * * * *"
+  workflow_dispatch: {}
+
+permissions:
+  contents: read
+
+jobs:
+  unit-tests:
+    name: Schema unit tests
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: python3 tools/manifest_check/test_check_manifest.py
+
+  probe-manifest:
+    name: Probe public manifest endpoints
+    runs-on: ubuntu-latest
+    needs: unit-tests
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Probe manifest URLs
+        run: python3 tools/manifest_check/check_manifest.py

tools/manifest_check/.gitignore

@@ -0,0 +1,2 @@
+__pycache__/
+*.pyc

tools/manifest_check/README.md

@@ -0,0 +1,67 @@
+# manifest-check
+
+Headless probe of the public ESP Web Tools manifest endpoints served by this
+repo's `functions/releases/[[path]].ts`. Reproduces issue
+[#2225](https://github.com/ESPresense/ESPresense/issues/2225) ("Failed to
+download manifest"), where `https://espresense.com/releases/{version}.json?flavor={flavor}`
+returns 5xx/404 and the web installer surfaces it as a generic "Failed to
+download manifest" toast with no diagnostic.
+
+## Run locally
+
+```sh
+python3 tools/manifest_check/check_manifest.py
+```
+
+Skip the per-binary HEAD validation (faster, useful when iterating on the
+script itself):
+
+```sh
+python3 tools/manifest_check/check_manifest.py --no-check-parts
+```
+
+Probe a different host (for local dev via `wrangler pages dev`):
+
+```sh
+python3 tools/manifest_check/check_manifest.py --base http://localhost:8788
+```
+
+## What the probe checks
+
+For each `version,flavor` listed in [`targets.txt`](./targets.txt):
+
+1. `GET /releases/{version}.json?flavor={flavor}` returns HTTP 200 with
+   `application/json`.
+2. The body parses as JSON.
+3. Required top-level keys are present (`name`, `version`, `builds`).
+4. Each `builds[]` entry has a non-empty `parts[]` array with `path` and
+   `offset`.
+5. Every `parts[].path` HEAD-resolves to 200 / 3xx (catches stale binaries
+   removed from the CDN).
+
+Single transient 5xx responses are retried up to 3 times with exponential
+backoff before failing the run; this keeps CI from flapping on transient
+upstream hiccups while still catching persistent failures like the one
+reported on [#2225](https://github.com/ESPresense/ESPresense/issues/2225).
+
+## Updating the target list
+
+[`targets.txt`](./targets.txt) is the source of truth for which (version,
+flavor) pairs the installer is expected to serve. Edit it when:
+
+- A new release ships and is added to the installer dropdown.
+- A new flavor (board / variant) enters the installer dropdown.
+- A release is removed from the public installer (e.g., yanked due to a
+  bricking bug).
+
+Lines starting with `#` are comments. Format is `version,flavor` per line.
+
+## CI
+
+[`.github/workflows/manifest-check.yml`](../../.github/workflows/manifest-check.yml)
+runs the probe on:
+
+- pull requests that touch this directory, the release handler, or the workflow,
+- pushes to `main` that touch the same,
+- an hourly cron (`17 * * * *`),
+- manual `workflow_dispatch`.

tools/manifest_check/check_manifest.py

@@ -0,0 +1,238 @@
+#!/usr/bin/env python3
+"""Probe the public ESP Web Tools manifest endpoints for ESPresense releases.
+
+Reproduces issue #2225 ("Failed to download manifest"), which surfaces when
+``https://espresense.com/releases/{version}.json?flavor={flavor}`` returns 5xx
+or 404 for a given version/flavor combination. The web installer surfaces that
+upstream failure as a generic "Failed to download manifest" toast with no
+diagnostic in the UI.
+
+This script fetches each (version, flavor) pair listed in ``targets.txt``,
+asserts the response is HTTP 200 with parseable JSON that conforms to the
+ESP Web Tools manifest schema, and HEADs each ``builds[].parts[].path`` to
+confirm the binaries are reachable. A small retry budget filters single
+transient 500s; persistent failures still fail the run.
+
+The script is stdlib-only on purpose so it can run unmodified in CI without a
+virtualenv setup step.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+import time
+import urllib.error
+import urllib.parse
+import urllib.request
+from dataclasses import dataclass, field
+from pathlib import Path
+
+DEFAULT_BASE = "https://espresense.com"
+DEFAULT_TARGETS = Path(__file__).with_name("targets.txt")
+USER_AGENT = "ESPresense-manifest-check/1 (+https://github.com/ESPresense/ESPresense.com)"
+
+REQUIRED_TOP_KEYS = ("name", "version", "builds")
+REQUIRED_PART_KEYS = ("path", "offset")
+
+
+@dataclass
+class Target:
+    version: str
+    flavor: str
+
+    def manifest_url(self, base: str) -> str:
+        query = urllib.parse.urlencode({"flavor": self.flavor})
+        return f"{base}/releases/{self.version}.json?{query}"
+
+
+@dataclass
+class Failure:
+    target: Target
+    stage: str
+    detail: str
+
+
+@dataclass
+class Report:
+    failures: list[Failure] = field(default_factory=list)
+    checked: int = 0
+
+    def fail(self, target: Target, stage: str, detail: str) -> None:
+        self.failures.append(Failure(target, stage, detail))
+
+
+def parse_targets(path: Path) -> list[Target]:
+    targets: list[Target] = []
+    for raw in path.read_text().splitlines():
+        line = raw.split("#", 1)[0].strip()
+        if not line:
+            continue
+        if "," not in line:
+            raise SystemExit(f"{path}: malformed line (expected 'version,flavor'): {raw!r}")
+        version, flavor = (part.strip() for part in line.split(",", 1))
+        if not version or not flavor:
+            raise SystemExit(f"{path}: empty version or flavor: {raw!r}")
+        targets.append(Target(version=version, flavor=flavor))
+    if not targets:
+        raise SystemExit(f"{path}: no targets")
+    return targets
+
+
+def http_request(url: str, *, method: str, timeout: float, retries: int, backoff: float) -> tuple[int, bytes, str]:
+    last_exc: Exception | None = None
+    for attempt in range(retries + 1):
+        req = urllib.request.Request(url, method=method, headers={"User-Agent": USER_AGENT})
+        try:
+            with urllib.request.urlopen(req, timeout=timeout) as resp:
+                body = resp.read() if method == "GET" else b""
+                return resp.status, body, resp.headers.get("Content-Type", "")
+        except urllib.error.HTTPError as exc:
+            if (exc.code < 500 and exc.code != 429) or attempt == retries:
+                body = exc.read() if method == "GET" else b""
+                return exc.code, body, exc.headers.get("Content-Type", "") if exc.headers else ""
+            last_exc = exc
+        except (urllib.error.URLError, TimeoutError) as exc:
+            last_exc = exc
+            if attempt == retries:
+                raise
+        time.sleep(backoff * (2 ** attempt))
+    if last_exc:
+        raise last_exc
+    raise RuntimeError("unreachable")
+
+
+def validate_schema(payload: dict) -> str | None:
+    for key in REQUIRED_TOP_KEYS:
+        if key not in payload:
+            return f"missing top-level key: {key}"
+    builds = payload.get("builds")
+    if not isinstance(builds, list) or not builds:
+        return "builds[] must be a non-empty list"
+    for idx, build in enumerate(builds):
+        if not isinstance(build, dict):
+            return f"builds[{idx}] is not an object"
+        if "chipFamily" not in build:
+            return f"builds[{idx}] missing chipFamily"
+        parts = build.get("parts")
+        if not isinstance(parts, list) or not parts:
+            return f"builds[{idx}].parts must be a non-empty list"
+        for pidx, part in enumerate(parts):
+            if not isinstance(part, dict):
+                return f"builds[{idx}].parts[{pidx}] is not an object"
+            for key in REQUIRED_PART_KEYS:
+                if key not in part:
+                    return f"builds[{idx}].parts[{pidx}] missing {key}"
+    return None
+
+
+def check_target(
+    target: Target, base: str, *, timeout: float, retries: int, backoff: float, check_parts: bool, report: Report
+) -> None:
+    report.checked += 1
+    url = target.manifest_url(base)
+    try:
+        status, body, content_type = http_request(
+            url, method="GET", timeout=timeout, retries=retries, backoff=backoff
+        )
+    except Exception as exc:  # noqa: BLE001 -- want the message verbatim
+        report.fail(target, "fetch", f"{url}: {exc}")
+        return
+
+    if status != 200:
+        snippet = body[:200].decode("utf-8", errors="replace").strip()
+        report.fail(target, "status", f"{url}: HTTP {status} (body: {snippet!r})")
+        return
+
+    if "json" not in content_type.lower():
+        report.fail(target, "content-type", f"{url}: got {content_type!r}, want application/json")
+        return
+
+    try:
+        payload = json.loads(body)
+    except json.JSONDecodeError as exc:
+        snippet = body[:200].decode("utf-8", errors="replace")
+        report.fail(target, "parse", f"{url}: {exc.msg} at offset {exc.pos} (body starts: {snippet!r})")
+        return
+
+    schema_error = validate_schema(payload)
+    if schema_error:
+        report.fail(target, "schema", f"{url}: {schema_error}")
+        return
+
+    if not check_parts:
+        return
+
+    seen_paths: set[str] = set()
+    for build in payload["builds"]:
+        for part in build["parts"]:
+            path = part["path"]
+            if path in seen_paths:
+                continue
+            seen_paths.add(path)
+            if path.startswith(("http://", "https://")):
+                part_url = path
+            else:
+                part_url = urllib.parse.urljoin(url, path)
+            try:
+                pstatus, _, _ = http_request(
+                    part_url, method="HEAD", timeout=timeout, retries=retries, backoff=backoff
+                )
+            except Exception as exc:  # noqa: BLE001
+                report.fail(target, "part-fetch", f"{part_url}: {exc}")
+                continue
+            if pstatus not in (200, 301, 302, 307, 308):
+                report.fail(target, "part-status", f"{part_url}: HTTP {pstatus}")
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(description=(__doc__ or "").split("\n", 1)[0])
+    parser.add_argument(
+        "--base", default=DEFAULT_BASE, help="Manifest host (default: %(default)s)"
+    )
+    parser.add_argument(
+        "--targets",
+        type=Path,
+        default=DEFAULT_TARGETS,
+        help="Path to a 'version,flavor' targets file (default: %(default)s)",
+    )
+    parser.add_argument(
+        "--timeout", type=float, default=15.0, help="Per-request timeout seconds (default: %(default)s)"
+    )
+    parser.add_argument(
+        "--retries", type=int, default=3, help="Retry count for 5xx and connection errors (default: %(default)s)"
+    )
+    parser.add_argument(
+        "--backoff", type=float, default=2.0, help="Initial backoff seconds, exponential (default: %(default)s)"
+    )
+    parser.add_argument(
+        "--no-check-parts", action="store_true", help="Skip the per-binary HEAD validation"
+    )
+    args = parser.parse_args(argv)
+
+    targets = parse_targets(args.targets)
+    report = Report()
+    for target in targets:
+        check_target(
+            target,
+            args.base.rstrip("/"),
+            timeout=args.timeout,
+            retries=args.retries,
+            backoff=args.backoff,
+            check_parts=not args.no_check_parts,
+            report=report,
+        )
+
+    print(f"Checked {report.checked} target(s).")
+    if not report.failures:
+        print("All manifests OK.")
+        return 0
+
+    print(f"FAILED: {len(report.failures)} problem(s):", file=sys.stderr)
+    for f in report.failures:
+        print(f"  [{f.stage}] {f.target.version}/{f.target.flavor}: {f.detail}", file=sys.stderr)
+    return 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

tools/manifest_check/targets.txt

@@ -0,0 +1,26 @@
+# ESPresense web-installer manifest endpoints to probe.
+# Format: version,flavor   (one pair per line; '#' starts a comment)
+#
+# Add a row when a release ships or when a new flavor enters the installer
+# dropdown. Drop a row when a release is removed from the public installer.
+#
+# Reference: issue #2225 ("Failed to download manifest"). Web installer fetches
+# https://espresense.com/releases/{version}.json?flavor={flavor} per selection.
+
+# 'latest' is whatever the installer's default points at; it must always work.
+latest,esp32
+latest,esp32c3
+latest,esp32s3
+latest,esp32c6
+latest,m5atom
+latest,m5stickc
+
+# v4.0.6 is the version flagged in #2225 with persistent intermittent 500s.
+v4.0.6,esp32
+v4.0.6,esp32c3
+v4.0.6,esp32s3
+v4.0.6,m5atom
+
+# v4.0.5 is the last-known-working release reported on #2225; keep as a sentinel.
+v4.0.5,esp32
+v4.0.5,m5atom