Inference server: first-class off-board GPU compute on the gaming PC#53
Open
MJohnson459 wants to merge 1 commit into
Open
Inference server: first-class off-board GPU compute on the gaming PC#53MJohnson459 wants to merge 1 commit into
MJohnson459 wants to merge 1 commit into
Conversation
…sk 173) Productionize the ad-hoc off-board inference pattern into a first-class "inference server" role on a dedicated Windows gaming PC with an NVIDIA GPU, serving depth and detection today and future services as they land, with the robot degrading gracefully when the server is absent. Host decision (documented in docs/inference-server.md): native Windows, not WSL2. The win-64 CUDA torch wheel solves cleanly in a pixi feature (torch-2.11.0+cu128 win_amd64, verified in pixi.lock), native bind puts the socket straight on the LAN with no WSL NAT/portproxy, and Task Scheduler gives boot auto-start without a VM layer. - pixi: new `inference-cuda` env (feature platforms = ["win-64"], torch from the pytorch.org cu128 wheel index) as an isolated own-solve sibling of `inference`/`inference-rocm` — verified it never perturbs the aarch64 robot or any other env's solve. Tasks: inference-cuda, depth/detect-server-cuda, inference-prefetch[-cuda|-rocm], plus robot-side inference-health/inference-bench. - Health/version endpoint: HEALTH_MAGIC request in the shared wire protocol + WireClient.health(); servers reply with a JSON status blob (model, device, GPU, torch). tools/inference_health.py probes it from the robot (torch-free). - Cross-platform supervisor tools/inference_server.py replaces the bash launcher (no bash on Windows); its SERVICES list is the multi-service seam — a new tenant is one row and inherits binding, supervision, health, and reconnect. - detect_server gains --device auto|cpu|cuda so OWLv2 runs on the GPU too. - Windows deploy scripts (mote_perception/deploy/windows): setup.ps1 (pixi + env + model prefetch + GPU check), install/uninstall_service.ps1 (boot Scheduled Task), run_inference.ps1 (self-restarting runner + dated logs). - Measurement: tools/inference_bench.py (torch-free round-trip latency/fps -> committed JSON) and benchmarks/README.md with the #152 CPU/ROCm baselines and a template for the CUDA numbers Michael fills at the PC. - Fallback verified/documented: server-absent = warn-and-skip (topic goes quiet, publisher stays alive, reconnects automatically); nav keeps running on lidar. - Docs: docs/inference-server.md (role, host decision, setup guide, prod behaviors, multi-service pattern, fallback matrix, measurement); README + CLAUDE updated. New tests cover the health round-trip and interleaving with infer. Verified repo-side: 17 wire tests pass (incl. new health tests), ruff check/format clean, pixi.lock re-solves with per-env platform isolation intact. On-PC steps (CUDA latency numbers, reboot/auto-start, LAN e2e) need Michael's session at the gaming PC per the setup guide. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLcxTNpEiB29frknCDug2V
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Inference server role: dedicated NVIDIA/Windows off-board compute.
Promotes the ad-hoc off-board inference pattern into a first-class inference-server role on a dedicated Windows gaming PC (NVIDIA GPU), serving depth + detection now and future services as they land, with graceful robot degradation when the server is absent.
HOST DECISION (docs/inference-server.md): native Windows, not WSL2 — the win-64 CUDA torch wheel solves cleanly in pixi (verified torch-2.11.0+cu128 win_amd64 in pixi.lock), native bind puts the socket straight on the LAN (no WSL NAT/portproxy), and Task Scheduler gives boot auto-start with no VM layer.
CHANGES:
VERIFIED repo-side: 17 wire tests pass incl. new health round-trip/interleave tests; ruff check+format clean; pre-commit hooks pass on commit; pixi.lock re-solves with per-env platform isolation intact; inference-health CLI tested end-to-end (UP + DOWN paths, exit codes).
HUMAN-IN-LOOP (per task): CUDA latency numbers, reboot/auto-start, and LAN e2e need Michael's session at the gaming PC following docs/inference-server.md.
Branch inference-server-role committed locally (not pushed per standing git rule); ready for voro pr.