add queue wait histogram, oldest queued age, in-flight cache

[tulioz]

closes queue-wait and stuck-task blindspots

[aviator-app]

Current Aviator status

Aviator will automatically update this comment as the status of the PR changes.
Comment /aviator refresh to force Aviator to re-examine your PR (or learn about other /aviator commands).

This PR was merged using Aviator.

Stack

• 👉 #29 add queue wait histogram, oldest queued age, in-flight cache (this PR)
• #28 add failed runtime, online worker count, hostname label (merged)
• main

---

See the real-time status of this PR on the Aviator webapp.

Use the Aviator Chrome Extension to see the status of your PR within GitHub.

[gemini-code-assist]

Code Review

This pull request adds tracking for in-flight Celery tasks to monitor queue wait times and the age of the oldest queued tasks, introducing new Prometheus metrics and CLI configuration for cache management. The implementation includes logic to handle task lifecycle events and manage an in-flight task cache. Feedback focused on potential performance bottlenecks during metric collection from the large cache and suggested an optimization for the TTL check using the chronological properties of the data structure.