feat: improved leopard saturation metrics

[pmcclory-pp]

This improves the Leopard saturation metrics. The original version of the metrics assumed that the processing concurrency for a subject was always set to the default of 1. So if the semaphore for a given subject had no available permits, then the subject for the worker was busy - if it did it was idle.

In our prod environment where we are using leopard we monkey patch nats-pure to increase the processing concurrency which breaks this assumption. Update to return the number of actual inflight requests for a subject (in flight means that less then the processing concurrency is in flight - it still might not get immediately run by the executors thread pool if that is saturated - more on that below).

The second issue was subject saturation was telling an incomplete story of the full worker saturation. Each worker gets a thread pool with 24 threads, and all subjects requests are assigned to that thread pool. So it is possible with enough subjects registered that no individual subject will be fully saturated but the threadpool itself is. Expose metrics on this as well.

Here's a summary of the metric changes:

Metrics

Subject metrics (labels: subject, worker)

• leopard_subject_busy_slots — thread slots actively processing a message for this subject on this worker
• leopard_subject_capacity_slots — total thread slots allocated for this subject on this worker (reflects actual configured processing concurrency, not the default of 1)
• leopard_subject_pending_messages — messages waiting to acquire a processing slot for this subject on this worker

Executor metrics (labels: worker)

• leopard_executor_active_threads — approximate number of active threads in the worker's subscription executor thread pool
• leopard_executor_max_threads — maximum threads available in the subscription executor thread pool
• leopard_executor_queued_tasks — tasks holding a semaphore permit but waiting for a free executor thread; nonzero only when the executor pool is fully saturated

Replaced metrics

• leopard_subject_busy_instances → leopard_subject_busy_slots / leopard_subject_capacity_slots
• leopard_subject_total_instances → leopard_subject_capacity_slots
• leopard_subject_pending_messages — retained but labels changed (now includes worker)

Note - opened as a feat: - but could see the argument that this is a breaking change - let me know if it should be major release

[bougyman]

Feat seems reasonable to me.