Skip to content

Record module failures - #340

Merged
obelisk merged 4 commits into
obelisk:mainfrom
wbssbw:prometheus/track-module-failures
Jul 24, 2026
Merged

Record module failures#340
obelisk merged 4 commits into
obelisk:mainfrom
wbssbw:prometheus/track-module-failures

Conversation

@wbssbw

@wbssbw wbssbw commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

This pull request adds new metrics to track module failures in the executor, improving observability for error monitoring. The main change is the introduction of a new IntCounterVec metric to count failures per module, along with the necessary code to increment this metric when a module fails during execution.

Metrics enhancements:

  • Added a new IntCounterVec named module_failures to the ModuleExecutionMetrics struct to count the number of failures per module.
  • Registered the new module_failures metric with the metrics handle during initialization.
  • Implemented a new method record_module_failure in ModuleExecutionMetrics to increment the failure counter for a specific module.
  • Updated the executor logic in process_message_with_module to record a module failure using the new metric whenever a module returns a non-zero value (indicating an error).

Copilot AI review requested due to automatic review settings July 8, 2026 14:24

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new Prometheus counter to track per-module failures in the executor to improve observability of module error rates.

Changes:

  • Introduces a module_failures: IntCounterVec in ModuleExecutionMetrics and registers it with the metrics handle.
  • Adds record_module_failure() helper to increment the counter for a given module label.
  • Increments the failure counter in process_message_with_module when a module returns a non-zero status.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
runtime/plaid/src/executor/mod.rs Records a module failure metric when the module entrypoint returns a non-zero value.
runtime/plaid/src/executor/metrics.rs Adds and registers the new module_failures counter and exposes record_module_failure().

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread runtime/plaid/src/executor/metrics.rs Outdated
Comment thread runtime/plaid/src/executor/metrics.rs Outdated
Copilot AI review requested due to automatic review settings July 8, 2026 14:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

Comment thread runtime/plaid/src/executor/metrics.rs
Comment thread runtime/plaid/src/executor/metrics.rs Outdated
Copilot AI review requested due to automatic review settings July 8, 2026 14:30
@wbssbw
wbssbw marked this pull request as ready for review July 8, 2026 14:32
@wbssbw
wbssbw requested a review from obelisk July 8, 2026 14:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

Comment thread runtime/plaid/src/executor/metrics.rs
Comment thread runtime/plaid/src/executor/mod.rs
.expect("expected unique collector");
handle
.register(Box::new(module_failures.clone()))
.expect("expected unique collector");

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This probably should have been caught in a previous review but expect could cause a panic. Are we not worried about that?

Copilot AI review requested due to automatic review settings July 24, 2026 19:26
@obelisk
obelisk merged commit 6f74960 into obelisk:main Jul 24, 2026
20 checks passed

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants