Skip to content

Issue: Silent notification drop under backpressure in NotificationClient #217

Description

@MayankSharmaCSE

Summary

The notification dispatcher inside (*NotificationClient).listen performs a non-blocking send to each subscriber's channel and silently discards the notification when the buffer is full or the receiver is gone. The drop path contains only the comment // message dropped , no log line, no metric, no error propagation. Because the txID is removed from the subscriber map before delivery is attempted , a dropped notification is unrecoverable: the caller waiting in WaitForEvent will time out as if the committer never responded.

Affected Code

File: tools/fxconfig/internal/client/notifications.go

// lines 223–228
for _, c := range notifications {
    select {
    case c.receiverQueue <- c.status:
    default:
        // message dropped
    }
}

Reproduction Conditions

The default branch is reached whenever any of the following holds at the moment of dispatch:

  • Subscriber timed out early - WaitForEvent already returned via its ctx.WithTimeout; the channel was never drained.
  • Late / duplicate status event - a second event for the same txID arrives after the first already filled the buffer slot.
  • Abandoned subscriber - the caller subscribed but never called WaitForEvent (e.g., error on the submission path before the wait).
  • Slow consumer - the subscriber goroutine is not scheduled fast enough to drain the channel before the dispatcher fires.

Root Cause

listen uses a non-blocking send to keep one slow subscriber from stalling
the entire dispatcher goroutine , that design intent is correct. However, the
fallthrough default branch has no instrumentation. The combination of:

  1. pre-delivery subscriber map deletion.
  2. A bare default with a comment.
    means any backpressure scenario silently loses state without any signal to
    operators or developers.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions