Skip to content

network2: housekeeping-driven senders can double-send before Sent is applied #789

Description

@scarmuega

Context

Follow-up from CodeRabbit review on #788.

The initiator sub-behaviors that send requests during housekeeping pop the request from their queue and emit the Send command before the corresponding InterfaceEvent::Sent advances the per-peer protocol state. Because visit_housekeeping is driven purely by the app calling InitiatorCommand::Housekeeping (no interface emits InterfaceEvent::Idle), two housekeeping passes can occur with no intervening poll_next (e.g. a fast housekeeping timer under load in the canonical select! driver loop).

In that window the per-peer state still looks idle, so a second request is sent for the same peer while one is already in flight. The state machine then rejects the second message (Err(InvalidInbound)), which flips state.violation = true → the peer is banned/disconnected. Additionally, if the outbound send fails (Error instead of Sent), the already-removed request is dropped with no retry.

Affected code

  • pallas-network2/src/behavior/initiator/leiosfetch.rsvisit_housekeeping pops via self.requests.remove(idx) then send_request, guarded only by peer_is_available (leios_fetch == Idle(None)).
  • pallas-network2/src/behavior/initiator/blockfetch.rssame pattern (requests.pop_front() + request_block_batch, guarded by blockfetch == Idle); a second RequestRange while Busy(_) likewise yields Err(InvalidInbound).

This is a pre-existing, codebase-wide pattern, not specific to Leios — hence a separate follow-up rather than a fix scoped to the Leios PR.

Proposed fix

Make housekeeping-driven sends robust to the queue→Sent gap, consistently across blockfetch and leiosfetch:

  • Defer removing the request from the queue until the send is confirmed (handle in visit_outbound_msg once the state transitions out of idle), or
  • Track a per-peer "pending send" flag set when the Send is queued and cleared on Sent/Error, so housekeeping won't re-pop while a send is unconfirmed.
  • On send failure (Error), requeue rather than drop.

Acceptance

  • No duplicate request can be emitted for a peer with one already in flight (no self-inflicted protocol violation under a fast housekeeping cadence).
  • An unsent request survives a send failure.
  • Behavior is consistent between blockfetch and leiosfetch.

Refs: #788 (CodeRabbit comment on leiosfetch.rs).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions