Skip to content

enhancement(acknowledgements): support best-effort sink branches - #26143

Draft
Jansen-w wants to merge 1 commit into
vectordotdev:masterfrom
Jansen-w:prototype/selective-sink-acknowledgements
Draft

enhancement(acknowledgements): support best-effort sink branches#26143
Jansen-w wants to merge 1 commit into
vectordotdev:masterfrom
Jansen-w:prototype/selective-sink-acknowledgements

Conversation

@Jansen-w

Copy link
Copy Markdown
Contributor

Summary

This draft explores selective sink participation in end-to-end acknowledgements. A sink with acknowledgements.enabled: false becomes a best-effort branch: it still receives events, but its event copies do not retain source finalizers and therefore cannot delay or affect the upstream acknowledgement result.

The implementation:

  • computes required and explicitly best-effort reachability during configuration compilation
  • carries the resulting requirement on fanout connections
  • removes source finalizers from best-effort event copies before they enter sink buffers
  • preserves existing behavior for sinks whose acknowledgement setting is unspecified
  • treats tap outputs as best effort

Known prototype limitations

  • acknowledgement requirement changes during topology reload do not yet rebuild affected sources or reconnect every unchanged upstream edge
  • best-effort branches still participate in fanout backpressure while their buffers accept events
  • memory and disk buffer lifecycle coverage needs to be expanded before this is ready for production

References

None.

Vector configuration

sources:
  input:
    type: http_server
    address: 0.0.0.0:8080

sinks:
  required:
    type: http
    inputs: [input]
    uri: https://required.example.com/events
    acknowledgements:
      enabled: true
    encoding:
      codec: json

  best_effort:
    type: http
    inputs: [input]
    uri: https://best-effort.example.com/events
    acknowledgements:
      enabled: false
    encoding:
      codec: json

How did you test this PR?

  • cargo test -p vector-core fanout::tests --lib
  • cargo test -p vector --features sources-file,sinks-file config::acknowledgements_tests::propagates_settings --lib
  • cargo test -p vector --features sources-http_server,sinks-http topology::test::end_to_end --lib
  • cargo check -p vector --lib
  • cargo clippy -p vector-core --tests -- -D warnings
  • cargo clippy -p vector --lib --features sources-http_server,sinks-http,sources-file,sinks-file -- -D warnings
  • cargo fmt --all -- --check
  • cargo vdev check changelog-fragments

The end-to-end test blocks a failing best-effort HTTP sink and verifies that the HTTP source responds successfully as soon as the required sink succeeds.

Is this a breaking change?

  • Yes
  • No

Does this PR include user facing changes?

  • Yes. A changelog fragment is included.
  • No.

@github-actions github-actions Bot added domain: topology Anything related to Vector's topology code domain: core Anything related to core crates i.e. vector-core, core-common, etc labels Aug 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

domain: core Anything related to core crates i.e. vector-core, core-common, etc domain: topology Anything related to Vector's topology code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant