Add guide on reassembling multi-segment messages - #407
Conversation
|
📦 Preview · 🟠 Still propagating The deploy completed, but the public URL has not served build |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7ce795c8a1
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
0f1ac73 to
8ae0c10
Compare
2ccacdd to
64ca62a
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 64ca62a1a6
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
64ca62a to
edf17ea
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: edf17ea38e
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
edf17ea to
9088017
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9088017348
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
9088017 to
6aa94b7
Compare
6aa94b7 to
bb00bf8
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bb00bf81c8
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
1d27d69 to
6ffce79
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1d27d694b9
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
6ffce79 to
6b8c66d
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6b8c66dd45
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
90398fe to
a3a1c2a
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a3a1c2a3ec
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
a3a1c2a to
396d3d5
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 396d3d5c81
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
396d3d5 to
f156ff6
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f156ff6540
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| - **Reused identifiers.** A reused single-segment id is disambiguated by its | ||
| payload; reused multi-segment ids are not reliably solvable, because a | ||
| continuation does not say which header it belongs to. |
There was a problem hiding this comment.
Fresh evidence is this limitation bullet, which now says reused single-segment ids are disambiguated by payload, but the gather pipeline groups only by message_id and the duplicate step keys only message_id, segment_number. When two one-segment messages reuse the same id within the same 3×span window, their payloads are concatenated into one message (or one is dropped after deduplicate), so this exception contradicts the documented reuse constraint and can make readers accept an unsafe reuse interval.
Useful? React with 👍 / 👎.
|
|
||
| ```tql | ||
| window size=$span * 3, every=$span, on=ts, idle_timeout=$span { | ||
| // ...sort, deduplicate, emit guard as before... |
There was a problem hiding this comment.
When readers splice this completeness snippet into the gather pipeline, this comment places the “emit guard as before” before summarize, but that guard relies on first_segment_window_start and header_window_start, which are only created by the summarize below. Following the snippet literally filters on missing fields or before grouping, so the snippet should show the guard after the summarized fields are produced.
Useful? React with 👍 / 👎.
f156ff6 to
87f676b
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 87f676b929
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| Each message comes out whole even though its segments arrive out of order and | ||
| seconds apart. What matters is the spread of one message's own segments, not their | ||
| arrival order or the real-time gaps in the stream. Remove `delay` to replay the |
There was a problem hiding this comment.
With the copied windows setting idle_timeout=$span, a live or sparse source that pauses for more than $span of wall-clock time between segments can have the target window force-closed before the next segment arrives, even when their ts values are still within the reassembly span; the later segment is then late/dropped or emitted separately. This sentence tells readers real-time gaps do not matter, so the guide should either size idle_timeout to the expected inter-segment wall-clock gap or call out that limitation.
Useful? React with 👍 / 👎.
Document the pattern for reassembling a logical message that a source split across several transport messages, matching the cisco::ise::reassemble operator: frame each segment, gather them with overlapping (hopping) event-time windows and an emit guard, deduplicate redelivered segments, tell reused single-segment ids apart, and report completeness so incomplete messages flow downstream instead of vanishing. Include a delay-based example for live timing. Add a reciprocal See also link on the window reference, and document on the raw parsing option that inference drops leading zeros from numeric identifiers. Assisted-by: Claude Opus 4.8 (1M context) (Claude Code)
87f676b to
510acd9
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 510acd975a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| The Tenzir Library's [Cisco package](https://github.com/tenzir/library/tree/main/cisco) | ||
| applies the whole pattern in this guide to Cisco ISE syslog, as the | ||
| `cisco::ise::reassemble` and `cisco::ise::parse` operators. |
There was a problem hiding this comment.
The linked Cisco package currently only exposes Umbrella DNS content, so readers who follow this callout cannot find cisco::ise::reassemble or cisco::ise::parse in the referenced main/cisco tree. Until those operators are published at that link, this sends users looking for the full implementation to a package that does not contain it; either link to the exact branch/path that has ISE support or avoid naming these operators here.
Useful? React with 👍 / 👎.
|
|
||
| By default the parser infers richer types from values, which can change them: a | ||
| zero-padded numeric identifier such as `0001133664` becomes the number | ||
| `1133664`, losing its leading zeros. Set `raw=true` to keep such fields as text | ||
| so they survive verbatim. |
There was a problem hiding this comment.
This partial is also rendered on JSON/YAML/XML parser pages, where the new example is misleading: a quoted JSON identifier like "0001133664" is already a native string and is not inferred into 1133664 unless a schema/conversion says so. Because this shared option text now tells readers to use raw=true for a transformation those parsers do not perform by default, please qualify the warning for grok/xsv-style inference or move the zero-padding example out of the generic partial.
Useful? React with 👍 / 👎.
2 participants
🔍 Problem
🛠️ Solution
guides/parsing/reassemble-multi-segment-messages(sidebar: Guides → Parsing).parse_grok, gather them with an outerwindowand an innergroupkeyed on the id, thensort+collect+firstto concatenate payloads in order.windowoutermost (notgroup) so state is bounded to the open windows rather than one subpipeline per near-unique id; the note explains why and when to invert.cisco::ise::*operators as a real-world implementation.💬 Review
- Example pipelines verified to run;
⚙️ Code PR: tenzir/library#161window-outer vsgroup-outer equivalence verified by running both on interleaved sample data; output is identical.markdownlintpasses on the changed file.