GH-49946: [Format] Better document equivalence between IPC file and streams#49947
GH-49946: [Format] Better document equivalence between IPC file and streams#49947pitrou wants to merge 1 commit intoapache:mainfrom
Conversation
github-actions
Bot
added
Component: Documentation
awaiting review
pitrou
force-pushed
the
ipc-file-stream-equivalence
branch
from
3da85f2 to
6ea2485
Compare
|
@github-actions crossbow submit preview-docs |
|
Revision: 6ea2485 Submitted crossbow builds: ursacomputing/crossbow @ actions-0208c5a8f4
|
pitrou
force-pushed
the
ipc-file-stream-equivalence
branch
from
6ea2485 to
28765e5
Compare
|
@github-actions crossbow submit preview-docs |
|
Revision: 28765e5 Submitted crossbow builds: ursacomputing/crossbow @ actions-6cfd80b0e4
|
pitrou
force-pushed
the
ipc-file-stream-equivalence
branch
from
28765e5 to
51fb5a1
Compare
|
@github-actions crossbow submit preview-docs |
|
Revision: 51fb5a1 Submitted crossbow builds: ursacomputing/crossbow @ actions-3cd6e7c16a
|
paleolimbot
left a comment
paleolimbot
left a comment
There was a problem hiding this comment.
Thank you for writing this up!
| The ``Buffer`` Flatbuffers value describes the location and size of a buffer's | ||
| data, relatively to the start of the RecordBatch message's body. |
There was a problem hiding this comment.
I believe the offsets can be global in the dissociated IPC protocol (metadata and bodies sent on separate streams), although I forget if that was ever actually implemented anywhere.
There was a problem hiding this comment.
Hmm, the question is then whether this section of the doc describes the IPC protocol in a narrow sense, or whether it applies to any variation over it such as the Dissociated IPC protocol.
For readability I think it's better if it describes the concrete IPC protocol as commonly implemented. What do you think?
There was a problem hiding this comment.
I don't mind either way, I was just reminded of it when Ben was working on the nanoarrow IPC writer. Will RST let you make a footnote?
| compliant writers SHOULD arrange the IPC File footer so that an IPC File can be | ||
| read using an IPC Stream reader with equivalent results. |
There was a problem hiding this comment.
It may be nice at some point to indicate in the "features" section of a flatbuffers Schema that the stream can definitely be read as an IPC stream (i.e., doesn't differ between what one would get from reading using the blocks in the footer). The fact that nanoarrow does this blindly is not great and I'll fix it, but it is a cool feature that you can do full scans without random access in most cases.
There was a problem hiding this comment.
What do you call the features section? Is it
Lines 71 to 81 in a0d2885
There was a problem hiding this comment.
That's what I was thinking of (but no need to deal with this now 🙂 )
github-actions
Bot
added
awaiting merge
github-actions
Bot
added
awaiting changes
2 participants
Rationale for this change
As discussed in https://lists.apache.org/thread/jpxl3yzm96wkxzb1clokxklsy32b3plh, we want to better document the rough equivalence between IPC files and streams.
What changes are included in this PR?
Are these changes tested?
N/A.
Are there any user-facing changes?
No.