oci: Add varlink APIs using "splitdirfdstream"

[cgwalters]

First, I discovered that actually fd-passing with varlink generally works well, and I was misguided in thinking we needed jsonrpc-fdpass.

Almost: one issue is that varlink doesn't have good support for passing a lot of file descriptors (which jsonrpc-fdpass was designed to handle).

But upon some reflection, I realized we don't need to pass a file descriptor per file, all use cases here are fine with a directory fd plus filename.

So here a new data stream format "splitdirfdstream" is implemented.

We first now use that internally when we're doing a direct pull from containers-storage for reflinking/hardlinkling.

But better: let's expose that data concept over varlink, where a varlink client can both pull or push container image layers that way.

This paves the way to a very clear mechanism for us to integrate with containers-storage or other storage stacks (like containerd) in an agnostic way.

We also now support cfsctl oci copy to copy across composefs repositories which is also implemented this way.

Generated-by: OpenCode (Claude Opus 4.8)

[giuseppe]

how will this work with podman-container-tools/container-libs#651 ?

I don't think the container-tools are going to add again varlink as a dependency for this use case only

[cgwalters]

One thing to bear in mind is there's two levels to this, the splitdirfdstream (replacing splitfdstream) and the higher level IPC mechanism by which that is passed around as representations of OCI layers. Only the latter involves varlink, and we could choose to keep using jsonrpc-fdpass for the latter in container-tools if you prefer.

Sorry about the proposed strategy change, but as I dug in more it just feels right - and I somehow again just missed the fd passing support in the varlink ecosystem. I guess one thing that changed is zlink is relatively new, and is well maintained and good code. (The varlink/rust project went through a messy time)

Would varlink be heavier than jsonrpc-fdpass? Hmm, let me see...I spent some tokens on this

varlink/go vs jsonrpc-fdpass-go: binary size comparison

🤖 Assisted-by: OpenCode (Claude Sonnet 4.6)

Measured against the containers-storage binary built from origin/main with
-ldflags="-s -w" -tags exclude_graphdriver_btrfs. Each branch adds an
equivalent stub service (listen → accept → receive → reply) in
cmd/containers-storage/rpc_bench_stub.go, guarded by a package-level var so
the linker cannot dead-code-eliminate it. Branches: bench/varlink,
bench/jsonrpc-fdpass.

Dependency	Size (KiB)	Δ KiB	Δ %	Δ packages
baseline	8,003.8	—	—	244
+varlink/go	8,071.9	+68.1	+0.851%	246
+jsonrpc-fdpass-go	8,083.9	+80.1	+1.001%	245

With a realistic service stub, both libraries are similar in weight (~68–80 KiB).
varlink/go is slightly smaller despite pulling in 2 extra packages vs 1,
likely because jsonrpc-fdpass-go depends on golang.org/x/sys for
syscall.RawConn fd passing whereas varlink/go has no external dependencies.

---

That said, varlink/go#43 needs doing.

[cgwalters]

Only the latter involves varlink, and we could choose to keep using jsonrpc-fdpass for the latter in container-tools if you prefer.

Or, it'd probably work to use gRPC for most metadata/controlplane but pass fds over a separate negotiated socket. I don't have a really strong opinion.

---

One other thing I'd say here that I think is a big cleanup is that our parsing of containers-storage: layouts now always goes through splitdirfdstream(+varlink) as an intermediary - we've added an abstraction layer there that we're now always testing. This helps pave the way much more clearly to plugging in an external binary.

Also the converse is now true - it should now be straightforward for external tooling to push content into composefs-rs in an efficient way.

[giuseppe]

That said, varlink/go#43 needs doing.

This is a blocker for the containers/storage integration. I have no preference whether it is varlink or jsonrpc, but should we hold this until we know we can use it in containers/storage too?

[cgwalters]

This is a blocker for the containers/storage integration. I have no preference whether it is varlink or jsonrpc, but should we hold this until we know we can use it in containers/storage too?

I generated a PR in varlink/go#44

I think we don't need to block this strictly speaking, it seems in the end better for us to temporarily use a patched varlink/go than to use the custom jsonrpc-fdpass right? Also, if we can take this track to finally replace the current experimental-image-proxy protocol with one thing it'd be overall a large win.

[giuseppe]

I think we don't need to block this strictly speaking, it seems in the end better for us to temporarily use a patched varlink/go than to use the custom jsonrpc-fdpass right? Also, if we can take this track to finally replace the current experimental-image-proxy protocol with one thing it'd be overall a large win.

@mheon are you fine with that?

[mheon]

Sorry for not following this one closely. Are we talking about doing a hard dependency on Varlink in order to use composefs with c/storage? Do we know what that's going to do to our binary size?

[giuseppe]

Sorry for not following this one closely. Are we talking about doing a hard dependency on Varlink in order to use composefs with c/storage? Do we know what that's going to do to our binary size?

yes, to add it as the RPC to communicate between Go and Rust.

@cgwalters made a comparison here:

#309 (comment)

[cgwalters]

OK, this one I think is ready for review/merge.

[mheon]

I don't think we have a fundamental disagreement with Varlink as an IPC protocol, though I am worried about protocol design. What kind of long term stability are we expecting with this? Are we going to have to do a hard pinning of composefs-rs against c/storage versions to ensure both ends are talking the same version of the protocol?

[cgwalters]

Are we going to have to do a hard pinning of composefs-rs against c/storage

Right, this is a confusing topic. At the current time, containers-storage uses /usr/bin/mkcomposefs from https://github.com/composefs/composefs - the C implementation.

We have an effort to fully replace that project with this one, including a new Rust mkcomposefs that has landed here that aims to be 100% compatible.

The trajectory is then to ship the composefs package with this project.

But that's not (directly) related to this PR or the PR Giuseppe is working on, which aim to expose varlink APIs on both ends.

I think the trajectory that would help the most is to replace skopeo --experimental-image-proxy (that's used by bootc/ostree) with a varlink-based interface, that's what's going on in podman-container-tools/container-libs#651 - but that needs updating to match this.

In that flow, it's more the other way around again: bootc/composefs(-rs) would be calling into skopeo ➡️ container-libs via varlink.

[cgwalters]

OK! I've gone over and cleaned this up more. I think it's ready for a wider review - I've trimmed out some garbage etc. One thing I had my agent do was inject random "dummy" fds into the producer stream to ensure that the consumer side was correctly reading the indices (and not just e.g. hardcoding 0 to access the dirfd), that shook out some bugs.

[alexlarsson]

I did a highlevel pass through this code, but man, there is a lot of code. Also, I think @giuseppe need to review the storage related parts.

[Copilot]

Pull request overview

This PR introduces a new layer-transfer mechanism based on a splitdirfdstream wire format and exposes it over the org.composefs.Oci varlink (zlink) interface, replacing the prior JSON-RPC+fd-passing approach for the userns helper. It also adds repo↔repo cfsctl oci copy support implemented via the same varlink transfer primitives.

Changes:

• Add the composefs-splitdirfdstream crate (transport/layout/framing + keepalive protocol) and remove the older composefs-splitfdstream crate.
• Implement/extend org.composefs.Oci varlink types + repo/cstor producer plumbing (including a new storage lock guard for c/storage interoperability).
• Add CLI + integration tests for cross-repo OCI image copy and update varlink integration tests to use socket activation.

Reviewed changes

Copilot reviewed 31 out of 31 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
crates/composefs/src/splitstream.rs	Adds chunk-walking helper for reference-preserving stream traversal.
crates/composefs-storage/src/userns_helper.rs	Replaces JSON-RPC helper protocol with zlink org.composefs.Oci service connection management.
crates/composefs-storage/src/tar_split.rs	Adds overlay metacopy stub detection, link-id propagation, and expanded tests.
crates/composefs-storage/src/lock.rs	Introduces shared flock guard for containers-storage layer-store safety during streaming.
crates/composefs-storage/src/lib.rs	Wires new modules/features (layer-transfer, cstor service exports, lock exports).
crates/composefs-storage/src/layer.rs	Simplifies layer-chain traversal using lower’s full ordered parent list + new tests.
crates/composefs-storage/Cargo.toml	Replaces userns-helper with layer-transfer feature and updates deps.
crates/composefs-splitfdstream/src/lib.rs	Removes the older split-fd stream implementation.
crates/composefs-splitfdstream/Cargo.toml	Removes the older split-fd stream crate manifest.
crates/composefs-splitdirfdstream/src/transport.rs	Adds the new splitdirfdstream transport/layout/framing implementation + tests.
crates/composefs-splitdirfdstream/README.md	Adds crate-level README describing purpose and docs location.
crates/composefs-splitdirfdstream/Cargo.toml	Adds new crate manifest and feature flags.
crates/composefs-oci/src/varlink_types.rs	Adds shared varlink wire types + OciProxy trait for org.composefs.Oci.
crates/composefs-oci/src/test_util.rs	Adds helpers to synthesize OCI layers/config/manifest for tests.
crates/composefs-oci/src/lib.rs	Exposes additional OCI helpers/constants and adds new modules for layer sync/transport.
crates/composefs-oci/src/layer_transport.rs	Adds repo-side GetLayer producer abstraction and framing orchestration.
crates/composefs-oci/src/cstor.rs	Refactors containers-storage import to use org.composefs.Oci layer streaming (direct + proxied).
crates/composefs-oci/Cargo.toml	Wires new varlink/containers-storage feature composition and adds splitdirfdstream dep.
crates/composefs-integration-tests/src/tests/varlink.rs	Switches varlink service spawning to socket activation and simplifies setup.
crates/composefs-integration-tests/src/tests/privileged.rs	Exposes loop-mount helpers for reuse by new tests.
crates/composefs-integration-tests/src/tests/mod.rs	Registers new copy_image integration test module.
crates/composefs-integration-tests/src/tests/copy_image.rs	Adds end-to-end tests for repo↔repo image copy (CLI + varlink + cross-algorithm).
crates/composefs-integration-tests/src/main.rs	Adds socket-activated cfsctl spawn helper used by integration tests.
crates/composefs-integration-tests/Cargo.toml	Adds test dependency needed for artifact-manifest construction.
crates/composefs-ctl/src/lib.rs	Adds oci copy command + a copy_image helper using varlink GetLayer/PutLayer/FinalizeImage.
crates/composefs-ctl/Cargo.toml	Wires new optional deps/features needed for copy and splitdirfdstream support.
Cargo.toml	Switches default-members from removed splitfdstream crate to splitdirfdstream.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.