Add extract-to-cache build mode for the Copilot CLI

[tclem]

Implements the SDK side of the CLI provisioning redesign in github/github-app#5700. Dev consumers stop needing a separate CLI install, an env-var dance, or the binary-size hit of bundled-cli. Release consumers behave exactly as before.

Approach

One existing cargo feature (bundled-cli, default on) selects between two modes that share the same per-version cache layout:

• On (release): build.rs downloads + SHA-verifies the archive and embeds it via include_bytes!(). Runtime extracts on first call via src/embeddedcli.rs::install_at. Unchanged.
• Off (dev): build.rs downloads + SHA-verifies the same archive and extracts the single binary directly into the per-user cache. Emits cargo:rustc-env=COPILOT_CLI_DEV_PATH=<path> and cargo:rustc-cfg=has_dev_cli. Runtime returns that path via env!().

Both modes write to <cache>/github-copilot-sdk/cli/<sanitized version>/<binary>. Extraction is idempotent (is_file() short-circuit + direct fs::write + cfg(unix) chmod) and matches the embed-mode install() pattern that already ships to Windows.

src/resolve.rs resolves the binary in this order: CliProgram::Path -> COPILOT_CLI_PATH (warn on stale) -> embedded extract (when bundled-cli is on) -> dev cache path (when off) -> Error::BinaryNotFound. No PATH scanning at any layer.

Pin file

Version + per-asset SHA-256 hashes live in a single rust/cli-version.txt, written by rust/scripts/snapshot-bundled-cli-version.sh. It is committed into published crates and into github-app's vendored slot. Format:

version=1.0.55-1
copilot-darwin-arm64.tar.gz=<sha>
... (one line per supported platform)

build.rs reads cli-version.txt when present (published crates, vendored slots) and otherwise falls back to ../nodejs/package-lock.json for contributor builds inside this repo. Same convention as the .NET and Go SDKs.

Committing the SHAs at publish time keeps the publish workflow as the supply-chain trust boundary: every consumer build verifies the downloaded archive against a hash captured before any later release-tag tampering could swap the archive and SHA256SUMS.txt together.

Reviewer notes

• ClientOptions::with_bundled_cli_extract_dir is a no-op in dev mode. The binary's path is baked into the crate at build time; there's no embedded archive to re-extract. Use CliProgram::Path or COPILOT_CLI_PATH to override at runtime.
• Cache path moves from <cache>/github-copilot-sdk-{version}/ to <cache>/github-copilot-sdk/cli/<version>/. Existing per-user installs become orphaned; harmless, users can clean whenever.
• CI shares BUNDLED_CLI_CACHE_DIR between the test (--no-default-features, dev mode) and bundle (default features, embed) jobs so the archive doesn't download twice per matrix shard. Windows, macOS, and Linux are in the matrix for both.
• Single-file archive. The github/copilot-cli release tarballs and zips ship one static binary at the root (copilot / copilot.exe); the extraction code expects exactly that layout in both modes.
• README ## Embedded CLI has a callout warning that default-features = false is dev-only (the resulting binary is not self-contained).

Test plan

• cargo test (default features): lib + 3 new cli_resolution_test cases.
• cargo test --no-default-features --features test-support: lib + 4 new cases (extra: dev_mode_extracted_binary_exists).
• cargo clippy --all-targets --features test-support -- -D warnings -D clippy::unwrap_used: clean.
• nightly cargo fmt --check: clean.
• RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features: clean.
• End-to-end on macOS: cargo check --no-default-features populates ~/Library/Caches/github-copilot-sdk/cli/1.0.55-1/copilot and ./copilot --version runs.
• Tampered hash: corrupting one entry in cli-version.txt reproduces the existing "Archive integrity check failed... could indicate a supply-chain attack" panic from cached_download.

The github-app consumer-side cleanup (removing .sdk-cli-pin, the install script, ensure-copilot-cli.ts, dev_cli_program(), etc.) lands in a separate PR there.

  _{Generated via Copilot (Claude Opus 4.7 (1M context)(Internal only)) on behalf of @tclem}

[Copilot]

Pull request overview

This PR updates the Rust SDK’s Copilot CLI provisioning so that dev builds (default-features = false, i.e. bundled-cli off) download + verify the CLI at build time and extract it directly into the per-user cache, while release builds continue embedding the verified archive and lazily extracting at runtime.

Changes:

• Add dev-mode “extract-to-cache” provisioning in build.rs, emitting COPILOT_CLI_DEV_PATH for runtime resolution.
• Update Rust runtime CLI resolution order and docs to include the dev-cache path and new cache layout.
• Add Rust integration tests and CI cache plumbing for the new build behavior.

Show a summary per file

File	Description
rust/tests/cli_resolution_test.rs	New integration tests covering env override, dev-cache extraction, and pin-file format.
rust/src/resolve.rs	Adds dev-cache resolution path when bundled-cli is disabled; updates docs/warnings.
rust/src/lib.rs	Gates embeddedcli on bundled-cli and updates public docs for resolution/extract-dir behavior.
rust/src/embeddedcli.rs	Updates docs + cache layout to github-copilot-sdk/cli/<version>/… and gates APIs on bundled-cli.
rust/scripts/snapshot-bundled-cli-version.sh	Renames snapshot output to cli-version.txt.
rust/README.md	Updates embedded/dev provisioning explanation, cache layout, and resolution priority docs.
rust/Cargo.toml	Renames packaged snapshot file to cli-version.txt; adds dirs build dependency.
rust/build.rs	Implements shared download+verify; branches into embed vs extract-to-cache modes; introduces dev-cache extraction.
rust/.gitignore	Ignores cli-version.txt instead of the old snapshot filename.
.github/workflows/rust-sdk-tests.yml	Adds CLI-version keying and shares the build.rs archive cache between test/bundle jobs.
.github/workflows/publish.yml	Updates publish verification step for cli-version.txt.

Copilot's findings

• Files reviewed: 11/11 changed files
• Comments generated: 8

[SteveSandersonMS]

Thanks @tclem!

The overall concept here looks great. I'm proposing a few amendments that I think generalize this more beyond the GitHub Copilot App workflows, but hopefully without taking away anything you want to use in those workflows.

For speed, I did that by pushing an extra commit to your branch. If you don't like it, we can undo that commit and discuss!

Overview

The following are small adjustments on top of what you already built.

1. New env var: COPILOT_SKIP_CLI_DOWNLOAD

Setting COPILOT_SKIP_CLI_DOWNLOAD=1 at build time makes build.rs return immediately — no download, no bundle, no cache write — regardless of the bundled-cli feature. Lets consumers who always supply the CLI at runtime (via CliProgram::Path / COPILOT_CLI_PATH) build fully offline.

2. Skip redundant downloads when bundled-cli is off

If the extracted binary is already at the conventional path, build.rs now skips cached_download entirely. The extracted binary is the cache — no separate archive cache to maintain. Previously each clean build re-fetched the archive before discovering it didn't need it.

3. New env var: COPILOT_CLI_EXTRACT_DIR (symmetric)

When set, both build.rs (write side) and the runtime resolver (read side) place/look-for the binary directly under that directory (no per-version subdir). Recommended consumer pattern is .cargo/config.toml [env] with relative = true, force = true so cargo applies the same value to build and to cargo run/cargo test. Gives consumers a way to relocate the extraction symmetrically (CI runners with ephemeral homes, vendored slots, etc.).

4. Bake only the version, not an absolute path

build.rs now always emits cargo:rustc-env=COPILOT_SDK_CLI_VERSION=. Both modes consume it. With bundled-cli off, the runtime resolver recomputes the path from COPILOT_SDK_CLI_VERSION + cfg!(windows) + optional COPILOT_CLI_EXTRACT_DIR — the same convention build.rs uses on the write side.

Previously the absolute extracted path was baked into the rlib via env!("COPILOT_CLI_DEV_PATH"). That leaks $HOME/$LOCALAPPDATA into compiled artifacts and breaks sccache / cross-machine target/ reuse. The version string is ~12 bytes, machine-independent, and is already the natural single source of truth for both modes (embed-mode constants now consume it too, eliminating the prior risk of build-time / runtime version drift).

5. Atomic extraction (restored)

extract_to_cache writes to a sibling - staging file and then fs::renames onto the final path. fs::rename of a file is atomic on both Unix and Windows, so a parallel cargo build on a fresh machine never observes a partially-written binary. Earlier commits in the PR had staging+rename and dropped it — putting it back.

6. Docs / terminology

• README and source comments avoid coupling "dev" with "bundled-cli off" — they're orthogonal concepts (you might well disable bundling in a release build, and you might well keep bundling on while developing). Behavior is now described in terms of whether bundled-cli is on or off, which is what consumers actually toggle.
• The README caveat for bundled-cli = off is stronger: the consumer is responsible for supplying a compatible CLI at runtime via CliProgram::Path / COPILOT_CLI_PATH; do not rely on whatever CLI happens to be installed on the target system. The build-machine auto-resolution is called out explicitly as a convenience-only special case that does not carry over when the produced binary is copied to another machine.
• ClientOptions::with_bundled_cli_extract_dir docs clarify it applies only when bundled-cli is on, and point at COPILOT_CLI_EXTRACT_DIR for the other case.

Tests

Your cli_resolution_test.rs is preserved and extended with two more cases:

• extracted_binary_present_at_conventional_path — asserts the file lands at the convention-derived path (mirroring what the runtime resolver computes), rather than at a baked path.
• extract_dir_runtime_override_is_honored — writes a fake binary into a tempdir, sets COPILOT_CLI_EXTRACT_DIR, and checks the resolver picks it up

Does this fit with what you need?

[tclem]

yes, thanks! i actually had an unpushed commit doing some of that but yours is better!

[tclem]

I'm going to fully round trip this with our client to make sure the dx lines up, but otherwise this is looking great!

[tclem]

Small refinement: gate rerun-if-changed=../nodejs/package-lock.json on lockfile presence.

Observed while adopting this PR on the github-app consumer side: when this crate is built from a vendored slot (no sibling nodejs/ directory), build.rs:14 emits a cargo:rerun-if-changed for a path that doesn't exist:

println!("cargo:rerun-if-changed=../nodejs/package-lock.json");

Cargo treats a missing rerun-if-changed path as "always rebuild the build script's outputs," so consumers see the SDK crate re-run + recompile on every cargo build — even when nothing actually changed. The download is fine (cached_download short-circuits on hit, extract_to_cache short-circuits on existing file), so it's not a perf or correctness bug, just cosmetic churn. But it defeats the "Dev is fast / per-build overhead must be ~zero unless the pinned version changes" framing in docs/copilot-cli-bundling.md.

Suggested one-liner: only emit the lockfile rerun when the lockfile is actually present (i.e. when we're building inside the copilot-sdk repo, where it'd be the fallback source for version resolution):

let lockfile = Path::new(env!("CARGO_MANIFEST_DIR")).join("../nodejs/package-lock.json");
if lockfile.exists() {
    println!("cargo:rerun-if-changed={}", lockfile.display());
}

cli-version.txt still gets its rerun-if-changed unconditionally (correct — it's always present in both deployment shapes, and is the source of truth for vendored slots).

Net result: vendored consumers get true zero-cost rebuilds when nothing's changed; in-tree contributor builds still re-run when they bump the lockfile.

  _{Generated via Copilot (Claude Opus 4.7) on behalf of @tclem}

[tclem]

Fixed in 4e65091e. Build.rs now stats the lockfile and emits the rerun-if-changed line only when it exists, so vendored consumers stop forcing a build-script rerun on every cargo build. In-tree contributor builds keep the lockfile-bump rerun behavior. cli-version.txt's rerun-if-changed stays unconditional as you noted.

  _{Generated via Copilot (Claude Opus 4.7 (1M context)(Internal only)) on behalf of @tclem}

[SteveSandersonMS]

Thanks!