Add file-based transfer APIs to build cache provider interface and all cache plugins by Copilot · Pull Request #5746 · microsoft/rushstack

Copilot · 2026-04-05T04:11:54Z

Summary

Adds optional file-based transfer APIs to the build cache provider interface (ICloudBuildCacheProvider) and implements them across all cache plugins (HTTP, Amazon S3, Azure Blob Storage). When enabled via the useDirectFileTransfersForBuildCache experiment flag, cache entries are transferred directly between local files and cloud storage without buffering entire contents in memory, preventing out-of-memory errors for large build outputs.

Details

Core changes:

ICloudBuildCacheProvider gains two optional methods: tryDownloadCacheEntryToFileAsync and tryUploadCacheEntryFromFileAsync. Providers that don't implement them gracefully fall back to the existing buffer-based APIs.
OperationBuildCache conditionally uses the file-based path when useDirectFileTransfersForBuildCache is enabled and the provider supports it. Includes cleanup of partial files on failed downloads.
WebClient is refactored to extract a shared _makeRawRequestAsync core used by both buffer and streaming request paths, with a new fetchStreamAsync method and Content-Encoding decompression support for streaming responses.
FileSystem in node-core-library gains createReadStream, createWriteStream, and createWriteStreamAsync methods (wrapped in _wrapException for consistent error handling).
FileSystemBuildCacheProvider is simplified — the stream method is removed since cloud providers now handle file I/O directly.

Plugin implementations:

HTTP plugin: Downloads via fetchStreamAsync → pipeline() to file. Uploads via createReadStream → fetchStreamAsync. Uses maxAttempts: 1 for uploads (stream consumed after first attempt), with credential fallback skipped for stream bodies.
Amazon S3 plugin: Downloads via streaming GET with retry support. Uploads hash the tarball on disk first via _hashFileAsync, then stream with the SHA-256 hash included in the AWS Signature V4 request — restoring full payload signing (no more UNSIGNED_PAYLOAD). No retry on uploads.
Azure Blob Storage plugin: Downloads via SDK-native blobClient.downloadToFile(). Uploads via blockBlobClient.uploadFile(). Parent directory creation ensured before download.

Gating:

Controlled by useDirectFileTransfersForBuildCache in experiments.json. Defaults to off. Falls back to buffer-based APIs if the cloud provider plugin doesn't implement the file-based methods.

How it was tested

Unit tests for HttpBuildCacheProvider (14 tests): buffer and file-based GET/SET, 404 cache miss, credential fallback skip for file uploads, write-not-allowed checks, retry behavior, pipeline assertions
Unit tests for AmazonS3Client (38 tests): buffer and file-based GET/SET, signed payload hash verification (not UNSIGNED-PAYLOAD), download retry on transient 5xx, no-retry on upload, credential validation, pipeline assertions
All existing tests continue to pass (51 S3 total, 14 HTTP total)
Verified backwards compatibility: when the experiment flag is off (default), all code paths are identical to the pre-change behavior

Impacted documentation

experiments.schema.json updated with useDirectFileTransfersForBuildCache description
common/reviews/api/rush-lib.api.md updated with new API surface
common/reviews/api/node-core-library.api.md updated with new FileSystem stream methods

iclanton · 2026-04-05T04:16:19Z

@copilot - add APIs to FileSystem in node-core-library for the fs APIs you're making use of in this PR. Ideally, none of our projects (except for FileSystem itself) should use fs. Everything should use FileSystem. The change to node-core-library will also need a changefile. Take a look at the conventions in FileSystem and follow that pattern.

Copilot · 2026-04-05T04:31:41Z

@copilot - add APIs to FileSystem in node-core-library for the fs APIs you're making use of in this PR. Ideally, none of our projects (except for FileSystem itself) should use fs. Everything...

Added createReadStream() and createWriteStream() static methods to FileSystem in node-core-library, along with FileSystemReadStream and FileSystemWriteStream type aliases (following the same pattern as FileSystemStats and FolderItem). Updated FileSystemBuildCacheProvider and OperationBuildCache to use the new FileSystem APIs instead of importing from node:fs directly. Change file for @rushstack/node-core-library included as a minor bump. Commit: 468caf7

iclanton · 2026-04-05T04:33:42Z

@copilot - are there async variants of those fs APIs? Should we expose and use those too?

Copilot · 2026-04-05T04:34:33Z

@copilot - are there async variants of those fs APIs? Should we expose and use those too?

No, fs.createReadStream() and fs.createWriteStream() don't have async variants — they're already inherently asynchronous. They return stream objects immediately and all the actual I/O happens asynchronously through the stream events/pipeline. The fs/promises module doesn't include them for this reason.

iclanton · 2026-04-05T05:15:42Z

@copilot - take a look at the other FileSystem APIs. Many have an optional options parameter with useful utilities. For example, the write operations have an option to create the parent folder if it doesn't exist. Consider adding useful options to these new APIs

Copilot · 2026-04-05T05:26:42Z

@copilot - take a look at the other FileSystem APIs. Many have an optional options parameter with useful utilities. For example, the write operations have an option to create the parent folder if it...

Added IFileSystemCreateWriteStreamOptions with ensureFolderExists to createWriteStream(), following the same pattern as IFileSystemWriteBinaryFileOptions and other write APIs. Updated FileSystemBuildCacheProvider to use the new option instead of manual ensureFolderAsync(). For createReadStream, there aren't useful options from the existing FileSystem conventions (read methods use encoding/convertLineEndings which don't apply to raw streams). Commit: 91161d8

iclanton · 2026-04-05T06:29:57Z

@copilot - can we add stream support to the other cache plugins?

Copilot · 2026-04-05T06:48:14Z

@copilot - can we add stream support to the other cache plugins?

Added streaming support to both remaining cache plugins:

Amazon S3 plugin (733ef5c):

Refactored AmazonS3Client._makeRequestAsync to extract signing into _buildSignedRequest
Added _makeStreamRequestAsync using WebClient.fetchStreamAsync() with UNSIGNED-PAYLOAD for AWS Sig V4 signing of streaming uploads
Added getObjectStreamAsync() (with retry support) and uploadObjectStreamAsync() (single attempt, since consumed streams can't be replayed)
Added tryGetCacheEntryStreamByIdAsync and trySetCacheEntryStreamAsync to AmazonS3BuildCacheProvider

Azure Storage plugin (733ef5c):

Added tryGetCacheEntryStreamByIdAsync using blobClient.download() with readableStreamBody
Added trySetCacheEntryStreamAsync using blockBlobClient.uploadStream()
Added _logBlobError helper for consistent error handling across buffer and stream methods

All three cache plugins (HTTP, S3, Azure) now implement both streaming methods. All existing tests pass (45 S3, 10 Azure).

iclanton · 2026-04-05T19:45:15Z

@copilot - Can you put usage of this functionality behind a Rush experiment?

Copilot · 2026-04-05T20:00:23Z

@copilot - Can you put usage of this functionality behind a Rush experiment?

Added a useStreamingBuildCache experiment flag in experiments.json. The streaming code paths in OperationBuildCache now only activate when both the experiment is enabled AND the cloud provider implements the streaming methods. Without the experiment enabled, all cache operations use the existing buffer-based approach. Commit: 2493ed8

iclanton

@copilot - This change introduces a lot of duplicated code. I've called out some instances of it. Can you review with that in mind and clean that up?

iclanton · 2026-06-15T02:02:19Z

@copilot - this branch has conflicts. Rebase or merge main into this branch and resolve conflicts.

…urns false Providers may catch errors internally and return false instead of throwing, leaving a partially written file on disk. Move the cleanup logic outside the catch block so it runs for any non-success case, preventing corrupted partial files from being treated as valid cache entries on subsequent builds.

…entry-for-http-plugin # Conflicts: # common/reviews/api/rush-lib.api.md # libraries/rush-lib/src/api/ExperimentsConfiguration.ts # libraries/rush-lib/src/schemas/experiments.schema.json

Copilot · 2026-06-15T02:19:29Z

@copilot - this branch has conflicts. Rebase or merge main into this branch and resolve conflicts.

Merged main and resolved three conflicts (experiments.schema.json, ExperimentsConfiguration.ts, rush-lib.api.md) — all from the new strictChangefileValidation experiment added on main. Also updated the HTTP plugin test file to use jest.mocked() instead of bare mocked() to match main's updated convention. All tests pass (14 HTTP, 51 S3, 10 Azure).

iclanton · 2026-06-15T03:10:00Z

+        body: entryStream,
+        warningText: 'Could not write cache entry',
+        // Streaming uploads cannot be retried because the stream is consumed
+        maxAttempts: 1


@copilot - what can be done about this concern?

…am uploads

Copilot

Pull request overview

This PR adds optional file-based transfer APIs to the cloud build cache provider interface and wires them through Rush’s build cache flow, enabling streaming uploads/downloads to avoid buffering large cache entries in memory (gated behind the useDirectFileTransfersForBuildCache experiment).

Changes:

Added optional tryDownloadCacheEntryToFileAsync / tryUploadCacheEntryFromFileAsync to ICloudBuildCacheProvider and integrated conditional usage into OperationBuildCache.
Refactored WebClient to support streaming responses (fetchStreamAsync) with shared request core and Content-Encoding decompression for streams.
Implemented/updated providers and tests for HTTP, Amazon S3, and Azure Blob storage to support the new file-based path.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
rush-plugins/rush-http-build-cache-plugin/src/test/HttpBuildCacheProvider.test.ts	Expands unit coverage for buffer vs file-based HTTP cache transfers and retry/credential behaviors.
rush-plugins/rush-http-build-cache-plugin/src/HttpBuildCacheProvider.ts	Adds file-based download/upload APIs and introduces stream-based request path via `WebClient.fetchStreamAsync`.
rush-plugins/rush-bridge-cache-plugin/src/BridgeCachePlugin.ts	Plumbs the experiment flag into cache plugin options.
rush-plugins/rush-azure-storage-build-cache-plugin/src/AzureStorageBuildCacheProvider.ts	Adds Azure SDK-native file-based download/upload implementations and shared helpers.
rush-plugins/rush-amazon-s3-build-cache-plugin/src/test/AmazonS3Client.test.ts	Adds coverage for file-based GET/PUT behavior, retry expectations, and payload signing hash.
rush-plugins/rush-amazon-s3-build-cache-plugin/src/test/snapshots/AmazonS3Client.test.ts.snap	Snapshot update for added/modified tests.
rush-plugins/rush-amazon-s3-build-cache-plugin/src/AmazonS3Client.ts	Adds streaming file transfers and reinstates payload signing via SHA-256 hash for file uploads.
rush-plugins/rush-amazon-s3-build-cache-plugin/src/AmazonS3BuildCacheProvider.ts	Adds file-based provider APIs and centralizes object-name computation.
libraries/rush-lib/src/utilities/WebClient.ts	Introduces streaming response API and shared raw request core; adds stream decompression support.
libraries/rush-lib/src/schemas/experiments.schema.json	Adds schema entry for `useDirectFileTransfersForBuildCache`.
libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts	Threads experiment option through cache plugin initialization.
libraries/rush-lib/src/logic/buildCache/test/OperationBuildCache.test.ts	Updates test setup to include the new option.
libraries/rush-lib/src/logic/buildCache/OperationBuildCache.ts	Uses file-based download/upload when enabled and supported, with cleanup of partial downloads.
libraries/rush-lib/src/logic/buildCache/ICloudBuildCacheProvider.ts	Extends interface with optional file-based transfer methods.
libraries/rush-lib/src/logic/buildCache/FileSystemBuildCacheProvider.ts	Refactors local cache path generation and existence checks.
libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts	Plumbs experiments configuration into cache/skip plugin initialization.
libraries/rush-lib/src/api/ExperimentsConfiguration.ts	Adds typed config surface for the new experiment.
common/reviews/api/rush-lib.api.md	Updates API report for new interface and option surface.
common/reviews/api/rush-amazon-s3-build-cache-plugin.api.md	Updates API report for new S3 client methods.
common/changes/@microsoft/rush/copilot-stream-cache-entry-for-http-plugin_2026-04-05-03-56.json	Adds change entry describing the new APIs and gating.

+    const webFetchOptions: IGetFetchOptions | IFetchOptionsWithBody = {
+      verb,
+      headers
+    };
+    if (verb === 'PUT' && body) {
+      (webFetchOptions as IFetchOptionsWithBody).body = body;
+    }


+  /**
+   * If implemented, the build cache will prefer to use this method over
+   * {@link ICloudBuildCacheProvider.tryGetCacheEntryBufferByIdAsync} to avoid loading the entire
+   * cache entry into memory, if possible. The implementation should download the cache entry and write it
+   * to the specified local file path.
+   *
+   * @returns `true` if the cache entry was found and written to the file, `false` if it was
+   * not found. Throws on errors.
+   */


+    } else if (status === 400 || status === 401 || status === 403) {
+      cleanup?.();
+      throw new Error(`Amazon S3 responded with status code ${status} (${statusText})`);
+    } else {
+      cleanup?.();
+      return {
+        hasNetworkError: true,
+        error: new Error(`Amazon S3 responded with status code ${status} (${statusText})`)
+      };
+    }


dmichon-msft · 2026-06-22T18:50:23Z

+          // Clean up any partial file left by the failed download so it isn't
+          // mistaken for a valid cache entry on the next build.
+          try {
+            await FileSystem.deleteFileAsync(targetPath);


We should also adopt a model of download + atomic rename rather than writing directly to the file, since you can theoretically have multiple rush repos pointing at the same build cache asking for the same cache entry, and especially with modern agentic development, that's a scenario we should support.

dmichon-msft · 2026-06-22T18:51:26Z

      "type": "boolean"
+    },
+    "useDirectFileTransfersForBuildCache": {
+      "description": "If true, the build cache will use file-based APIs to transfer cache entries to and from cloud storage. This avoids loading the entire cache entry into memory, which can prevent out-of-memory errors for large build outputs. The cloud cache provider plugin must implement the optional file-based methods for this to take effect; otherwise it falls back to the buffer-based approach.",


Call out that this also allows a cache entry to exceed the limit of a single Buffer

@copilot - Update this.

Updated in e6d0c4f — the experiment description now also calls out that the direct-file path allows cache entries to exceed the limit of a single Buffer.

dmichon-msft · 2026-06-22T18:53:45Z

+  return await new Promise<string>((resolve, reject) => {
+    const hash: crypto.Hash = crypto.createHash(HASH_ALGORITHM);
+    const stream: FileSystemReadStream = FileSystem.createReadStream(filePath);
+    stream.on('data', (chunk: string | Buffer) => hash.update(chunk));
+    stream.on('end', () => resolve(hash.digest('hex')));
+    stream.on('error', reject);
+  });


More ergonomic to use the async iterator syntax:

Suggested change

return await new Promise<string>((resolve, reject) => {

const hash: crypto.Hash = crypto.createHash(HASH_ALGORITHM);

const stream: FileSystemReadStream = FileSystem.createReadStream(filePath);

stream.on('data', (chunk: string | Buffer) => hash.update(chunk));

stream.on('end', () => resolve(hash.digest('hex')));

stream.on('error', reject);

});

const hash: crypto.Hash = crypto.createHash(HASH_ALGORITHM);

const stream: FileSystemReadStream = FileSystem.createReadStream(filePath);

for await (const chunk of stream) {

hash.update(chunk);

}

return hash.digest('hex');

Also, do we need to clean up the stream or is it automatically taken care of on error?

Depending on how often we perform this operation, we will likely end up wanting to move this to a worker thread that uses a single preallocated Buffer and fs.readSync alternated with hash.update until exhaustion.

@copilot - take this suggestion and add a comment about moving this to a worker later.

Addressed in 9f098b5. _hashFileAsync() now uses async iteration over the file stream, and I added a note that we can move the hashing work to a worker thread later if this becomes a hotspot.

dmichon-msft · 2026-06-22T18:56:44Z

+      response.stream.resume();
+      throw new Error(`Amazon S3 responded with status code ${response.status} (${response.statusText})`);
+    }
+    response.stream.resume();


Just resume once before the if, seeing as neither branch cares about the response content.

@copilot - Take a look at this.

Addressed in 9f098b5. uploadObjectFromFileAsync() now resumes the response stream once before checking ok, since neither path needs to read the response body.

dmichon-msft · 2026-06-22T18:58:39Z

  EnvironmentConfiguration
 } from '@rushstack/rush-sdk';
-import { WebClient, type IWebClientResponse } from '@rushstack/rush-sdk/lib/utilities/WebClient';
+import {


Should this get moved to its own package?

We can consider doing that later.

dmichon-msft · 2026-06-22T19:01:34Z

    }

-    const bodyLength: number | 'unknown' = (body as { length: number })?.length || 'unknown';
+    const bodyLengthDesc: string = Buffer.isBuffer(body) ? `${body.length} bytes` : 'unknown length';


Do we need to implement the stream semantics ourselves with a file descriptor so that we can fstat and get the length to have it in the header? In general, transport integrity will be better if we have the MD5 hash and content-length.

@copilot - Take a look at this.

Updated in 9f098b5. Direct-file uploads now set Content-Length from the on-disk file size before streaming the request body. I did not add Content-MD5 here because this plugin targets generic HTTP endpoints and does not currently have a consistent integrity contract for that header.

Copilot AI assigned Copilot and iclanton Apr 5, 2026

Copilot created this pull request from a session on behalf of iclanton April 5, 2026 04:11 View session

github-project-automation Bot added this to Bug Triage Apr 5, 2026

github-project-automation Bot moved this to Needs triage in Bug Triage Apr 5, 2026

Copilot started work on behalf of iclanton April 5, 2026 04:16 View session

Copilot AI requested a review from iclanton April 5, 2026 04:32

Copilot finished work on behalf of iclanton April 5, 2026 04:32

Copilot started work on behalf of iclanton April 5, 2026 04:33 View session

Copilot finished work on behalf of iclanton April 5, 2026 04:35

Copilot started work on behalf of iclanton April 5, 2026 05:15 View session

Copilot finished work on behalf of iclanton April 5, 2026 05:27

Copilot started work on behalf of iclanton April 5, 2026 06:30 View session

Copilot AI changed the title ~~Add streaming APIs to build cache provider interface and HTTP plugin~~ Add streaming APIs to build cache provider interface and all cache plugins Apr 5, 2026

Copilot finished work on behalf of iclanton April 5, 2026 06:49

Copilot started work on behalf of iclanton April 5, 2026 19:45 View session

Copilot finished work on behalf of iclanton April 5, 2026 20:01

iclanton reviewed Apr 5, 2026

View reviewed changes

Comment thread ...rushstack/node-core-library/copilot-stream-cache-entry-for-http-plugin_2026-04-05-04-27.json Outdated

iclanton reviewed Apr 5, 2026

View reviewed changes

Comment thread common/changes/@microsoft/rush/copilot-stream-cache-entry-for-http-plugin_2026-04-05-03-56.json Outdated

iclanton reviewed Apr 5, 2026

View reviewed changes

Copilot started work on behalf of iclanton June 15, 2026 02:02 View session

Copilot finished work on behalf of iclanton June 15, 2026 02:11

Copilot started work on behalf of iclanton June 15, 2026 02:12 View session

Copilot AI added 2 commits June 15, 2026 02:14

Merge remote-tracking branch 'origin/main' into copilot/stream-cache-…

d7ea288

…entry-for-http-plugin # Conflicts: # common/reviews/api/rush-lib.api.md # libraries/rush-lib/src/api/ExperimentsConfiguration.ts # libraries/rush-lib/src/schemas/experiments.schema.json

fix: merge main and update test to use jest.mocked()

407c7c9

Copilot finished work on behalf of iclanton June 15, 2026 02:19

iclanton reviewed Jun 15, 2026

View reviewed changes

Copilot started work on behalf of iclanton June 15, 2026 03:11 View session

Restructure S3 response handling and pre-resolve credentials for stre…

4f40de1

…am uploads

Copilot finished work on behalf of iclanton June 15, 2026 03:22

Copilot AI requested a review from iclanton June 15, 2026 03:22

iclanton reviewed Jun 15, 2026

View reviewed changes

Comment thread rush-plugins/rush-amazon-s3-build-cache-plugin/src/AmazonS3Client.ts Outdated

Copilot started work on behalf of iclanton June 15, 2026 14:04 View session

Destructure response variable in _handleGetResponseAsync

585f972

Copilot finished work on behalf of iclanton June 15, 2026 14:11

Copilot AI requested a review from iclanton June 15, 2026 14:11

iclanton requested a review from Copilot June 19, 2026 23:58

Copilot started reviewing on behalf of iclanton June 19, 2026 23:58 View session

Copilot AI reviewed Jun 20, 2026

View reviewed changes

dmichon-msft reviewed Jun 22, 2026

View reviewed changes

Copilot started work on behalf of iclanton June 26, 2026 01:06 View session

docs: clarify build cache direct file transfer experiment

e6d0c4f

Copilot finished work on behalf of iclanton June 26, 2026 01:14

Copilot started work on behalf of iclanton June 26, 2026 01:14 View session

Copilot AI added 2 commits June 26, 2026 01:22

Merge origin/main into copilot/stream-cache-entry-for-http-plugin

d416787

Refine streaming cache upload handling

9f098b5

Copilot finished work on behalf of iclanton June 26, 2026 01:29

Uh oh!

Conversation

Copilot AI commented Apr 5, 2026 • edited by iclanton Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Details

How it was tested

Impacted documentation

Uh oh!

iclanton commented Apr 5, 2026

Uh oh!

Copilot AI commented Apr 5, 2026

Uh oh!

iclanton commented Apr 5, 2026

Uh oh!

Copilot AI commented Apr 5, 2026

Uh oh!

iclanton commented Apr 5, 2026

Uh oh!

Copilot AI commented Apr 5, 2026

Uh oh!

iclanton commented Apr 5, 2026

Uh oh!

Copilot AI commented Apr 5, 2026

Uh oh!

iclanton commented Apr 5, 2026

Uh oh!

Copilot AI commented Apr 5, 2026

Uh oh!

Uh oh!

Uh oh!

iclanton left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

iclanton commented Jun 15, 2026

Uh oh!

Copilot AI commented Jun 15, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Copilot AI commented Apr 5, 2026 •

edited by iclanton

Loading