util/compression: enable zstd.WithConcurrentBlocks#6936
Draft
thaJeztah wants to merge 2 commits into
Draft
Conversation
…vels The toZstdEncoderLevel utility was introduced in cab33b1, and has a custom mapping of zstd compressions levels to their go (klauspost/compress) counterparts. Swap our local implementation for the one in klauspost/compress so that new levels implemented in klauspost/compress will also be mapped if they arrive. This does change some mappings; | Input level | Before | After | |-------------|------------------------|--------------------------| | `< 0` | `SpeedDefault` | `SpeedFastest` | | `6` | `SpeedDefault` | `SpeedBetterCompression` | | `9` | `SpeedBestCompression` | `SpeedBetterCompression` | While updating, also slightly refactor; - use an options slice to skip setting the option if no config was set, and to allow adding additional options. - inline zstdWriter where used, as it was only used in a single place. Signed-off-by: Sebastiaan van Stijn <[email protected]>
thaJeztah
changed the title
This option was added in klauspost/compress v0.19; > Parallel Stream Compression > > For maximum throughput on large streams, use WithConcurrentBlocks(true) > together with WithEncoderConcurrency(n) where n is the number of CPU cores > you want to use. This splits the input into large sections (jobs) that are > compressed simultaneously by multiple goroutines, similar to how the C zstd > library does multithreaded compression. Given that BuildKit creates a new Writer for each blob, we may need to look what the best value is for WithEncoderConcurrency. Signed-off-by: Sebastiaan van Stijn <[email protected]>
thaJeztah
force-pushed
the
compress_WithConcurrentBlocks
branch
from
eddc195 to
b161b5c
Compare
thaJeztah
changed the title
thaJeztah
commented
Jul 7, 2026
| return func(dest io.Writer, _ string) (io.WriteCloser, error) { | ||
| return zstdWriter(comp)(dest) | ||
| opts := []zstd.EOption{ | ||
| zstd.WithEncoderConcurrency(runtime.GOMAXPROCS(0)), // TBD: default is GOMAXPROCS for each writer. |
Member
Author
There was a problem hiding this comment.
Not sure what the best value is to use here; effectively this is explicitly setting the default.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
util/compression: enable WithConcurrentBlocks
This option was added in klauspost/compress v0.19; https://pkg.go.dev/github.com/klauspost/[email protected]/zstd#section-readme
Given that BuildKit creates a new Writer for each blob, we may need
to look what the best value is for WithEncoderConcurrency.
Benchmarks as mentioned in upstream;