Parallelize checksum validation#14
Open
hangduykhiem wants to merge 2 commits into
Open
Conversation
Chunk verification for a single file was a fully serial for-loop, so validating a large depot file (thousands of chunks) pegged one CPU core for a long time while other cores sat idle, causing significant thermal load on Android devices during file verification. Fans out chunk reads/Adler32 checks across coroutines on Dispatchers.IO, bounded by a Semaphore sized to available processors (capped at 16), so verification throughput scales with device core count instead of being limited to a single thread.
Relocates MAX_VALIDATE_CONCURRENCY and defaultValidateConcurrency() next to the other object-level declarations at the top of Util, matching this repo's convention of declaring constants near the top of the class instead of inline between functions. Adds UtilValidateSteam3FileChecksumsTest covering: all-chunks-match, mismatched-chunk detection, result ordering, correctness across different concurrency values, and a larger chunk-count smoke test. No prior test coverage existed for validateSteam3FileChecksums.
Author
|
Logs of successful samples here: logs.txt Notice that Hopefully I can download borderlands 2 much faster if this is approved and merged. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
On GameNative, I get stuck at checksum validation for Borderlands 2 for an hour.
Up on checking, the
validateSteam3FileChecksumsis doing it 1 thread at at time. So I decided to send a patch.This PR add
Default.IOto parallelize checksum validation. It has a semaphore to keep it never spawning more thanavailableProcessorsorMAX_VALIDATE_CONCURRENCYWhy do we want
availableProcessorseven if this is running on IO dispatcher with default 64 thread pool instead of Default? We don't want to use Default because we can exhaust the computation during download which isn't nice, so we use I/O. At the same time, you can't usefully run more concurrent Adler32 computations than you have cores. => I actually wonder if this should be documented. Let me know and I'll send another patch.Checklist