Skip to content

fix(compactor): Fix flaky TestBlocksCleaner_ShouldRemoveBlocksOutside…#7486

Open
yeya24 wants to merge 1 commit intocortexproject:masterfrom
yeya24:fix-blocks-cleaner-flaky-test
Open

fix(compactor): Fix flaky TestBlocksCleaner_ShouldRemoveBlocksOutside…#7486
yeya24 wants to merge 1 commit intocortexproject:masterfrom
yeya24:fix-blocks-cleaner-flaky-test

Conversation

@yeya24
Copy link
Copy Markdown
Contributor

@yeya24 yeya24 commented May 6, 2026

…RetentionPeriod

Switch from filesystem bucket to in-memory bucket in this test. The filesystem bucket's Upload is non-atomic (os.Create truncates, then io.Copy writes), creating a race window where the HeartBeat goroutine's write can be observed as an empty file by a concurrent read. The in-memory bucket's Upload is atomic (mutex-protected), eliminating the race.

What this PR does:

Fix flaky test by switching to use an inmemory bucket to eliminate possible race condition when the visit marker is being uploaded.

--- FAIL: TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod (0.06s)
    blocks_cleaner_test.go:822: 
        	Error Trace:	/__w/cortex/cortex/pkg/compactor/blocks_cleaner_test.go:822
        	Error:      	Received unexpected error:
        	            	failed to read cleaner visit marker: visit marker file: markers/cleaner-visit-marker.json, content: , error: unexpected end of JSON input: unmarshal visit marker JSON
        	Test:       	TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod

Which issue(s) this PR fixes:
Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
  • docs/configuration/v1-guarantees.md updated if this PR introduces experimental flags

…RetentionPeriod

Switch from filesystem bucket to in-memory bucket in this test. The
filesystem bucket's Upload is non-atomic (os.Create truncates, then
io.Copy writes), creating a race window where the HeartBeat goroutine's
write can be observed as an empty file by a concurrent read. The
in-memory bucket's Upload is atomic (mutex-protected), eliminating the
race.

Signed-off-by: Ben Ye <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant