fix data race in span lifecycle#247
Merged
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This PR hardens the core tracing implementation against span/segment lifecycle concurrency misuse by introducing per-span synchronization and an explicit “freeze” point for safe lock-free reporting, plus a synchronized correlation context to prevent fatal concurrent map access during propagation. It also adds targeted recovery in long-lived goroutines (metrics/profile collection and reporter transforms) and introduces comprehensive race/e2e regression tests and a benchmark for the new hot path.
Changes:
- Add per-span locking + freeze semantics to make concurrent mutation/end safe and to publish immutable span data to the collector/reporter.
- Replace segment correlation storage with a synchronized
CorrelationContextand propagate via snapshots to avoid concurrent map iteration/writes. - Add recover guards and new stress/race/e2e tests + a benchmark to prevent regressions.
Reviewed changes
Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| plugins/core/tracing.go | Snapshot correlation on exit-span injection; clone runtime on ContinueContext to avoid shared-map races. |
| plugins/core/test_base.go | Make test reporter storage thread-safe; initialize correlation defaults in test tracer. |
| plugins/core/span_tracing.go | Update segment collector lifecycle and span end/async-finish paths for freeze + safe collection. |
| plugins/core/span_default.go | Introduce opLock + freeze (ended) to serialize span mutations and drop late writes safely. |
| plugins/core/correlation.go | Add synchronized correlation store with snapshot/clone support. |
| plugins/core/context.go | Optimize runtime context cloning and make runtime map lazy-allocated. |
| plugins/core/metrics.go | Add per-iteration recover to prevent user meter panics from killing the process. |
| plugins/core/profile.go | Add per-chunk recover and tighten locking around profile task state updates. |
| plugins/core/propagating.go | Clarify that wire-format SpanContext.CorrelationContext remains a plain map for encode/decode. |
| plugins/core/reporter/grpc/grpc.go | Register recover before transform + send to avoid collector-goroutine crashes. |
| plugins/core/reporter/kafka/kafka.go | Register recover before transform + send to avoid collector-goroutine crashes. |
| plugins/core/reporter/grpc/send_tracing_recover_test.go | New test ensuring SendTracing recover wraps transform panics. |
| plugins/core/reporter/kafka/send_tracing_recover_test.go | New test ensuring SendTracing recover wraps transform panics. |
| plugins/core/span_freeze_test.go | New unit tests for freeze semantics, correlation snapshots, async behavior, and runtime cloning. |
| plugins/core/segment_datarace_test.go | New -race regression tests covering concurrent mutators, freeze vs reporting, double-End, async finish, etc. |
| plugins/core/span_hostile_workload_test.go | New hostile e2e workload exercising real pipeline under concurrency abuse patterns. |
| plugins/core/span_crash_e2e_test.go | Subprocess e2e test asserting no panic/runtime fatal under hostile workload with runtime hardening. |
| plugins/core/span_sync_bench_test.go | New benchmark to track performance impact of span lifecycle locking. |
| CHANGES.md | Release note entries for the race fixes and added recover protections. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
wu-sheng
approved these changes
Jun 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.