fix(grpc+consumer): preserve context logger enrichment end-to-end#130
Open
ruangervasi-altpay wants to merge 1 commit into
Open
fix(grpc+consumer): preserve context logger enrichment end-to-end#130ruangervasi-altpay wants to merge 1 commit into
ruangervasi-altpay wants to merge 1 commit into
Conversation
Two fixes for trace_id propagation into logs: 1. gRPC server: the base logger interceptor ran AFTER supplied interceptors, replacing the context logger and silently dropping any enrichment (trace_id, audit fields) before the handler executed. Reordered to metadata -> base logger -> supplied interceptors -> handler, so supplied interceptors (e.g. backend-commons logs.UnaryServerInterceptor) enrich a logger that actually reaches the handler. Regression test asserts the handler's log line carries the enriched trace_id and grpc_method through the full chain. 2. RabbitMQ consumer: the lifecycle logger (built before the handler runs) now includes the OTEL trace_id extracted from the message headers. This covers decode errors, ack/nack failures, retries and the application handler itself -- not just handler logs enriched by downstream middleware.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Found during the trace_id rollout at Alternative Payments (review by @CassioRoos on getalternative/svc-notification#219).
Problem 1 — gRPC interceptor ordering drops enrichment
NewServerappendedgrpcLoggerInterceptor(params.Logger)AFTER all supplied interceptors. Supplied interceptors that enrich the context logger (audit fields, trace_id via backend-commonslogs.UnaryServerInterceptor()) had their enrichment silently replaced by the base logger before the handler ran. Reproduced with a real traced request throughNewThunderGrpcServer: client had a valid trace ID, handler log had neithertrace_idnorgrpc_method.Fix: attach the base logger BEFORE supplied interceptors:
Regression test:
pkg/grpc/server_test.gochains the interceptors exactly likeNewServerand asserts the handler's actual log line carries the enrichedtrace_id+grpc_method(not a helper-in-isolation test).Problem 2 — consumer lifecycle logs missing trace_id
The RabbitMQ consumer builds its lifecycle logger before invoking the handler, but never added the OTEL trace_id (already extracted from message headers via
ExtractTrace). Unknown-topic, decode, ack/nack, retry, and dead-letter logs had no trace_id — only handler logs enriched by downstream middleware did.Fix: enrich the lifecycle logger with
trace_idfrom the span context at the consumer boundary, so the ENTIRE message lifecycle inherits it.Test plan
go build ./...cleango test ./...— all packages pass (5 ok)