fix(metrics): update FSC to adopt provider-side collector caching#1907
Conversation
1e5db57 to
2b66f80
Compare
|
Hi @EvanYan1024 , thanks for submitting this. What about a more radical solution that modifies the metrics.Provider implementation to cache the metrics and return the ones that exist already? I fear that we might have similar issues elsewhere. What do you think? |
|
Agreed, this direction is better — pushing the deduplication down into the provider as a single choke point fixes the whole class of issues in one go, not just these finality collectors. |
a244c3e to
2608bce
Compare
2608bce to
e27bc75
Compare
|
I think @mbrandenburger can be interested in this one. If we inject the caching mechanism directly in the smart client, we are protected also there. |
|
@adecaro Moved the fix into the smart client's provider as suggested: hyperledger-labs/fabric-smart-client#1582 (issue hyperledger-labs/fabric-smart-client#1581). The provider now caches collectors by fully-qualified name, so repeated requests share the collector registered first and skip the registry entirely. Once that lands, this PR can be closed or reduced to a dependency bump. |
|
Hi @EvanYan1024 , so hyperledger-labs/fabric-smart-client#1582 has been merged. As soon as FSC cuts a new tag, we can update also panurus and we should be good. What do you think? |
|
Sounds good @adecaro! Once FSC cuts the tag I'll update this PR: drop the decorator caching, bump FSC, and adapt the |
83d7151 to
4daa56c
Compare
|
@adecaro I've gone ahead and reworked this PR: it now bumps FSC to the hyperledger-labs/fabric-smart-client#1582 merge commit ( |
hyperledger-labs/fabric-smart-client#1582 caches collectors inside the prometheus provider, fixing duplicate registration at the source. Bump FSC across modules, drop the tmsProvider recover workaround, and adapt to the NewMetricsProvider signature change. Signed-off-by: Evan <[email protected]>
4daa56c to
8cdfa21
Compare
|
Thanks @EvanYan1024. We have a much robust solution now. Great effort 🙏 |
Motivation
Metric collectors are requested per finality listener and per TMS, and the FSC prometheus provider used to swallow duplicate registrations (
SkipRegisterErr: true): later requesters silently received an unregistered collector, so their observations never showed up on/metrics, and every request paid a registry round-trip under the registry lock. We hit both in our test environments under sustained load.Per the review discussion here, the deduplication was moved down into the smart client: hyperledger-labs/fabric-smart-client#1582 (merged) caches collectors inside the prometheus provider, so every consumer is protected. This PR adopts it and removes the token-sdk-side workaround.
Changes
fabric-smart-clienttov0.14.3-0.20260716084917-c10decb043ec(the fix(metrics): cache collectors in the prometheus provider hyperledger-labs/fabric-smart-client#1582 merge commit) across all modules, with the transitive updates that come with it (fabric-lib-go, fabric-x-common, grpc). Happy to swap this to the next FSC tag once it is cut.recoverFromDuplicateworkaround intmsProvider: the provider no longer panics on duplicate registration, so the recover path is dead code — and it had a latent bug (recovering swallowed the panic and returned a nil collector to the caller).operations.NewMetricsProvidersignature change (theskipRegisterErrparameter was removed in fix(metrics): cache collectors in the prometheus provider hyperledger-labs/fabric-smart-client#1582).Tests
token/core/common/metricskeeps the TMS label-decoration tests; the recover-path tests are removed with the workaround.go test -race ./token/...passes against the bumped FSC.Fixes #1906