Commit b0c979a
Add diagnostic logging to shard setup and repo facet initialization
Summary:
TW task tsp_global/mononoke/mononoke_git_server/59 had 9 shards stuck in Setup
state for 1+ hour with no errors logged. Investigation revealed that
repo_factory.build() uses futures::try_join! to build ~30 facets concurrently —
all must complete for build() to return. SQL-backed facets have 10s query timeouts
but non-SQL facets (blobstore fetches, Scribe subscriptions) have no timeout. A
single stuck facet blocks the entire build() and keeps the shard in InProgress
state indefinitely with no diagnostic output.
This diff adds start/finish logging to pinpoint which facet is stuck:
- Setup task: logs completion time and success/failure
- WBC: logs after create_subscription and after coordinator spawn
- Preloaded commit graph: adds blobstore_key to Started/Finished logs for repo identification
- Suspect facets in repo_factory: start/finish logging for bonsai_tag_mapping,
git_ref_content_mapping, commit_graph, cgdm_changeset_divider, hook_manager
Reviewed By: RajivTS
Differential Revision: D102354775
fbshipit-source-id: f075dee72d31aa989910abcd2be69bda56b363fd1 parent e0cd737 commit b0c979a
4 files changed
Lines changed: 260 additions & 159 deletions
File tree
- eden/mononoke
- cmdlib/sharding/src
- repo_attributes
- bookmarks/warm_bookmarks_cache/src
- commit_graph/preloaded_commit_graph_storage/src
- repo_factory/src
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
133 | 133 | | |
134 | 134 | | |
135 | 135 | | |
136 | | - | |
137 | 136 | | |
138 | | - | |
139 | | - | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
140 | 154 | | |
141 | 155 | | |
142 | 156 | | |
| |||
Lines changed: 9 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
648 | 648 | | |
649 | 649 | | |
650 | 650 | | |
| 651 | + | |
| 652 | + | |
| 653 | + | |
| 654 | + | |
651 | 655 | | |
652 | 656 | | |
653 | 657 | | |
| |||
678 | 682 | | |
679 | 683 | | |
680 | 684 | | |
| 685 | + | |
| 686 | + | |
| 687 | + | |
| 688 | + | |
| 689 | + | |
681 | 690 | | |
682 | 691 | | |
683 | 692 | | |
| |||
Lines changed: 2 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
310 | 310 | | |
311 | 311 | | |
312 | 312 | | |
313 | | - | |
| 313 | + | |
314 | 314 | | |
315 | 315 | | |
316 | 316 | | |
| |||
319 | 319 | | |
320 | 320 | | |
321 | 321 | | |
| 322 | + | |
322 | 323 | | |
323 | 324 | | |
324 | 325 | | |
| |||
0 commit comments