osdtrace: recover by-value param location from the SysV-AMD64 ABI; add Resolute amd64v3 DWARF JSONs#154
Merged
Merged
Conversation
When a traced parameter has no DW_AT_location the parser bailed
(translate_param_location returns false, and the caller asserts). GCC drops
the location of a forwarded by-value aggregate under -march=x86-64-v3 -- e.g.
reqid in ReplicatedBackend::submit_transaction on Ubuntu amd64v3 and CentOS
Stream el10 (RHEL 10's x86-64-v3 baseline) -- which aborted osdtrace there.
The value is still at its ABI incoming-argument slot, so reconstruct the
location from the SysV-AMD64 calling convention: classify the formal
parameters (INTEGER vs stack MEMORY), assign INTEGER eightbytes to
RDI/RSI/RDX/RCX/R8/R9 and the rest (plus MEMORY-class aggregates) to the
stack, and resolve the target's slot via the call-frame CFA (the same
machinery DW_OP_fbreg uses). Parameter classes we cannot classify with
confidence make the fallback bail, so we never emit a guessed-wrong location.
Validated on ceph-r2 (amd64v3, Ceph 20.2.0): the recovered reqid location is
identical to the amd64 build's ({reg RSP, offset 0x30, stack}), and a live
read at the uprobe yields a valid client reqid.
Ubuntu 26.04 ships an opt-in amd64v3 package variant
(APT::Architecture-Variants "amd64v3") compiled for -march=x86-64-v3. It
keeps the same version string and dpkg architecture as the baseline amd64
build but has different function addresses and a different ELF build-id, so
it needs its own DWARF JSON.
Add osd + rados DWARF JSON for Ceph 20.2.0-0ubuntu2 amd64v3, generated on an
Ubuntu 26.04 amd64v3 host. These depend on the ABI incoming-argument
location fallback in the preceding commit: without it, reqid in
ReplicatedBackend::submit_transaction (whose location the v3 compiler drops)
would abort generation. reqid is recovered to the same slot the amd64 build
emits natively ({reg RSP, offset 0x30, stack}).
Build-ids: ceph-osd 20bbaa35..., libceph-common.so.2 23210fa6... -- distinct
from the amd64 files (0e6c06a0..., 0f51328e...), so embedded-DWARF build-id
matching selects the correct variant automatically.
Add ubuntu-26.04 to the build-ubuntu and build-ubuntu-cephadm runner matrices so the build, DWARF-compare, MicroCeph, embedded-DWARF and cephadm functional tests run on Resolute. Ubuntu 26.04 cloud images default to the amd64v3 package variant, which shares its version string and dpkg architecture with the baseline amd64 build but has different function addresses and a different ELF build-id. dwarf-compare.sh keyed the reference DWARF JSON by package version, which is ambiguous between the two variants. Key by the ELF build-id embedded in the freshly generated JSON instead: it selects the correct reference (amd64 vs amd64v3) automatically, whichever variant the runner happens to install, and the script now prints exactly which reference file was matched. If no reference build-id matches, it fails with a clear "missing reference" error. All existing ubuntu reference files already carry per-module build-ids, so build-id keying works on the 22.04 and 24.04 runners unchanged.
…broken install_cephadm accepted the apt-installed cephadm whenever `apt-get install` returned 0, without checking it runs. On Ubuntu 26.04 (Resolute) the cephadm package is broken: it imports python modules (`ceph`, `yaml`) that aren't pulled in as dependencies, so every invocation — including `bootstrap` — dies with `ModuleNotFoundError`. The build-ubuntu-cephadm job on 26.04 therefore failed at "bootstrap failed to produce FSID" for all releases. Verify the distro cephadm actually executes (`cephadm --help`, which forces the module imports but needs no root/podman/network) before committing to it; if it doesn't run, fall through to the existing container-extraction path. The in-image /usr/sbin/cephadm is a self-contained python zipapp that bundles its own deps, so it runs regardless of host packaging. Also validate the container-extracted cephadm and fail clearly if it too cannot run. 22.04/24.04 keep using the apt cephadm (its --help succeeds); only the broken Resolute case is redirected to the container binary.
cephadm bootstrap creates /var/run/ceph with `install -d -o 167 -g 167` (167 is the ceph uid/gid baked into the ceph container images). Ubuntu 26.04 replaced GNU coreutils with uutils, whose `install` rejects a numeric owner that has no /etc/passwd entry — GNU install accepted it — so bootstrap failed on the bare 26.04 runner with `install: invalid user: '167'`. Create a matching ceph user/group (uid/gid 167) before bootstrap when they don't already resolve, falling back to the name "ceph167" if "ceph" is taken by a differently-numbered user. No-op on 22.04/24.04. Verified on a bare ubuntu:26.04 container: the previously failing `install -d -o 167 -g 167 /var/run/ceph/...` now succeeds.
…allers After bootstrap the test invokes bare `cephadm` (shell / orch status / ceph -s / osd apply / pool ops) in ~15 places, assuming a working cephadm on PATH. On 26.04 only /tmp/cephadm (the container-extracted binary) works; the on-PATH distro cephadm is broken, so `_orch_backend_ready`'s `cephadm shell -- ceph orch status` failed for every bootstrap attempt even though bootstrap itself completed (rc=0) — the test then rejected a healthy cluster and failed after 3 attempts. When install_cephadm falls back to the container binary, also install it on PATH (overwrite the broken on-PATH copy, else drop it in /usr/local/bin) so the bare `cephadm` calls use the working binary. 22.04/24.04 are unaffected (their distro cephadm works and this path isn't taken). Verified on a bare ubuntu:26.04 container: bare `cephadm --help` goes from broken to working after the replacement.
There was a problem hiding this comment.
Pull request overview
This PR improves Cephtrace’s DWARF-based tracing robustness on x86-64-v3 (“amd64v3”) builds by adding an ABI-based fallback to recover missing parameter locations, and by checking in/build-id-matching new reference DWARF JSONs for Ubuntu 26.04’s amd64v3 package variant. It also updates the cephadm functional test setup for Ubuntu 26.04 packaging/tooling regressions and expands CI coverage to include ubuntu-26.04.
Changes:
- Add a SysV AMD64 ABI fallback in the DWARF parser to recover missing
DW_AT_locationfor certain by-value aggregate parameters. - Switch Ubuntu DWARF reference selection in tests to build-id matching and add Ubuntu 26.04 amd64v3 reference JSONs.
- Harden cephadm test setup for Ubuntu 26.04 (broken cephadm package; uutils
installnumeric-owner behavior) and add ubuntu-26.04 to CI matrices.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
tests/lib/cephadm-setup.sh |
Validates distro cephadm actually runs; falls back to container zipapp and ensures uid/gid 167 exists for bootstrap on Ubuntu 26.04. |
tests/dwarf-compare.sh |
Selects reference DWARF JSON by ELF build-id to disambiguate amd64 vs amd64v3 builds with identical version strings. |
src/dwarf_parser.h |
Declares new ABI-fallback helpers for missing parameter locations. |
src/dwarf_parser.cc |
Implements SysV-AMD64 incoming-argument classification + CFA-relative stack-slot recovery when DW_AT_location is absent. |
files/ubuntu/radostrace/20.2.0-0ubuntu2_amd64v3_dwarf.json |
Adds Ubuntu 26.04 amd64v3 radostrace reference DWARF JSON (distinct build-id/addresses). |
files/ubuntu/osdtrace/osd-20.2.0-0ubuntu2_amd64v3_dwarf.json |
Adds Ubuntu 26.04 amd64v3 osdtrace reference DWARF JSON (distinct build-id/addresses). |
doc/dwarf-json-files.md |
Documents the new Ubuntu 26.04 Tentacle amd64v3 variant and suffix naming/build-id selection behavior. |
.github/workflows/pr-build.yaml |
Adds ubuntu-26.04 to build + cephadm functional-test matrices. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // ReplicatedBackend::submit_transaction on Ubuntu amd64v3 / CentOS el10): | ||
| // the value is still at its ABI incoming-argument slot, so recover it from | ||
| // the SysV-AMD64 calling convention instead of giving up. | ||
| if (abi_param_location(func, vardie, pc, varloc)) { |
Owner
Author
There was a problem hiding this comment.
Addressed in 23c224f: abi_param_location() is now compiled out on non-x86-64 (#if !defined(__x86_64__) → return false), so other architectures take the normal missing-location failure path instead of applying SysV-AMD64 register/stack rules. A compile-time gate is sufficient because the tools always run natively on the host they trace, so build arch == target arch. Verified x86-64 behavior is unchanged by regenerating the amd64v3 DWARF JSON on Ubuntu 26.04 and deep-comparing against the checked-in reference (identical; reqid still recovered).
Comment on lines
+196
to
+198
| # `install: invalid user: '167'`. Create a matching ceph user/group when uid | ||
| # 167 does not already resolve. No-op on 22.04/24.04 and anywhere ceph is | ||
| # already installed. |
Comment on lines
+207
to
+208
| getent group 167 >/dev/null 2>&1 || groupadd -g 167 "$g" 2>/dev/null || true | ||
| useradd -u 167 -g 167 -M -r -s /usr/sbin/nologin "$u" 2>/dev/null || true |
Comment on lines
+121
to
+123
| d = json.load(open(path)) | ||
| except (OSError, ValueError): | ||
| continue |
abi_param_location encodes the SysV-AMD64 calling convention -- INTEGER argument register numbers (RDI=5, RSI=4, ...) and MEMORY-class stack layout -- but ran whenever DW_AT_location was missing, on any architecture. On an arm64 build the same DWARF register numbers name different registers (regnum 5 is x5, arguments live in x0-x7), so the fallback could emit a wrong-but-plausible location instead of failing. osdtrace/radostrace always run natively on the host they trace, so the target architecture equals the build architecture: compile the recovery out on non-x86-64 (return false), letting the caller take the normal missing-location failure path. x86-64 behavior is unchanged -- verified by regenerating the amd64v3 DWARF JSON on Ubuntu 26.04 and deep-comparing against the checked-in reference (identical, reqid still recovered). Addresses the Copilot review comment on PR #154.
xtrusia
reviewed
Jul 2, 2026
The plain ubuntu-26.04 runner exercises the amd64v3 ceph build only: GitHub's 26.04 cloud images ship with APT::Architecture-Variants "amd64v3" enabled, so dwarf-compare installs the v3 packages and validates the _amd64v3 reference JSONs. Nothing covered the Resolute baseline-amd64 references. Add a (ubuntu-26.04, amd64) matrix cell that drops an apt.conf.d snippet with `#clear APT::Architecture-Variants;` before installing anything, so apt resolves the baseline amd64 ceph + dbgsyms and dwarf-compare's build-id keying selects the plain osd-20.2.0-0ubuntu2_dwarf.json references. The #clear mechanism was verified on a variant-enabled 26.04 host: the ceph-common candidate flips from resolute/main amd64v3 to amd64. The MicroCeph/embedded/qemu functional tests deploy a snap-bundled ceph that is independent of the host apt variant, so the new cell skips them and runs only build + dwarf-compare; the default cells are unchanged (they gain a ", default" suffix in the job name from the new matrix dimension).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On
-march=x86-64-v3builds of Ceph, GCC drops theDW_AT_locationof a forwarded by-value aggregate parameter. Concretely,reqid(anosd_reqid_t, passed by value) inReplicatedBackend::submit_transactionhas no location in the DWARF. This affects:APT::Architecture-Variants "amd64v3")The parameter DIE keeps its name and type but carries no
DW_AT_locationof any form (verified withllvm-dwarfdump/gdb). On these builds the parser previously either emitted a garbage location or, after the recent return-value plumbing, hitassert(ok)and aborted osdtrace generation on that function.The plain
amd64/el9builds are unaffected (therereqidisDW_OP_fbreg +40).Fix
When a traced parameter has no
DW_AT_location, reconstruct its incoming-argument slot from the System V AMD64 calling convention instead of giving up:RDI/RSI/RDX/RCX/R8/R9, overflow + MEMORY-class aggregates to the stack,DW_OP_fbregalready uses (so it comes out as{reg RSP, offset, stack}at the entry probe).Parameter classes that can't be classified with confidence (SSE/float, small split aggregates) make the fallback bail, so a wrong location is never emitted — it degrades to the existing graceful failure.
The value is genuinely there: the function reads it (
vmovdqu 0x28(%rax),rax=CFA) and forwards it toissue_op; only its debug location was dropped.Also: Resolute amd64v3 DWARF JSONs
Adds
osd/radosDWARF JSON for Ceph20.2.0-0ubuntu2amd64v3 (Ubuntu 26.04 / Tentacle). The amd64v3 variant keeps the same version string anddpkgarch asamd64but has different function addresses and a different ELF build-id, so it needs its own JSON. Build-id matching selects the right variant automatically; the_amd64v3filename suffix disambiguates the checked-in files and the-iimport path.Validation
reqidlocation is identical to the amd64 build's ({reg RSP, offset 0x30, stack}); a live read at the uprobe yields a validclientreqid.ceph-osd;reqidrecovered to the same slot. Confirms the fix is toolchain-independent (Ubuntu GCC 16 and RHEL GCC).osdtraceandradostracebuild clean (shareddwarf_parser.cc).func2pcdiffers from the committed amd64 files on every address (distinct build), andreqid's recovered location matches the amd64 baseline.