pdd sync subprocess runner may use installed pdd instead of checkout during self-sync

[Serhan-Asad]

Problem

During local investigation of #1641 / PR #1644, one pdd sync agentic_bug_orchestrator failure block showed that the subprocess path was using the installed package instead of the checkout:

pdd.__file__: /opt/anaconda3/lib/python3.12/site-packages/pdd/__init__.py
pdd --version: pdd, version 0.0.276
git SHA: cf1a456a2e3e0bf45a12bbd3ae534393da4078d8
git status: dirty
source: site-packages

From the same checkout, python -m pdd --version reported 0.0.278.dev0.

This may make self-sync/debugging misleading: the user thinks they are testing the current branch, but subprocess sync can execute whatever pdd binary appears first on PATH.

Suspected area

pdd/agentic_sync_runner.py has _find_pdd_executable() that currently prefers shutil.which("pdd"), then builds subprocess commands from that executable.

That behavior is reasonable for normal installed CLI usage, but it may be risky inside a source checkout, especially when syncing PDD itself or when the parent process was invoked as python -m pdd.

Possible direction, not a final design

Consider making subprocess sync prefer the same codebase/interpreter as the parent process when running from a checkout, for example:

• use [sys.executable, "-m", "pdd"] when imported pdd.__file__ is under the current repo/worktree
• only use shutil.which("pdd") when the parent process is also an installed CLI path
• add a warning when pdd on PATH and imported pdd.__file__ disagree
• include the resolved command in sync logs/core dumps

The sync owner should decide the exact policy because installed CLI behavior may still be desired in some automation contexts.

Acceptance criteria ideas

• A test covers a repo-checkout invocation where PATH contains an older pdd, and sync subprocess command still targets the checkout/interpreter.
• A diagnostic warning or core-dump field makes version/source mismatch obvious.
• Existing installed-package usage is not broken.

Related

This is separate from the compatibility-surface question in #1647. It does not explain every sync failure, but it can make investigation results unreliable.

[prompt-driven-github]

🚀 Job Queued!

Job ID: MSGchvW9Z7GPgZo25OVR
Triggered by: @Serhan-Asad
Label: bug

View Live Progress

Budget cap: none

You can add or change a cap by commenting:

/strength 0.7
/pdd budget N
/pdd settings
/pdd stop

Run controls require write access; /pdd settings is read-only.

[prompt-driven-github]

🚀 Job Queued!

Job ID: mD8nWIsDLYCmOrRyAZRq
Triggered by: @Serhan-Asad
Label: sync

View Live Progress

Budget cap: none

You can add or change a cap by commenting:

/strength 0.7
/pdd budget N
/pdd settings
/pdd stop

Run controls require write access; /pdd settings is read-only.

[prompt-driven-github]

[prompt-driven-github]

Step 1/12: Search for duplicate issues

Status: No duplicates found

Search Performed

Searched for issues using the following key terms:

• subprocess pdd executable PATH checkout
• _find_pdd_executable agentic_sync_runner
• pdd version mismatch site-packages installed
• shutil which pdd executable subprocess sync
• sync self-sync checkout binary
• sys.executable python -m pdd subprocess
• pdd binary PATH wrong version installed
• pdd sync runner subprocess version mismatch
• self-sync pdd diagnostic warning

Issues reviewed: ~60 total across all searches (open and closed)

Findings

No existing issue covers the same root cause: _find_pdd_executable() in agentic_sync_runner.py resolving shutil.which("pdd") to an installed package binary rather than the source-checkout interpreter, causing subprocess sync to execute a different version than the parent process.

Related issues confirmed as distinct (not duplicates):

• pdd-bug: report recoverable step failures as degraded instead of fatal #1641 (OPEN) — About degraded vs. fatal step-comment semantics in pdd bug. This is where the problematic behavior in pdd sync subprocess runner may use installed pdd instead of checkout during self-sync #1648 was observed, but pdd-bug: report recoverable step failures as degraded instead of fatal #1641 does not address subprocess executable resolution.
• Investigate seeding pdd sync generation with existing compatibility surface #1647 (OPEN) — About seeding pdd sync generation with existing compatibility surface data. pdd sync subprocess runner may use installed pdd instead of checkout during self-sync #1648 explicitly notes this is a separate concern.
• pdd sync: honor PDD_SYNC_MAX_WORKERS and bound child output capture to reduce SIGKILL risk #1565 (CLOSED) — About PDD_SYNC_MAX_WORKERS and child output capture; subprocess process management but unrelated to executable path selection.
• Fix pdd sync crashes on malformed incremental responses and invalid generated Python #1612 (CLOSED) — About sync crashes on malformed incremental responses; unrelated root cause.

No existing PRs or branches referencing issue #1648 were found.

Conclusion: This is a new, distinct issue. The subprocess executable path selection logic (shutil.which("pdd") preference) during self-sync in pdd/agentic_sync_runner.py has not been reported or fixed previously.

---

Proceeding to Step 2: Documentation Check

---

Posted by PDD orchestrator (trusted credentials).

🤖 Ground Truth

(no changes)

[prompt-driven-github]

PDD Agentic Sync Progress

Issue: #1648

Module	Status	Phase	Duration	Cost
agentic_sync_runner	Failed	-	11m 15s	$3.10

Total cost: $3.47
Paused: agentic_sync_runner failed

Failed module details

agentic_sync_runner

│ Total time: 189.06s | Total cost: $1.6117 | Overall status: Failed │
build_test_churn_hard_failure_from_error.exc).
removed:GITHUB_FAILED_DETAIL_MIN_CHARS, 
removed:GITHUB_FAILED_DETAIL_PER_MODULE_LIMIT, removed:HEARTBEAT_INTERVAL, 
removed:_run_gh_command, removed:build_conformance_hard_failure_from_error, 
removed:build_public_surface_hard_failure_from_error, 
removed:build_test_churn_hard_failure_from_error, removed:codecs, 
An unexpected error occurred during sync for 'python': Architecture conformance 
error for agentic_sync_runner_python.prompt: the prompt's <pdd-interface> 
build_conformance_hard_failure_from_error.exc (annotation: declared 
`ArchitectureConformanceError`, found `Any`), 
build_public_surface_hard_failure_from_error.exc (annotation: declared 
`PublicSurfaceRegressionError`, found `Any`), 
build_test_churn_hard_failure_from_error.exc (annotation: declared 
`TestChurnError`, found `Any`). Output: 
│ python   │ Failed │    $1.6117 │ Architecture conformance error for          │
│          │        │            │ build_conformance_hard_failure_from_error.… │
│          │        │            │ `ArchitectureConformanceError`, found       │
│          │        │            │ `PublicSurfaceRegressionError`, found       │
│          │        │            │ build_test_churn_hard_failure_from_error.e… │
│          │        │            │ (annotation: declared `TestChurnError`,     │

=== architecture conformance failure ===
prompt: │
output: /tmp/pdd_job_mD8nWIsDLYCmOrRyAZRq_to_uqjv0/pdd/agentic_sync_runner.py
expected: <unknown>
found: <unknown>
missing: build_conformance_hard_failure_from_error.exc, build_public_surface_hard_failure_from_error.exc, build_test_churn_hard_failure_from_error.exc

Reproduce locally: pdd sync agentic_sync_runner

--- env ---
pdd.__file__: /opt/venv/lib/python3.12/site-packages/pdd/__init__.py
pdd --version: pdd, version 0.0.276.dev10
git SHA: 44df4a5bcec29c1848da20415ac17d4f80e470d1
git status: clean
source: site-packages

[prompt-driven-github]

Step 2/12: Check documentation for user error

Status: Confirmed Bug

Documentation Reviewed

• pdd/__main__.py — inline docstring explaining subprocess invocation intent
• pdd/agentic_sync_runner.py — _find_pdd_executable() (line 236) and _build_command() (line 2042)
• pdd/agentic_change_orchestrator.py — subprocess invocation pattern (line 1415–1421)
• pdd/agentic_sync.py — subprocess invocation pattern (line 1419)
• pdd/prompts/agentic_sync_runner_python.prompt — prompt spec for the runner (line 191)
• CONTRIBUTING.md, README.md, docs/sync_architecture_analysis.md, docs/faq.md

Findings

The behavior reported is inconsistent with the documented intent in the codebase itself.

pdd/__main__.py explicitly documents the preferred pattern:

"""Allow ``python -m pdd ...`` to invoke the same CLI as the ``pdd`` script.

The package's console-script entry point is ``pdd = "pdd.cli:cli"`` (see
``pyproject.toml``). Without this module, ``python -m pdd`` raises
``No module named pdd.__main__``, which breaks subprocess invocations that
prefer ``sys.executable -m pdd`` for venv parity (so the subprocess uses the
same interpreter and installed package as the caller, regardless of which
``pdd`` binary is first on PATH).
"""

This module was explicitly created to support sys.executable -m pdd for venv parity — precisely to avoid the PATH-resolved binary disagreeing with the calling process. The problem in #1648 is exactly the failure mode the __main__.py docstring is protecting against.

Other parts of the codebase already follow the correct pattern:

• pdd/agentic_change_orchestrator.py (line 1415): explicitly uses [sys.executable, "-m", "pdd", "update", ...] with a comment: "Use sys.executable + -m pdd so the heal subprocess uses the [correct interpreter]"
• pdd/agentic_sync.py (line 1419): uses [sys.executable, "-m", "pdd"] directly

The bug is in _find_pdd_executable() in pdd/agentic_sync_runner.py:

def _find_pdd_executable() -> Optional[str]:
    """Find the pdd executable path."""
    candidate = shutil.which("pdd")   # ← resolves installed binary, ignores current interpreter
    if candidate:
        return candidate
    # Falls back to sys.executable only if shutil.which finds nothing
    ...

The same pattern exists in pdd/server/routes/commands.py (line 440) and pdd/server/jobs.py (line 204).

Summary of discrepancy:

• __main__.py and the agentic_change_orchestrator / agentic_sync patterns document that subprocess pdd invocations should use sys.executable -m pdd for interpreter parity.
• _find_pdd_executable() in agentic_sync_runner.py (and the server modules) instead prefers shutil.which("pdd") first, which can resolve to an installed binary from a different version than the parent process — exactly the mismatch observed in pdd-bug: report recoverable step failures as degraded instead of fatal #1641.

This is a confirmed bug: the subprocess executable resolution logic in agentic_sync_runner.py contradicts the documented and partially-implemented intent to use sys.executable -m pdd for venv parity.

---

Proceeding to Step 3: Triage

---

Posted by PDD orchestrator (trusted credentials).

🤖 Ground Truth

(no changes)

[prompt-driven-github]

Step 3/12: Assess if enough info to proceed

Status: Fast-Track

Pre-Diagnosed Issue Detected

The issue author has provided a detailed root cause analysis, and Step 2 independently confirmed it with additional file-level evidence.

Identified Root Cause:
_find_pdd_executable() in pdd/agentic_sync_runner.py (line 236) prefers shutil.which("pdd") when resolving the subprocess executable. In a source checkout, this resolves to the installed site-packages binary rather than the current interpreter/package. This causes pdd sync subprocesses to silently execute an older installed version (e.g., 0.0.276) while the parent process runs the checkout version (e.g., 0.0.278.dev0), making self-sync debugging unreliable.

The same antipattern exists in pdd/server/routes/commands.py (line 440) and pdd/server/jobs.py (line 204).

The fix direction is clear: use [sys.executable, "-m", "pdd"] instead of shutil.which("pdd") — exactly the pattern already used correctly in pdd/agentic_change_orchestrator.py (line 1415) and pdd/agentic_sync.py (line 1419), and explicitly documented as the intent in pdd/__main__.py's module docstring.

Evidence Quality:

• File paths with line numbers: Yes — pdd/agentic_sync_runner.py line 236 (_find_pdd_executable), line 2042 (_build_command); pdd/server/routes/commands.py line 440; pdd/server/jobs.py line 204
• Causal explanation: Yes — shutil.which("pdd") resolves to the installed binary on PATH, bypassing the current interpreter's package, contradicting the documented intent in __main__.py and the pattern used by agentic_change_orchestrator.py and agentic_sync.py
• Reproduction evidence: Yes — logs showing pdd.__file__: /opt/anaconda3/lib/python3.12/site-packages/pdd/__init__.py with version 0.0.276 while python -m pdd --version reports 0.0.278.dev0 from the same checkout

Summary

The bug is fully diagnosed with specific file paths, line numbers, a causal explanation, and reproduction evidence. The fix is well-defined: replace the shutil.which("pdd")-first logic in _find_pdd_executable() with sys.executable -m pdd (consistent with agentic_change_orchestrator.py, agentic_sync.py, and the documented intent of __main__.py). Optionally add a warning when PATH pdd and imported pdd.__file__ disagree. Diagnosis steps can be skipped.

---

Fast-tracking to Step 6: Root Cause Analysis (skipping Steps 4-5)

---

Posted by PDD orchestrator (trusted credentials).

🤖 Ground Truth

(no changes)

[prompt-driven-github]

Step 6/12: Analyze root cause

Root Cause Identified: Yes

Summary

_find_pdd_executable() in pdd/agentic_sync_runner.py (line 236) calls shutil.which("pdd") first and returns that PATH-resolved binary path as soon as it finds any pdd executable. When a user has both a source checkout (e.g., 0.0.278.dev0) and an older installed package (e.g., 0.0.276) on the same machine, shutil.which("pdd") resolves to the installed site-packages binary, which is then fed to _build_command() (line 2044). Every pdd sync subprocess therefore executes the installed version rather than the checkout, making self-sync results and debugging unreliable. The same incorrect pattern exists as independent duplicate implementations in pdd/server/routes/commands.py (line 434) and pdd/server/jobs.py (line 199).

Technical Details

• Location: pdd/agentic_sync_runner.py:236–247
• Problematic Code:

def _find_pdd_executable() -> Optional[str]:
    """Find the pdd executable path."""
    candidate = shutil.which("pdd")   # resolves installed binary, ignores current interpreter
    if candidate:
        return candidate              # returns immediately — checkout version never considered

    py_dir = Path(sys.executable).parent
    for name in ("pdd", "pdd.exe"):
        p = py_dir / name
        if p.exists():
            return str(p)
    return None                       # sys.executable -m pdd only used when ALL of these fail

• Why it fails: shutil.which("pdd") resolves to whichever pdd binary appears first on PATH. In a developer checkout, that is typically the globally-installed version (e.g., from conda/pip), not the checkout being tested. The function never considers sys.executable -m pdd, which is the interpreter-parity approach documented in pdd/__main__.py and already used correctly in pdd/agentic_change_orchestrator.py (line 1421) and in the else-branch of _build_command() itself (line 2048).

Call graph through _build_command():

# agentic_sync_runner.py:2042
def _build_command(self, basename, in_flight_cost=0.0):
    pdd_exe = _find_pdd_executable()    # returns site-packages binary path
    if pdd_exe:
        cmd = [pdd_exe, "--force"]      # subprocess uses wrong version
    else:
        cmd = [sys.executable, "-m", "pdd", "--force"]  # correct path, never reached

Same bug in sibling copies:

# server/routes/commands.py:434 and server/jobs.py:199 — identical logic
def _find_pdd_executable() -> Optional[str]:
    pdd_path = shutil.which("pdd")   # same wrong preference
    if pdd_path:
        return pdd_path
    ...

Correct pattern already in use:

# agentic_change_orchestrator.py:1415–1421
# Use sys.executable + -m pdd so the heal subprocess uses the
# same Python venv as the orchestrator. A bare ["pdd", ...]
# would pick up whatever pdd binary is on PATH, which can be
# a different version when devs have a global install plus a
# project-local one.
result = subprocess.run(
    [sys.executable, "-m", "pdd", "update", "--sync-metadata", drift.code_path],
    ...
)

External Research

• Search terms: shutil.which subprocess version mismatch python sys.executable -m module, subprocess venv parity python -m vs script binary, python subprocess same interpreter sys.executable module invocation
• Sources checked: Python docs (subprocess, sys.executable), PyPA packaging guidance
• Relevant findings: Python packaging best practice for launching subprocesses from within a package is to use [sys.executable, "-m", "<package>"] rather than resolving a script name via PATH. This guarantees the subprocess uses the same interpreter and installed version as the caller. This is documented within the project itself in pdd/__main__.py's module docstring, written precisely to support this pattern.

Repository History

• Relevant commits: Only one commit is visible in this shallow clone (44df4a5). No deeper history for pdd/agentic_sync_runner.py is available.
• Related PRs: PR Resolve foreign and nested prompts against their owning architecture.json #1624 (merged, fix/issue-1614) — visible but unrelated. No prior PRs found addressing executable resolution.
• Similar past issues: pdd-bug: report recoverable step failures as degraded instead of fatal #1641 (OPEN) — where the version mismatch was first observed in a failure log. The _env_fingerprint() diagnostic in agentic_sync_runner.py (line 761) already captures pdd.__file__, pdd --version, git SHA, and source (repo vs site-packages) in core-dump blocks, confirming the observation. The fingerprint itself runs ["pdd", "--version"] via PATH, which is intentional for diagnostic contrast (it should show what PATH resolves, not what sys.executable would use).

Experiments Performed

1. Traced _find_pdd_executable() call chain: _build_command() at line 2044 calls _find_pdd_executable() and only falls through to [sys.executable, "-m", "pdd"] when the function returns None — but the function returns a non-None path whenever any pdd binary exists on PATH, so the fallback is never reached in the affected scenario.
2. Confirmed agentic_sync.py imports _find_pdd_executable from agentic_sync_runner.py (line 29) and uses it in the dry-run subprocess at line 1415 — fixing agentic_sync_runner.py covers this caller automatically.
3. Confirmed durable_sync_runner.py does NOT define or call _find_pdd_executable and has no direct pdd subprocess invocation — safe.
4. Verified pdd/server/routes/commands.py (line 434) and pdd/server/jobs.py (line 199) each define their own independent copy of _find_pdd_executable() with the same shutil.which("pdd") preference — these are sibling bugs not covered by the issue's proposed fix.

Fix Scope

• Type: SIBLING_PATTERN
• Why this classification: The root cause is a specific function _find_pdd_executable() that is defined in three separate files, each copy containing the identical shutil.which("pdd")-first logic. Each copy drives its own subprocess invocations. Fixing the primary copy in agentic_sync_runner.py automatically covers agentic_sync.py (which imports from it), but the two server-module copies require independent fixes.
• Sibling search performed: Grepped all .py files under pdd/ for _find_pdd_executable and shutil\.which.*pdd. Found exactly three definition sites.
• Confirmed siblings (NEEDS_FIX):
  • NEEDS_FIX: pdd/server/routes/commands.py:434 | Duplicate _find_pdd_executable() with identical shutil.which("pdd")-first logic; drives _build_pdd_command_args() at line 490
  • NEEDS_FIX: pdd/server/jobs.py:199 | Duplicate _find_pdd_executable() with identical shutil.which("pdd")-first logic; drives _build_subprocess_command_args() at line 228
• Looked-but-safe (SAFE_EVIDENCE):
  • SAFE_EVIDENCE: pdd/durable_sync_runner.py | durable_sync_runner.py (full file) | No _find_pdd_executable definition or call, no direct pdd subprocess invocation
  • SAFE_EVIDENCE: pdd/agentic_sync.py:1415 | pdd/agentic_sync.py:29 | Imports _find_pdd_executable from agentic_sync_runner — covered by fixing the primary definition
  • SAFE_EVIDENCE: pdd/agentic_sync_runner.py:2048 | pdd/agentic_sync_runner.py:2047-2048 | else-branch already uses [sys.executable, "-m", "pdd"] — correct, but unreachable in practice because the buggy function above always returns a path first

Sibling Bugs

• Variable reference audit:
  • pdd/agentic_sync_runner.py:236 — definition (WRITE/DEFINE): _find_pdd_executable() — returns shutil.which("pdd") path if found
  • pdd/agentic_sync_runner.py:2044 — read (call in _build_command()): uses returned path to build subprocess command
  • pdd/agentic_sync.py:29 — import (read): re-exports the function
  • pdd/agentic_sync.py:1415 — read (call in dry-run function): same execution path affected
  • pdd/server/routes/commands.py:434 — definition (WRITE/DEFINE): independent copy, same bug
  • pdd/server/routes/commands.py:490 — read (call in _build_pdd_command_args()): drives server-side subprocess invocations
  • pdd/server/jobs.py:199 — definition (WRITE/DEFINE): independent copy, same bug
  • pdd/server/jobs.py:228 — read (call in _build_subprocess_command_args()): drives job-runner subprocess invocations
• State symmetry check: Not applicable — this is not a save/restore pattern.
• Independent bugs found: The two server-module copies are independent bugs (they have their own definition rather than sharing the primary one), not merely callers of the broken primary. They would not be fixed by patching only agentic_sync_runner.py.

Proposed Fix Validation

• Issue proposes a fix: Yes
• Proposed fix coverage: PROPOSED_FIX_EXPANSION
• Issue-proposed scope: Fix pdd/agentic_sync_runner.py _find_pdd_executable() to prefer [sys.executable, "-m", "pdd"] when running from a source checkout; optionally add a warning when PATH pdd and pdd.__file__ disagree; include resolved command in logs/core dumps
• Step 6 full scope:
  1. pdd/agentic_sync_runner.py:236 — primary fix (covers agentic_sync.py import automatically)
  2. pdd/server/routes/commands.py:434 — sibling fix required (independent duplicate copy)
  3. pdd/server/jobs.py:199 — sibling fix required (independent duplicate copy)
• Expansion items (PROPOSED_FIX_EXPANSION): pdd/server/routes/commands.py:434 _find_pdd_executable() duplicate with same shutil.which bug, pdd/server/jobs.py:199 _find_pdd_executable() duplicate with same shutil.which bug

Fix Location

• File(s) to modify: pdd/agentic_sync_runner.py, pdd/server/routes/commands.py, pdd/server/jobs.py
• Why this location: These are the three definition sites of _find_pdd_executable(). The callers (_build_command, _build_pdd_command_args, _build_subprocess_command_args) are correct in structure — only the helper function returns the wrong executable path.
• Existing pattern: pdd/agentic_change_orchestrator.py:1415–1421 — explicitly uses [sys.executable, "-m", "pdd"] with a comment explaining why PATH-resolution is unsafe. This is the authoritative pattern to replicate. pdd/__main__.py provides the module-level entry point that makes python -m pdd work, explicitly created to support this approach.
• Fix approach: Change _find_pdd_executable() in all three files to prefer [sys.executable, "-m", "pdd"] unconditionally (return None from the function and rely on the caller's else branch), OR restructure it to check whether the current pdd.__file__ is under the same interpreter's site-packages before deciding whether PATH resolution is safe. The simplest, most consistent approach is to make _find_pdd_executable() always return None so callers always use [sys.executable, "-m", "pdd"] — matching agentic_change_orchestrator.py. Optionally add a diagnostic warning when shutil.which("pdd") resolves to a different path than Path(sys.executable).parent / "pdd".

---

Proceeding to Step 7: Prompt Classification

---

Posted by PDD orchestrator (trusted credentials).

🤖 Ground Truth

(no changes)

[prompt-driven-github]

❌ PDD Execution Failed

Execution failed: PDD command failed with exit code 1.
Output: st: $1.6117 | Overall status: Failed │
build_test_churn_hard_failure_from_error.exc).
removed:GITHUB_FAILED_DETAIL_MIN_CHARS,
removed:GITHUB_FAILED_DETAIL_PER_MODULE_LIMIT, removed:HEARTBEAT_INTERVAL,
removed:_run_gh_command, removed:build_conformance_hard_failure_from_error,
removed:build_public_surface_hard_failure_from_error,
removed:build_test_churn_hard_failure_from_error, removed:codecs,
An unexpected error occurred during sync for 'python': Architectur
Cost: $3.4668
Model: anthropic