Skip to content

TOOL-30021 Enable sdb upstream tracking + bootstrap upstreams branch on first sync#393

Merged
prakashsurya merged 2 commits into
developfrom
projects/spec-0003-phase-2
May 26, 2026
Merged

TOOL-30021 Enable sdb upstream tracking + bootstrap upstreams branch on first sync#393
prakashsurya merged 2 commits into
developfrom
projects/spec-0003-phase-2

Conversation

@prakashsurya
Copy link
Copy Markdown
Contributor

@prakashsurya prakashsurya commented May 26, 2026

Background

Spec 0003 Phase 2 onboards sdb as an upstream-tracked package against sdimitro/sdb. Phase 1 (delphix/sdb#360) merged 2026-05-06 and landed the bootstrap merge from sdimitro/develop plus the dh-virtualenv overlay. The linux-pkg side is wiring update_upstream() up against sdimitro/sdb's develop, and having build() derive PACKAGE_VERSION from setuptools_scm so the .deb's metadata version follows upstream sdb tags automatically.

Problem

The first check-updates run against the new sdb recipe fails before reaching update_upstream at all, e.g.

Running: [token passed] git fetch https://github.com/delphix/sdb.git --no-tags +upstreams/develop:upstream-HEAD
fatal: couldn't find remote ref upstreams/develop
Error: git fetch failed

fetch_repo_from_git requires upstreams/$DEFAULT_GIT_BRANCH to already exist on the package repo's origin before the update_upstream stage even runs; update_upstream_from_git only fast-forwards an existing local upstream-HEAD, it doesn't create it. The linux-pkg README's onboarding section (around line 740) says you're supposed to push that branch by hand at repo-creation time, but the branch protection on delphix/sdb restricts pushes to upstreams/* to delphix-devops-bot only, so a human spec author can't do it themselves; the bot only pushes during the normal sync flow, which can't run because the branch doesn't exist. Chicken-and-egg.

The same gap applies to every future upstream-tracked package that gets onboarded.

Solution

The solution is to bootstrap upstream-HEAD from UPSTREAM_GIT_URL / UPSTREAM_GIT_BRANCH directly when the package repo's upstreams/<branch> is missing, and touch $WORKDIR/upstream-updated so sync-with-upstream.sh pushes the bootstrapped ref back via the bot creds it already uses at the push step. git push of a non-existent remote branch creates it, so the first sync run becomes the moment upstreams/<branch> is born on origin; subsequent runs are no different from the existing fast-forward path.

To make the conditional fetch possible, git_fetch_helper's internal || die is dropped. The four existing callers all use logmust, which handles failure identically, so the new fallback caller can branch on the exit code without losing the fail-hard behavior elsewhere.

Two files change: the bootstrap fallback in lib/common.sh, and the sdb recipe at packages/sdb/config.sh, which surfaces the gap and is needed to verify the fallback end-to-end against a real upstream-tracked recipe.

Testing Done

Jenkins check-updates #1148 on this branch with PACKAGES=sdb, UPDATE_PACKAGES=false. The bootstrap path hits, upstream-HEAD populates from sdimitro/sdb develop, and "Package sdb has updates available" reports cleanly, e.g.

fatal: couldn't find remote ref upstreams/develop
NOTE: upstreams/develop does not exist on https://github.com/delphix/sdb.git; bootstrapping upstream-HEAD from https://github.com/sdimitro/sdb.git (develop).
Running: [token passed] git fetch https://github.com/sdimitro/sdb.git --no-tags +develop:upstream-HEAD
 * [new branch]      develop    -> upstream-HEAD
Running: touch .../upstream-updated
Success: Package sdb has updates available.

For contrast, #1147 at the prior branch tip confirms the original failure mode.

Post-merge verification (out of scope here):

  • Seed job picks up sdb's update_upstream() definition and auto-generates update-package/sdb.
  • First nightly cron run, after delphix/devops-gate#4438 also lands, exercises the real push of upstreams/develop to delphix/sdb and merges upstream into develop. That also satisfies the long-deferred Phase 1 AC Add unit tests #2 as a side-effect.
  • build-package/sdb/pre-push to verify the .deb's Version: field derives from setuptools_scm (Phase 2 AC Add package definition for make-jpkg #11).

Spec context: delphix/cd-aidlc#62. JIRA: TOOL-30021.

…ssing

fetch_repo_from_git previously failed hard if a package's
upstreams/$DEFAULT_GIT_BRANCH branch did not yet exist on origin,
blocking the very first upstream-sync run for any newly onboarded
upstream-tracked package. The package-onboarding flow expected this
branch to be pre-seeded manually before automation could run, but the
branch protection (push restricted to delphix-devops-bot) makes that
manual step inaccessible to the spec authors typically onboarding the
package.

Detect the missing-branch case and bootstrap upstream-HEAD directly
from UPSTREAM_GIT_URL/UPSTREAM_GIT_BRANCH, marking
$WORKDIR/upstream-updated so sync-with-upstream.sh pushes
upstream-HEAD back to origin -- git push creates the upstreams/<branch>
branch on origin in the process. Requires both UPSTREAM_GIT_URL and
UPSTREAM_GIT_BRANCH to be set; falls back to the previous die with a
clearer message otherwise.

To make the bootstrap fallback possible, git_fetch_helper no longer
dies internally on failure -- the four existing logmust callers still
fail-hard via logmust's own error handling, while the new fallback
caller can branch on the exit code.

Follow-up to delphix/cd-aidlc spec 0003 Phase 2, surfaced by the
first run of check-updates against the new sdb upstream-tracking
recipe.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
@prakashsurya prakashsurya changed the title TOOL-30021 Enable sdb upstream tracking + bootstrap upstreams/<branch> on first sync TOOL-30021 Enable sdb upstream tracking + bootstrap upstreams branch on first sync May 26, 2026
Wire packages/sdb/config.sh up to the upstream-tracking automation
introduced by the prior commit, completing Phase 2 of
delphix/cd-aidlc spec 0003 (Adopt sdimitro/sdb as upstream;
repackage via dh-virtualenv).

The build() override derives PACKAGE_VERSION from setuptools_scm so
the .deb's metadata version follows upstream sdb tags automatically;
pyproject.toml's local_scheme="no-local-version" keeps the value
debian-version-safe (no embedded "-"), letting linux-pkg's
set_changelog append the standard "-1delphix.<ts>" revision suffix
without further massaging.

Phase 1 (delphix/sdb#360, merged 2026-05-06) landed the bootstrap
merge from sdimitro/develop plus the dh-virtualenv overlay,
including python3-setuptools-scm in Build-Depends so the build()
override here works with no companion delphix/sdb change.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
@prakashsurya prakashsurya force-pushed the projects/spec-0003-phase-2 branch from 1b64822 to 9bb44b7 Compare May 26, 2026 22:51
@prakashsurya prakashsurya enabled auto-merge (rebase) May 26, 2026 23:05
@prakashsurya prakashsurya merged commit 69e204a into develop May 26, 2026
12 checks passed
@prakashsurya prakashsurya deleted the projects/spec-0003-phase-2 branch May 26, 2026 23:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants