Skip to content

Fix reshard source-DB drop using empty database name#922

Merged
benben merged 1 commit into
mainfrom
ben/reshard-fix-source-drop
Jul 9, 2026
Merged

Fix reshard source-DB drop using empty database name#922
benben merged 1 commit into
mainfrom
ben/reshard-fix-source-drop

Conversation

@benben

@benben benben commented Jul 9, 2026

Copy link
Copy Markdown
Member

Bug (observed on a real cnpg→cnpg reshard)

The cleanup step of an otherwise-successful cnpg→cnpg reshard failed like this (note the double space — the database name is empty):

dropping source database  on shard-A (WITH FORCE)
DROP DATABASE on the source failed — old catalog database still exists on shard-A and must be dropped manually: drop database : ERROR: zero-length delimited identifier at or near """" (SQLSTATE 42601)

The copy and cutover were fine (cleanup is deliberately non-fatal but loud), but the old catalog database was left orphaned on the source shard.

Root cause

  • The admin handler fills op.SourceEndpoint/SourceUser/SourceDatabase at create time only for external sources (controlplane/admin/reshard.go); for a cnpg source they stay "".
  • recordSource (runner) reads the real source info from the duckling CR status and persists it to the op row via o.fields(...) — but never updated the in-memory o.op.
  • cleanupSource passed o.op.SourceDatabase (still "") to DropDatabaseDROP DATABASE "" → SQLSTATE 42601. The o.source endpoint struct built in recordSource carried the correct database all along (the copy used it successfully).

Fix

  • recordSource now mirrors the persisted source_endpoint/source_user/source_database onto the in-memory o.op, keeping every later reader (cleanup logging, the end-of-op report, the ext→cnpg rollback spec) consistent.
  • cleanupSource drops o.source.Database — the copy-path source of truth — and, belt-and-suspenders, logs an ERROR and skips the drop if the name is ever empty instead of issuing a zero-length identifier.

Test

Regression assertion in TestReshardHappyPathCnpgToCnpg (controlplane/provisioner/reshard_runner_test.go): the fake copier's dropdb call must name the database recorded from the duckling status. The test op's SourceDatabase is unset exactly like production, so pre-fix it fails with:

dropdb database = "", want "mdstore_org" (the recorded source database)

and passes after the fix. go build -tags kubernetes ./... and go test -tags kubernetes ./controlplane/provisioner ./controlplane/admin -run TestReshard -count=1 are green.

Note on e2e: per the resharding contract in CLAUDE.md, the cnpg→cnpg positive path (the only path that reaches cleanupSource) is unit-only until the second mw-dev shard lands, so the regression lives in the runner unit test.

🤖 Generated with Claude Code

For cnpg sources the admin handler leaves op.SourceDatabase empty at
create time (it only fills it for external sources), and recordSource
persisted the real name to the op row without updating the in-memory
op. cleanupSource then issued DROP DATABASE "" — a zero-length
delimited identifier error — leaving the old catalog database orphaned
on the source shard after an otherwise successful cnpg-to-cnpg reshard.

- recordSource now mirrors source_endpoint/user/database onto the
  in-memory op after persisting, so cleanup logging, the report, and
  the ext-to-cnpg rollback all see them.
- cleanupSource drops o.source.Database (the copy-path source of
  truth) and skips loudly if the name is somehow empty instead of ever
  sending a zero-length identifier.
- Regression assertion in TestReshardHappyPathCnpgToCnpg: the dropdb
  call must name the database recorded from the duckling status
  (previously "", reproducing the production failure).
@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown

Test Impact Plan

Deterministic summary of how this PR changes tests, CI runners, and coverage-risk signals.

Summary

Area Added Changed Deleted
Test files 0 1 0
E2E/journey files 0 0 0
Workflow files 0 0 0

Signals

  • Test cases: +0 / -0
  • Assertions: +2 / -0
  • Skips or known failures added: 0
  • Workflow continue-on-error added: 0
  • Workflow path filters added: 0
  • Test commands removed from justfile: 0
  • E2E/journey retry lines added: 0

Coverage risk: neutral or increased

No coverage-reduction warnings detected.

@benben benben merged commit f5042d6 into main Jul 9, 2026
28 checks passed
@benben benben deleted the ben/reshard-fix-source-drop branch July 9, 2026 12:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant