Channel relay: the api-key→scope mirror is load-bearing — cleanup scope to decide (post-incident)

[eanzhao]

Context

The aevatar-side channel-bot registration subsystem (POST /api/channels/registrations facade → NyxLark/NyxTelegramProvisioningService → the ChannelBotRegistration GAgent/readmodel "mirror" → NyxIdRelayScopeResolver) looked like a candidate for cleanup as "redundant self-registration": a Lark/Feishu bot can be registered directly on NyxID (nyxid channel-bot register + api-key + route), and the agent-facing tools (nyxid_channel_bots, nyxid_api_keys, nyxid_services) already expose those same NyxID operations.

A cleanup that deleted the facade and the mirror was merged + deployed, then reverted after it took down every relay Lark bot. This thread captures the current state and the open question: what is actually safe to clean, and what would a correct, larger cleanup require?

The incident — why the mirror is NOT redundant

Inbound flow: NyxID relays a Lark message to aevatar /api/webhooks/nyxid-relay with two tokens:

• callback token (X-NyxID-Callback-Token) — RS256 JWT with integrity/routing claims only (api_key_id, message_id, body_sha256, platform). No identity, no scope.
• user token (X-NyxID-User-Token) — the bot owner's relay access token: sub = bot-owner NyxID uuid, scope = OAuth scopes (read write), agent_scope = api_key_id. No aevatar scope_id claim.

aevatar needs the aevatar tenant scope_id to route the turn to the right ConversationGAgent. The old code resolved it as callback.(scope_id ?? sub ?? NameIdentifier) and, when that is null (which it always is in prod), fell back to the mirror (api_key_id → scope_id, written at registration). Deleting the mirror made every callback resolve to null → Results.Unauthorized() (401) → no bot replied.

Root cause: aevatar's scope_id is a separate tenant id that diverges from the bot owner's NyxID identity (sub). e.g. a registration scope_id = 2c5c9b72… while the owner uuid = 2db990b5…. Neither relay token carries 2c5c9b72…; only the mirror maps api_key_id → 2c5c9b72….

Load-bearing vs redundant (current understanding)

Piece	Status
ChannelBotRegistration readmodel + NyxIdRelayScopeResolver (api-key → scope mirror)	LOAD-BEARING — the only scope source for relay callbacks (the callback JWT has no scope)
NyxLarkProvisioningService.PersistRelayApiKeySecret (relay api-key full_key → secrets store)	DEAD — threw on the mainnet read-only EnvironmentSecretsStore (the original 502); never read by the live reply path; Telegram never persisted it
facade NyxID resource creation (channel-bot / api-key / route)	redundant with the nyxid_channel_bots / nyxid_api_keys / nyxid_services agent tools — but it also writes the mirror, which the tools/CLI do not
NyxReplyCredentialRef / DurableReplyCredentialRef / WorkflowRunDeliveryGAgent durable-cred plumbing	likely removable (now always empty), but touches actor-state protos — needs care

Already landed (the safe subset, separate PR)

1. Removed the dead relay-api-key secret persistence (_secretsStore.Set in NyxLarkProvisioningService) — unbreaks facade registration on the read-only-secrets mainnet host (so it can create the mirror) without changing the reply path; aligns Lark with Telegram, which never persisted it.
2. Added a sub fallback to relay scope resolution: mirror first, then user-token (scope_id ?? uid ?? sub). Additive — only fires when there is no mirror entry (e.g. a bot registered directly on NyxID via CLI/tools), routing it to the bot-owner identity scope. Existing mirror-registered bots are unchanged.

Open questions — cleanup scope to decide

1. Can the mirror be removed entirely? Only if either (a) the scope model is aligned so a bot's aevatar scope == the owner's NyxID identity (then relay sub suffices — but bots bound to a distinct scope_id need a data migration), or (b) NyxID carries the aevatar scope_id in the relay token (a NyxID-side change). The sub fallback covers the "scope == owner identity" case only; it is not a full replacement.
2. Slim the facade? Could the facade be reduced to only writing the mirror (api-key → scope), delegating NyxID resource creation to the agent tools/CLI — i.e. one thin "bind this api-key to this scope" entry point?
3. What is the canonical registration entry point going forward — the facade, the agent tools, or the CLI? (The CLI/tools path produces no mirror; with the sub fallback it works only when scope == owner identity.)
4. Remove the NyxReplyCredentialRef / WorkflowRunDeliveryGAgent durable-cred plumbing (now always empty)?

Lesson recorded for the team: do not trust static code-reading (“the callback JWT carries sub”) over runtime logs for a load-bearing path. The deletion + its adversarial review both reasoned from code; production logs showed the opposite.