Skip to content

Fix --agent drift on paid submit command guidance #10

Description

@chasehuh

Summary

First-time Sume CLI dogfooding exposed a real schema/docs/skills drift: bundled skills and docs instruct agents to add --agent to paid submit commands, but the current submit commands reject --agent before execution.

This breaks the intended agent workflow for Avatar and Avatar Video creation. The agent correctly follows the skill guidance, attempts a paid submit with --agent --json, gets unknown option '--agent', then has to recover by removing --agent. The failed attempts do not submit paid jobs, but the behavior is confusing and makes the skill guidance unreliable.

Why this matters

SUM-619 skill routing expects agents to rely on local skill exports and sume tools schema rather than guessing. If the exported skills recommend a command shape the CLI rejects, a fresh agent loses trust in the routing layer and may start improvising around safety/redaction guidance.

This is especially important for paid generation commands because the guidance currently says agent-readable submit output should use --agent --json to avoid leaking sensitive URLs or private identifiers. If submit commands intentionally do not support --agent, the skills/docs need to say that explicitly and route agents to jobs watch/result --agent --json for redacted readback. If submit commands should support agent mode, the CLI should accept it consistently.

Dogfood evidence

Thread: Sume CLI first-time skills dogfood (019f35bd-1bc2-75a1-b88b-1143770af841)

Flow observed:

  1. Fresh agent started unauthenticated and ran no-cost discovery.
  2. User asked for 3 Korean cosmetics-ad avatars.
  3. Agent correctly stopped because auth was missing.
  4. User approved auth; agent ran browser login and confirmed configured auth without printing secrets.
  5. Agent requested explicit paid approval before generation.
  6. User approved.
  7. Agent attempted to submit avatar creates using the skill-recommended --agent --json pattern.
  8. The CLI rejected the submit attempts before any paid/API call with unknown option '--agent'.
  9. Agent recovered by removing --agent, submitted 3 jobs, and moved to bounded jobs watch polling.

The recovery is acceptable, but the first failure is a real product/DX issue in the skill/CLI contract.

Local no-cost repro

Audited commit:

sumelabs/cli main @ 90f3eb95a803945de919a970fe3d8b60ac954bb5
CLI version 0.1.1

tools schema marks avatars.create as needing agent redaction and being a paid mutating command:

/Users/huhchaewon/sume/cli/dist/sume --json tools schema avatars.create

Observed schema summary:

{
  "name": "avatars.create",
  "command": "sume avatars create --confirm-submit --json ...",
  "safety": {
    "mutating": true,
    "paid_generation_call": true,
    "requires_agent_redaction": true,
    "requires_confirmation": true,
    "returns_sensitive_url": true
  }
}

But avatars create --help does not include --agent:

/Users/huhchaewon/sume/cli/dist/sume avatars create --help

Parser-only repro, no confirm flag and no API submit:

/Users/huhchaewon/sume/cli/dist/sume --json avatars create \
  --type prompt \
  --avatar-handle qa_agent_flag_drift \
  --prompt "Parser-only QA prompt" \
  --agent

Observed:

{
  "error": {
    "code": "invalid_argument",
    "message": "unknown option '--agent'\n(Did you mean --age?)",
    "details": {
      "commander_code": "commander.unknownOption"
    }
  }
}

Same parser-only issue exists for Avatar Video submit:

/Users/huhchaewon/sume/cli/dist/sume --json avatar-videos create \
  --avatar-handle qa_handle \
  --script "Parser-only QA script" \
  --agent

Observed:

{
  "error": {
    "code": "invalid_argument",
    "message": "unknown option '--agent'",
    "details": {
      "commander_code": "commander.unknownOption"
    }
  }
}

No provider calls or paid jobs were run for these repro commands.

Affected surfaces seen in repo text

Examples currently recommending --agent on submit commands include:

  • agent/sume-avatar/SKILL.md
    • sume avatars create ... --confirm-paid --agent --json
    • photo/props examples also include --agent
  • agent/sume-avatar-video/SKILL.md
    • sume avatar-videos create ... --confirm-paid --agent --json
  • agent/sume-min-latency/SKILL.md
    • fast Avatar Video loop includes --agent on create
  • agent/sume-spend-capped-dogfood/SKILL.md
    • paid QA example includes --agent on create
  • agent/sume/references/eval-scenarios.md
    • selected avatar to video says sume avatar-videos create --confirm-paid --agent --json
  • docs/agent-workflows.md
    • submit examples include --agent
    • later says: “Use --agent --json for submit, watch, and result outputs read by agents.”
  • README.md
    • says: “Use --agent --json on submit commands when an agent is reading the response.”

The generated src/lib/bundled-skills-data.ts also contains these examples, so any source skill edits must regenerate bundled skill data.

Expected behavior

Pick one contract and make all surfaces agree.

Recommended contract:

  1. Submit commands that can return sensitive URLs/private readback should accept --agent and return redacted agent-safe output consistently.
  2. sume tools schema <submit-tool> should advertise only flags the CLI actually accepts.
  3. Bundled skills/docs should use exactly the accepted command shape.
  4. If submit commands intentionally should not support --agent, then:
    • remove --agent from submit examples;
    • update requires_agent_redaction/tool descriptions to explain that submit output is not agent-redacted and agents should immediately use jobs watch/result --agent --json for readback;
    • make the safety story explicit so agents do not infer the wrong flag.

Given current skill intent, adding --agent support to submit commands seems cleaner than weakening the agent-safe submit story.

Acceptance criteria

  • sume avatars create --help and sume avatar-videos create --help either include --agent, or every skill/docs example stops using it for submit commands.
  • Parser-only commands with --agent no longer fail with unknown option if --agent is the chosen contract.
  • sume tools schema avatars.create --json and sume tools schema avatar-videos.create --json accurately reflect the supported CLI flags and redaction behavior.
  • Bundled skills source and generated src/lib/bundled-skills-data.ts are consistent.
  • Regression tests cover:
    • avatars.create submit command examples vs help/schema;
    • avatar-videos.create submit command examples vs help/schema;
    • no old sume.so surfaces are introduced;
    • paid submit commands still require explicit confirmation;
    • agent-safe mode does not print full media URLs, signed URLs, API keys, provider payloads, or private identifiers.
  • First-time dogfood flow can follow exported skills without hitting an unknown option before paid submit.

Validation used for this issue

  • Read-only local CLI schema/help inspection.
  • Parser-only repro commands that fail before API submission.
  • No paid/provider calls for the repro.
  • No secrets printed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions