Agentic customer-support drafting over a live e-commerce backend. Grounded drafts, a deterministic policy gate, and a human on every send.
In production: the workflow this repo rebuilds has drafted real support replies for this shop for months and took over most of the manual drafting.
▶ Watch the demos · four live clips: the research pass, a grounded draft, an injection attempt in the inbox, and the quarantine artifact.
Concierge reads the support inbox of an online shop I own and prepares reply drafts for a human to review and send. For each incoming thread it runs an agent loop that pulls order status, live stock, the customer's prior correspondence, and an internal knowledge base through a typed MCP tool layer, then writes a grounded draft straight into Gmail Drafts. It is a standalone service on the Claude Agent SDK (Node.js / TypeScript), not a chat UI: the architecture is a poll loop, a tightly scoped tool surface, a deterministic policy gate, and an audit trail.
Prime directive: draft only, never send. Enforced three times independently: the OAuth grant carries no send scope, the codebase has no send path, and the agent has no send tool. Sending is always a human action in Gmail.
Real. The design is the real system: the typed tool layer, the deterministic policy gate, the human-in-the-loop draft flow, the audit trail, and the eval harness. The original draft-assist workflow has run against this shop for months and took over most of the manual drafting, and this repository is its standalone rebuild on the Claude Agent SDK, migrating into place through shadow parity. What ships in this repo is a sanitized, runnable sample of that design: the policy gate and its eval harness are the real logic, while the agent loop and the tools are runnable, typed illustrations over synthetic data, with the tool layer exercised end to end by the smoke run.
Sanitized. The shop's brand and its suppliers are withheld and referred to only as an online shop I own. The demo clips are recorded on a test mailbox with synthetic customer emails running against the live shop backend, so you see the real research and drafting path with no real customer data on screen.
Four clips, each one beat, recorded live.
01_research.mp4
What to watch: one command handles the thread, and the trace shows the agent reading the conversation and checking live shop data before it writes anything.
02_grounded-draft.mp4
What to watch: the recommendation and the stock status come from the live systems, not from the model's imagination, and the draft waits in Gmail for a person to send.
03_attacker.mp4
What to watch: email content is data, not instructions, so no customer reply is drafted and nothing is disclosed or actioned.
04_quarantine.mp4
What to watch: the thread is quarantined into an internal warning draft with no recipient, so it cannot be sent by accident, spelling out what the injection attempted.
The design is a short list of deliberate decisions, each one about where things go wrong in production.
The tool surface is the sandbox. The agent gets a small set of typed, allowlisted MCP tools and nothing else: no filesystem, no shell, no web. Everything it can do to the outside world is a named tool with a schema, and the mail tools can read, search, and draft only, with no send path at all.
Retrieval is deterministic, not a vector pipeline. The data is structured and authoritative (orders, stock, mail history), so the agent does exact lookups against sources of truth instead of approximate similarity search. A RAG layer would add a failure mode here, not remove one.
Email is untrusted data. Message content is treated as adversarial input, never as instructions. When a thread tries to manipulate the agent, it is quarantined into a recipient-less warning draft rather than answered.
A deterministic gate sits between the model and any draft. Before a draft can exist, its body passes a policy gate in code that rejects secret-shaped strings, internal pricing vocabulary, unsupported claims, and style tells. The model is asked to follow the rules; this layer makes the non-negotiable ones non-negotiable.
The human is the only send path, and edits are protected. Drafts land in Gmail for review. The service supersedes only its own untouched drafts and never clobbers one a human has edited, and solved-ness is derived from message ids, so a fresh customer reply always re-opens the thread.
Every run is auditable and cost-bounded. Each tool call, policy rejection, draft, and per-run cost is written to an append-only audit log, under per-run thread caps, per-thread turn caps, a pinned model, and a hard monthly spend limit.
flowchart TB
T["Trigger (cron / scheduler)"] --> POLL["Cheap poll: unhandled threads, no model"]
POLL --> LOOP["Per-thread agent loop (Claude Agent SDK)"]
LOOP -->|"typed, allowlisted, read-only"| TOOLS["MCP tools: order status · live stock · prior correspondence · knowledge base · mail read / search / draft"]
TOOLS --> LOOP
LOOP --> DEC{"Email content is data, not instructions"}
DEC -->|"normal request"| CAND["Grounded reply draft"]
DEC -->|"injection / manipulation"| QUAR["Quarantine draft, no recipient"]
CAND --> GATE["Deterministic policy gate (code)"]
QUAR --> GATE
GATE -->|"pass"| GD["Gmail Drafts"]
GATE -->|"secret · pricing · bad claim · style"| BLOCK["Blocked and flagged for a human"]
GD --> HUMAN["Human reviews, edits, sends"]
BLOCK --> HUMAN
LOOP --> AUDIT[("Audit trail: every tool call, rejection, draft, cost")]
- Claude Agent SDK (Node.js / TypeScript): the service and the per-thread agent loop.
- Typed MCP tool layer (zod schemas): mail (read / search / draft) and shop (order status, live stock, customer research), all read-only and allowlisted.
- Deterministic policy gate (TypeScript): output checks enforced at the tool layer before any draft exists, reused verbatim by the eval harness.
- Gmail Drafts: the human-in-the-loop surface and the only send path.
- Append-only audit trail (JSONL): every tool call, policy rejection, draft, and per-run cost.
- Docker: containerized single-pass runs.
- Anthropic Claude: model pinned via config, behind a hard monthly spend cap.
Correctness here is not a vibe check; it is a runnable harness.
Seven policy fixtures across four violation classes, two real production incidents folded in as permanent regression cases, all at zero model cost.
The policy gate is the eval. Every fixture draft runs through the exact gate the production draft tool enforces, in CI, on every change, at zero model cost (npm run eval). A green build means the hard rules still hold.
An injection canary is a permanent fixture. One fixture is a draft that would leak internal pricing if an injection had succeeded, and it must be caught. Clean fixtures must pass untouched, including a real false-positive found in a live test: product URLs carry long random slugs that first looked like leaked tokens, the fix scopes the token check to prose, and the case is now a regression test.
Production incidents become regression cases. Two real incidents (a discounted-clearance complaint and a license-generation failure) are folded in as permanent cases, moving toward a model-in-the-loop groundedness stage that replays sanitized threads and judges whether the answer is actually supported by tool output.
Cheap, model-free checks gate every change. A smoke run exercises the whole tool layer end to end with no model and no cost, and the eval harness runs the policy fixtures. Both run before anything touches a live mailbox.
I own the architecture, the tool boundaries, the failure modes, and the call on what is safe to ship; the implementation was built by directing Claude Code, and the reply rules the agent follows were tuned over weeks of reviewed production drafts.
Production. One of a set of production AI systems built the same way.
- Portfolio: github.com/janvrsinsky
- LinkedIn: linkedin.com/in/janvrsinsky