Org-wide Meet transcript ETL + vector store + read-only MCP server

[hhff]

What this is

The first slice of the MCP & agent-features effort: a durable, org-wide vector store fed by an ETL pipeline, with Google Meet transcripts as the first source, exposed to an agent through a read-only MCP server. Built inside Stacks (Rails 6.1) so it reuses the existing Postgres, Google Workspace auth, and ActiveAdmin.

Design spec: docs/superpowers/specs/2026-06-28-org-vector-store-and-etl-design.md · Plan: docs/superpowers/plans/2026-06-28-org-vector-store-and-etl.md · Deploy runbook: docs/meet-etl-deploy.md

How it works

• Generic core (documents → chunks, polymorphic embeddings side-table, mentions, document_contacts, source_syncs) with pgvector (neighbor) + Postgres full-text. Search is source-agnostic, so future connectors light up automatically.
• Connector/ETL framework (Stacks::Etl::Connector): extract → classify → resolve contacts → chunk → embed → load, idempotent upserts, per-source watermark, SystemTask-wrapped rake tasks.
• Meet connector (source Feature/hotfixes #1): hybrid Meet REST API (ongoing, rich) + Drive transcript-Doc sweep (90-day backfill — the API only retains ~30 days). Rich meetings/participants/segments project into the generic core.
• Calendar enrichment: matches each meeting to its Calendar event (precisely — by Meet code / exact title, never by time alone) to recover the real title (re-enabling title-based exclusion) and attendee emails (resolved to Contacts).
• Org-wide multi-user sweep: impersonates every active Workspace user via domain-wide delegation, error-isolated; dedup by global IDs.
• Local embeddings (informers + quantized mxbai-embed-large-v1, 1024-dim) — no API key, no chunk text leaves our infra.
• Read-only MCP server (official mcp gem, Streamable HTTP at /api/mcp, stateless) — search / get_document / list_documents / list_sources, authenticated with the existing X-Api-Key private key.

Privacy / governance

• Exclusion wall: sensitive meetings (1:1s, performance review, comp, HR, …) are auto-classified excluded and never chunked, embedded, or returned by any MCP tool or search mode — verified end-to-end. They keep their full transcript for the human record (ActiveAdmin only).
• Reversible: excluded docs retain segments; an ActiveAdmin Include & index action chunks+embeds from stored segments (no re-fetch).
• Identity is anchored on the existing contacts table.

Admin

New top-level MCP menu → ETL subpages (Meetings, Documents, Source syncs); Chunks + Mentions are reached by drilling into a Document. Document show has Include & index / Exclude actions.

Testing

~60 new tests / 150+ assertions, plus the full suite green (313 runs, 0 failures). Schema verified to load cleanly from scratch (db:test:prepare). Enrichment + ingest verified against the live Workspace.

Deploy notes (see runbook)

• Heroku Postgres must be Standard+ tier (pgvector); the enable_extension "vector" migration fails otherwise.
• DWD scopes meetings.space.readonly + drive.readonly and the Meet API must be enabled (already done).
• Overnight 90-day backfill: heroku run:detached --size=performance-l "rake 'stacks:etl:backfill_meet_all[90]'"; nightly stacks:etl:sync_meet_all on a Performance dyno.

Known follow-ups (non-blocking, in the runbook)

Cross-source dedup (Drive backfill vs Meet API sync), semantic-search IN (...) scaling, baking the embedding model into the slug, and the intelligence layer (decisions/commitments/tasks/opportunities) as a separate spec.

Out of scope

Intelligence layer, connectors beyond Meet, multi-tenant/productization — all deliberately deferred.

🤖 Generated with Claude Code

[hhff]

🟠 Important (data loss) — Late-finalizing transcripts are permanently missed

The watermark advances to Time.current and the next Meet-API run filters start_time >= cursor. A meeting whose transcript is still generating when the run sees it gets stored empty, and is then permanently excluded from re-fetch. Fix: advance the cursor with a lookback (e.g. run_started_at - 2.days) or re-scan a trailing window so not-yet-final transcripts get re-pulled.

(automated self-review, code-review workflow)

[hhff]

🟠 Important (data loss) — Drive-sourced chunks get occurred_at=nil → excluded from date-scoped search

DriveSource#parse_segments sets started_at: nil, so every Drive (backfill) chunk has occurred_at = nil, and Search.filtered date_range predicate excludes them all. Since the 90-day backfill is Drive-based, date-bounded search misses the entire backfilled history. Fix: in DriveSource, fall back to the doc's created_time for segment/chunk occurred_at.

(automated self-review, code-review workflow)

[hhff]

🟠 Important — All ingestion paths share one SourceSync cursor (clobbering)

sync_meet, the Drive backfill, and every per-user sweep run share SourceSync.for(:meet), overwriting each other's cursor['since'] and stats. A sweep that runs between incremental syncs moves the watermark to a wrong time → the next sync silently skips a window. list_sources also reports only the last writer. Fix: key SourceSync per path/user, or don't advance the shared cursor from multi-user/backfill runs (they already pass an explicit since).

(automated self-review, code-review workflow)

[hhff]

🟠 Important (ops) — sweep_all_users! reports success even when every user failed

mark_as_success runs unconditionally after the loop. If org-wide auth breaks, every per-user run is swallowed into failed but the SystemTask is green — a total ingestion outage goes undetected. Fix: mark_as_error (or raise) when ok.zero? && emails.any?, and surface the failed count on the task.

(automated self-review, code-review workflow)

[hhff]

🟠 Important (backfill robustness) — An invalid attendee email aborts the meeting's ingest

resolve_email → create_or_find_by! under validates :email, format: Devise.email_regexp. A Calendar attendee/speaker email that fails the regexp raises RecordInvalid inside the un-rescued ingest transaction, killing that meeting (and, in a sweep, the user's remaining meetings). Fix: validate/normalize the email and skip (or null-contact) when it can't be a Contact, rather than raising.

(automated self-review, code-review workflow)

[hhff]

🟡 Correctness — contact filter with an unknown email returns wrong results

When contact: is given but no Contact matches, c&.id is nil so the scope becomes WHERE speaker_contact_id IS NULL — returning unrelated unattributed chunks as if they were that person's. Fix: if a contact filter is supplied but resolves to nil, return empty (scope.none).

(automated self-review, code-review workflow)

[hhff]

🟡 Correctness — No cross-source dedup: same meeting via Drive + Meet API = two documents

Drive keys on file.id, Meet API on cr.name → the same meeting ingested by both sweeps becomes two corpus-eligible Documents, and the agent sees/cites it twice. The spec calls for reconciling onto one row. Fix: unify via the transcript's Drive doc id (the Meet API transcript exposes docs_destination.document) as a shared dedup key.

(automated self-review, code-review workflow)

[hhff]

🟡 Correctness — Drive title-match can cross-assign attendees between same-title meetings

The Drive path matches the first Calendar event with an equal normalized title in the ±2h window. Two different 'Weekly Sync' meetings in that window → attendees from the wrong one. Fix: when matching by title, also pick the closest start time, and/or skip if more than one event shares the title.

(automated self-review, code-review workflow)

[hhff]

🟡 Correctness (non-migrate bootstrap) — schema.rb omits chunks.content_tsv → keyword/hybrid search breaks on db:schema:load

The generated content_tsv column + GIN index aren't dumpable, so they're omitted from schema.rb and only recreated in test_helper.rb. Prod (migrations) is fine, but any db:setup/db:reset/review-app built via db:schema:load lacks the column → Chunk.keyword_search 500s. Fix: recreate it in an idempotent post-load path (a seeds/initializer or structure.sql), or document that schema:load isn't supported.

(automated self-review, code-review workflow)

[hhff]

🔵 Minor (scaling) — Semantic search materializes all eligible chunk ids into an IN(...) list

The corpus wall is enforced by plucking every eligible chunk id and passing an unbounded owner_id IN (...), which defeats the HNSW index as the corpus grows. Fix: enforce eligibility via a JOIN to chunks/documents in SQL rather than a materialized id list.

(automated self-review, code-review workflow)

[hhff]

Self-review (code-review workflow, high effort — 39 agents, every finding independently verified)

The architecture, the exclusion/privacy wall (verified airtight across keyword/semantic/hybrid + all MCP tools), and MCP auth (X-Api-Key fail-closed) hold up. The 10 verified findings are all correctness/robustness, not structural — but 5 would bite the overnight org-wide backfill specifically, so I'd fix those before kicking it off:

Fix before the overnight backfill (🟠):

1. connector.rb — incremental cursor advances to now, so transcripts that finalize after a run are permanently missed (data loss).
2. chunker.rb/drive_source.rb — Drive chunks get occurred_at = nil, so date-scoped search misses the entire 90-day backfill.
3. connector.rb/meet.rb — all ingestion paths share one SourceSync cursor and clobber each other's watermark.
4. meet.rb — sweep_all_users! reports green even if every user failed (silent total-outage).
5. contact.rb — an invalid attendee email raises mid-transaction and aborts that meeting's (and, in a sweep, the user's remaining) ingest.

Correctness cleanups (🟡): contact-filter-with-unknown-email returns wrong rows (search.rb); no cross-source dedup → duplicate docs when Drive + API overlap (drive_source.rb); same-title Calendar match can cross-assign attendees (calendar_enricher.rb); schema.rb omits content_tsv so db:schema:load-built DBs break keyword search.

Scaling (🔵): semantic search materializes all eligible chunk ids into an IN (...) list (search.rb).

Details + suggested fixes are in the inline comments. None block merge of the foundation; #1–#5 are worth landing before the production backfill run.