Skip to content

docs: add requests table S3 archival design doc (#175)#206

Open
Achyuth0506 wants to merge 1 commit into
saayam-for-all:mainfrom
Achyuth0506:docs/175-requests-archival-design
Open

docs: add requests table S3 archival design doc (#175)#206
Achyuth0506 wants to merge 1 commit into
saayam-for-all:mainfrom
Achyuth0506:docs/175-requests-archival-design

Conversation

@Achyuth0506

@Achyuth0506 Achyuth0506 commented Jul 7, 2026

Copy link
Copy Markdown

Design document for Issue #175 - periodic archival of the request table to AWS S3.

What's included

  • Problem statement and goals covering three consumers - internal analytics, RAG agent, and third-party vendors
  • Three-stage pipeline architecture - Collection, Scrubbing, and Serving with separate S3 staging areas
  • Schema mapping from actual DDL (virginia_dev_saayam_rdbms.request) to Parquet with logical-to-real column translation table
  • PII handling with explicit per-column decisions based on real schema - HMAC hashing, city/state coarsening, boolean derivations for doc and audio fields
  • Dimension/lookup table resolution strategy for six FK columns
  • Four-layer free-text scrubbing approach with honest residual risk acknowledgment
  • Vendor serving design with three access model options and rationale for Option C
  • Four alternatives considered with tradeoffs
  • Cost estimate at current and 10x scale with per-service breakdown
  • Six-phase rollout plan with rollback strategy
  • 18 open questions tracked with owners and status

Pending Inputs

  • Row count and growth rate
  • Read replica availability
  • DB version (RDS vs Aurora, major version)
  • last_update_date UPDATE trigger confirmation
  • to_public column confirmation

Acceptance Criteria Checklist

  • Readable by someone outside data eng
  • Non-obvious tradeoffs surfaced
  • PII handling has explicit per-column decisions
  • Cost estimate at current and 10x scale included
  • At least two alternatives documented with rationale
  • Implementation plan concrete enough to estimate in story points
  • Vendor access model confirmed at design review
  • Design reviewed and signed off by data eng team

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant