Skip to content

v1 -> v2 MailingList table migration - #234

Merged
PatrickMTonne merged 4 commits into
developfrom
task/v1-to-v2-mailinglist-migration-command
May 12, 2026
Merged

v1 -> v2 MailingList table migration#234
PatrickMTonne merged 4 commits into
developfrom
task/v1-to-v2-mailinglist-migration-command

Conversation

@PatrickMTonne

@PatrickMTonne PatrickMTonne commented May 5, 2026

Copy link
Copy Markdown
Contributor

Context

Adds a one-time Django management command, copy_to_dynamodb, that migrates v1 MailingList records from PostgreSQL into the v2 Course Emailer's DynamoDB table so existing mailing lists continue to work after the v1 → v2 cutover. The command copies every MailingList row modified within a configurable window (default: the last 2 years).

Summary

  • New command mailing_list/management/commands/copy_to_dynamodb.py with flags:
    • --table-name (required) — target DynamoDB table
    • --dry-run — skip put_item calls
    • --overwrite — replace existing items instead of skipping
    • --max-age-years N — only copy lists modified within N years (default 2; 0 disables)
    • --output-file PATH — per-list CSV audit trail
  • Idempotent by default via ConditionExpression="attribute_not_exists(PK)"; existing v2 items are preserved and counted as skipped_existing
  • Non-zero exit code when any errors are recorded
  • Field mapping from v1 to v2:
    • access_level=members/everyonevalid_senders=all_members, active=True
    • access_level=staffvalid_senders=staff_only, active=True
    • access_level=readonlyvalid_senders=all_members, active=False
    • course_settings.always_mail_staffinclude_all_staff (defaults to True when no CourseSettings row exists)
    • date_created/date_modified → ISO 8601 UTC created_at/updated_at
    • section_id included only when present; subject_prefix, sis_section_id, and name are omitted (v2's own sync_with_canvas backfills them on first course access and won't clobber migrated settings)
  • boto3 added as a direct dependency in pyproject.toml

Testing

  1. Dry-run against the v2 dev DynamoDB table with a CSV audit trail: ENV=dev DJANGO_SETTINGS_MODULE=lti_emailer.settings.local uv run python manage.py copy_to_dynamodb --table-name <dev-table> --dry-run --output-file dry-run.csv
  2. Real run against dev without --dry-run; inspect the DynamoDB table and confirm new items have correct PK/SK, active, valid_senders, include_all_staff, ISO UTC timestamps, and no subject_prefix/sis_section_id/name
  3. Load a migrated course in the v2 frontend and confirm the mailing list appears with correct access-level settings.

Notes

  • v1 PROD shows 16,217 modified email lists from the past two years:
  SELECT COUNT(*) FROM ml_mailing_list
  WHERE date_modified >= NOW() - INTERVAL '2 years';

@PatrickMTonne
PatrickMTonne marked this pull request as ready for review May 7, 2026 14:01
@andresmauro17
andresmauro17 requested review from andresmauro17 and removed request for andresmauro17 May 12, 2026 18:43

@andresmauro17 andresmauro17 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was testing this locally using the CSV output file and the DynamoDB table.

All 31 existing records were copied correctly.

It should be fine if we can test all the records in the DB, and the code looks like it will work as expected.

Good implementation @PatrickMTonne 👍

@PatrickMTonne
PatrickMTonne merged commit 0f52567 into develop May 12, 2026
2 checks passed
@PatrickMTonne
PatrickMTonne deleted the task/v1-to-v2-mailinglist-migration-command branch May 12, 2026 19:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants