[DSGVO] 1. Erasure / anonymization workflow — Art. 17, Art. 5(1)(e)

[thomiel]

Currently there is only hard C3sMember.delete_by_id, and
membership_member_delete.py carries a TODO admitting it is not restricted
to non-accepted applications. Needed:

• Restrict hard deletion to non-accepted applications; accepted members must
  go through membership loss (deleting them would violate GenG
  record-keeping).
• An anonymize() operation that nulls PII while keeping the statutory
  ledger data (membership number, shares, dues aggregates).
• A retention-review view listing records whose clocks have expired: lost
  members past retention, and stale never-completed applications.
• The same retention pass for artifacts outside the DB: generated PDFs in
  invoices/ and certificate files.

[thomiel]

Implementation plan: erasure workflow + delete-view restriction

Two parts. Part A is a small, self-contained fix and should land first.
Part B builds the anonymization/retention machinery.

Part A — Restrict hard deletion (fixes the TODO)

Status: implemented (2026-07-13). In addition to the plan below, the
bulk range deletion delete_afms in membership_acquisition.py turned out
to be equally unguarded and now skips accepted members as well. The
delete icon was removed entirely from membership_members.pt because that
page lists only accepted members; membership_member_detail.pt already hid
the delete button for accepted members and needed no change.

Files:

• c3smembership/presentation/views/membership_member_delete.py
• c3smembership/presentation/templates/pages/membership_members.pt
• c3smembership/presentation/templates/pages/membership_member_detail.pt
• c3smembership/presentation/templates/pages/membership_acquisition.pt

Changes:

1. In delete_entry:
   • Guard against a nonexistent member id: C3sMember.get_by_id returns
     None and the current code would crash on member.id. Flash an error
     and redirect instead.
   • Refuse deletion when member.membership_accepted is true (or a
     membership_number exists). Flash a message pointing staff to the
     membership-loss process and redirect without deleting. Log the refused
     attempt.
   • Keep hard deletion working for non-accepted applications (duplicates,
     test entries) — that is its legitimate purpose.
2. In the three templates, hide or disable the delete button/link for rows
   where membership is accepted, so the restriction is visible in the UI and
   not only enforced server-side.
3. Remove the TODO comment from the module docstring once enforced.

Tests (new file
c3smembership/presentation/views/tests/test_membership_member_delete.py,
following the existing view-test pattern):

• deleting a non-accepted application succeeds and logs,
• deleting an accepted member is refused, record still present,
• nonexistent member id does not crash and flashes an error.

Part B — Anonymization and retention review

B1. Schema migration (alembic)

One new revision (pattern: alembic/versions/d4c59e012aa_add_member_privacy_consent.py):

• members.anonymized — DateTime, nullable; set when the record is
  anonymized (also serves as the "is anonymized" flag).
• Make members.date_of_birth nullable (currently nullable=False), so
  anonymization can null it instead of writing a fake sentinel date.

B2. Model: C3sMember.anonymize()

In c3smembership/data/model/base/c3smember.py, an instance method that:

Clears (PII): firstname, lastname, email, _password,
address1, address2, postcode, city, date_of_birth, locale,
accountant_comment, name_of_colsoc, court_of_law,
registration_number, certificate_token, mtype_confirm_token,
email_confirm_token.

Rewrites (unique constraints): email_confirm_code →
'ANON-{id}' (column is unique and used as reference everywhere; a
deterministic non-personal value keeps integrity).

Keeps (statutory ledger / statistics): id, membership_number,
membership_type, membership_accepted, membership_date,
membership_loss_date, membership_loss_type, is_legalentity,
country, shares and dues relationships, submission/signature/payment
dates.

Sets: anonymized = datetime.now().

Keeping country (and only country) preserves the statistics views;
postcode + date of birth are dropped because in combination they can
re-identify.

B3. Business layer: retention policy and candidates

New module c3smembership/business/data_protection.py:

• Policy constants (with a comment citing the legal basis):
  • MEMBER_RETENTION_YEARS = 10 — HGB §257 / AO §147, counted from the
    end of the year of membership_loss_date,
  • APPLICATION_RETENTION_MONTHS = 18 — stale, never-completed
    applications (Art. 5(1)(e); no legal basis to keep them longer).
• get_erasure_candidates() returning two lists:
  1. Lost members past retention: membership_loss_date set,
     retention expired, anonymized is null.
  2. Stale applications: membership_accepted falsy, no membership
     number, date_of_submission older than the application retention,
     anonymized is null. Duplicates (is_duplicate) are included.
• anonymize_member(member_id):
  1. verify the member is actually a candidate (defense in depth —
     refuse otherwise),
  2. delete the member's invoice PDFs on disk (via the dues invoice
     repository, which knows the file naming) — the PDFs contain name and
     address,
  3. call member.anonymize(),
  4. LOG.info who anonymized which member id and when (Art. 5(2)).

Query implementations go into
c3smembership/data/repository/member_repository.py following the existing
repository pattern.

B4. Presentation: retention review page

• New view module
  c3smembership/presentation/views/data_protection.py, permission
  'manage':
  • GET /data_protection — render the two candidate lists (lost members
    past retention; stale applications) with per-row and select-all
    "Anonymize" actions and a confirmation step, mirroring the
    deletion_confirmed pattern of the delete view.
  • POST action calling anonymize_member per selected id, flashing a
    summary (n anonymized, m refused).
• New template presentation/templates/pages/data_protection.pt.
• Route/config registration in a new
  presentation/configuration/data_protection_config.py, wired up like
  membership_application_config.py; add a link in the toolbox/dashboard
  template.

Staff-triggered (human in the loop) rather than a cron job for the first
iteration; a scheduled reminder can be added later once the workflow is
trusted.

B5. UI safety in existing views

• Member detail / edit / listing templates: show an "anonymized" badge and
  hide edit/certificate/dues actions for anonymized records.
• Exclude anonymized records from the CSV export and search/autocomplete
  (they contain no PII anymore, but empty rows are noise).

B6. Tests

• Model: anonymize() clears exactly the PII fields, keeps ledger fields,
  preserves email_confirm_code uniqueness, sets anonymized.
• Repository: candidate queries — boundary dates (retention exactly
  expired / one day short), already-anonymized excluded, accepted members
  never appear among stale applications.
• Business: anonymize_member refuses non-candidates; invoice files
  removed.
• View: permission required; confirmation required; flash summaries.

B7. Suggested sequencing

1. Part A (delete restriction) — small PR, immediately closes the
   "deleting what must be kept" hole.
2. B1 + B2 + tests (schema + model).
3. B3 + tests (policy + business logic).
4. B4 + B5 + tests (UI).

Retention periods in B3 must be confirmed with the coop's data protection
officer / tax advisor before the first productive anonymization run.

[thomiel]

implemented stage 1/4: "Restrict hard deletion to non-accepted applications; accepted members must go through membership loss (deleting them would violate GenG record-keeping)." see 3fe4333

feature branch: https://github.com/C3S/c3sMembership/tree/feature/dsgvo-enhancements

[thomiel]

Part B implementation report

Part B (anonymization and retention review) of the erasure workflow
planned in DSGVO-c3sMembership.md was implemented and verified on
2026-07-17 as the three remaining commits of the ref #7 series
(2/4–4/4). Part A (delete-view restriction) had already landed as
commit 1/4. AI used: Claude Fable 5. All three commits are raw and
unreviewed AI outputs. Testers and reviewers welcome!

Glossary: PII (Personally Identifiable Information) is the common
English shorthand for personenbezogene Daten (Art. 4(1) DSGVO) — any
data identifying a person on its own (name, email, address) or in
combination (e.g. postcode + date of birth).

What was built

Schema — commit 2/4, fc6cea3

• Alembic revision e7a1f3c9d2b4 adds members.anonymized
  (timestamp, doubling as the "is anonymized" flag) and makes
  date_of_birth nullable.
• C3sMember.anonymize() clears all PII, rewrites the unique
  reference code to ANON-{id} and keeps the statutory ledger data
  (membership number, type, dates, shares/dues relations, country).

Business layer — commit 3/4, 65dafa0

• c3smembership/business/data_protection.py with the policy
  constants: 10 years counted from the end of the year of the
  membership loss (HGB §257, AO §147), 18 months for stale
  never-completed applications (Art. 5(1)(e) DSGVO).
• Erasure candidate queries in the member repository.
• DataProtection.anonymize_member() verifies candidacy (defense in
  depth), deletes the member's archived invoice PDFs, clears PII from
  the dues_invoices rows and logs who anonymized which dataset
  (Art. 5(2)).

User interface — commit 4/4, 839d2e9

• Retention review page at /data_protection (linked from the
  toolbox) listing both candidate groups with select-all and the same
  confirmation-dialog pattern as the delete view (fail-safe without
  JavaScript).
• Anonymized records get a badge in the members list and detail page,
  lose their edit/certificate/mail actions, are refused by the edit
  view and are excluded from CSV export, autocomplete/search and the
  acquisition dashboard.

Deviations from and extensions to the plan

• Anonymization also clears the personal data columns of the member's
  invoice records (dues_invoices.email and token) — the invoices
  table carries PII the original plan missed — and deletes the
  archived invoice PDFs via the new
  DuesInvoiceArchiving.remove_archived_invoices. Invoices are found
  by member_id (new DuesInvoiceRepository.get_by_member_id), which
  also works for applications without a membership number.
• Anonymized applications are excluded from the acquisition dashboard
  listing (nonmember_listing) — they would otherwise remain visible
  forever as empty rows.
• The edit view refuses anonymized datasets server-side (the edit form
  cannot handle the nulled date_of_birth), in addition to hiding the
  edit buttons.
• The stale-application cut-off is clamped to the first day of the
  month, erring a few days on the side of keeping data.
• Two test files were named test_data_protection_business.py /
  test_data_protection_presentation.py because pytest cannot handle
  duplicate basenames across the non-package test directories.

Verification

Beyond the automated tests (296 unit tests pass), the flow was driven
end-to-end against the live development server:

• The review page listed exactly the right candidates from the dev
  database (the member lost in 2015 plus three 2022 applications; the
  active member absent).
• An unconfirmed POST was refused; a confirmed POST anonymized the
  lost member and refused the active one ("1 anonymized / 1 refused"
  flash messages). The database row afterwards read ANON-2 with the
  ledger data intact.
• Probes: the detail page shows the badge without an edit button,
  /edit/2 redirects with a flash message, the people autocomplete no
  longer crashes on the nulled lastname, and the CSV export omitted
  the anonymized member.
• The development database was restored from a backup afterwards.

The only failing integration tests (test_regenerate_pdf,
test_email_confirmation, webdriver) fail identically on the clean
tree — environment issues, not regressions.

Notes and open points

• Broken alembic stamp trap: running the integration suite
  (test_initialization.py) stamps alembic_version to head via
  alembic.ini without applying migrations. The development
  database was repaired (re-stamped to the previous revision, then
  alembic upgrade head applied properly). This will bite again
  whenever the integration tests run against a not-yet-migrated
  database.
• Open: the retention periods (10 years / 18 months) must be
  confirmed with the coop's data protection officer / tax advisor
  before the first productive anonymization run.