Compliance PoC — PII redaction & data retention (#275)#279
Draft
illeatmyhat wants to merge 14 commits into
Draft
Compliance PoC — PII redaction & data retention (#275)#279illeatmyhat wants to merge 14 commits into
illeatmyhat wants to merge 14 commits into
Conversation
Proof-of-concept for the two halves of the Compliance epic, implemented in the altk_evolve package. PII redaction (CPEX): - altk_evolve/pii: pluggable PIIRedactor with a CPEX regex backend (cpex-pii-filter / PIIDetectorRust). 'semantic' mode is a documented seam (CPEX has no embedding detector). Lazy-imported and config-gated, so the [pii] extra is optional and disabled flows are untouched. - Applied at the BaseEntityBackend.update_entities choke-point, so PII is scrubbed before it is persisted OR used as a conflict-search query — covers every write path (CLI, MCP, API, Phoenix sync). Wired in by EvolveClient from EvolveConfig.pii (NullRedactor default). Data retention: - altk_evolve/retention: RetentionPolicy/RetentionRule schema (YAML/JSON) and a backend-agnostic RetentionEngine. Supports age-based and unused-based flag/delete, plus session (trajectory) retention with provenance cascade delete of derived memories (metadata.source_task_id == trace_id). - EvolveClient.record_access stamps metadata.last_accessed for the unused signal. - CLI: `evolve retention run --policy <file> [--apply]` (dry run by default). Tests: 20 unit tests (pure logic + real-CPEX end-to-end redaction at the write choke-point + retention engine over an in-memory client). mypy + ruff clean. Co-Authored-By: Claude Opus 4.8 <[email protected]>
examples/pii_redaction_demo.py writes made-up memories full of fake PII (a fictional persona name, email, phone, SSN, card, private IP) into a throwaway Evolve namespace with redaction enabled, then reads them back to show every PII item was replaced with inert filler at the write choke-point — while non-PII text survives verbatim. Also expose custom_patterns / whitelist_patterns on PIIConfig so callers can catch entities the built-in regex flags miss (e.g. specific names, since the CPEX regex backend has no NER). The demo uses custom_patterns for the persona. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Adds the package's PII redaction to the evolve-lite plugin world so memory is scrubbed before it lands in .evolve/. - plugin-source/lib/pii.py: stdlib-only redactor (the plugin lib forbids third-party deps since these scripts run in the host's Python). CPEX's cpex-pii-filter is imported lazily with a NullRedactor fallback + stderr warning when absent; 'semantic' mode is the same documented seam. Config via a `pii:` block in evolve.config.yaml. - learn/save_entities.py: build a redactor from load_config() and scrub each entity before dedup/persist. No-op unless pii.enabled. - Rendered to all variants via `just compile-plugins`; the shared lib ships to all four, the wired save path ships to claw-code (claude/codex/bob use native memory / exclude the learn skill). - mypy: register the plugin 'pii' module in ignore_missing_imports (resolved at runtime via sys.path, matching the existing config/audit/entity_io entries). Co-Authored-By: Claude Opus 4.8 <[email protected]>
…275) - examples/retention_demo.py: seeds a namespace (stale guideline, fresh one, old session + derived memory), backdates ages on disk, then runs a policy to show unused-flag, age-delete, and provenance cascade-delete (dry-run + apply). - DEMO.md: self-contained walkthrough for presenting the PoC — setup, both demo commands with captured output, real-usage config/CLI snippets, talking points, and honest limitations. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Swap the PII demo's mask text from [INERT] to [REDACTED] (also the PIIConfig default) and refresh the captured output in DEMO.md to match. Co-Authored-By: Claude Opus 4.8 <[email protected]>
examples/pii_benchmark.py scores the CPEX redactor against a deterministic labeled gold set (text with exact PII spans) and reports recall (the "did we remove it" number), precision (over-redaction), F1, per-entity recall, and a record-level leak rate. Accepts --data PATH for a real JSONL corpus (ai4privacy / Presidio) instead of the built-in synthetic set. Results on the built-in set: structured PII (email/phone/ssn/card/ip) recall 1.00 at precision 1.00; names/addresses leak (no NER) → 0.68 overall, lifting to 0.91 with custom name patterns. DEMO.md gains a benchmark section. Co-Authored-By: Claude Opus 4.8 <[email protected]>
pii_benchmark.py gains --dataset to stream an ai4privacy-style Hugging Face set
(privacy_mask is already {value,start,end,label}) and a "supported-types recall"
metric (of the types CPEX targets, how much it removes) alongside overall
recall. New [bench] extra ships `datasets`.
Real-data finding (ai4privacy/pii-masking-200k, 1000 en records): email/IP hold
at 1.00, but phone/SSN drop to ~0.45 and credit_card to 0.03 on varied/non-Luhn
formats — vs 1.00 on the clean synthetic set. Overall recall 0.11 (CPEX targets
~5 of ~42 types), supported-types recall 0.71, precision 1.00. DEMO.md documents
this with the format-sensitivity caveat.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
…emantic (#275) Implements the semantic seam with IBM READI (readi-privacy): ReadiSemanticRedactor wraps READIAnalyzer.detect (transformer spaCy NER + dictionary ensemble) and masks the detected spans (READI only detects; no built-in redactor). Package import is validated at construction so a missing [readi] extra degrades to a NullRedactor; the ~450MB en_core_web_trf model loads lazily on first detect. New [readi] extra (pins spacy-curated-transformers, which en_core_web_trf needs) + risk_assessment mypy override + readi_detection_type config. Benchmark gains --mode regex|semantic|both. On 150 ai4privacy records, semantic lifts overall recall 0.12 -> 0.45 at precision 1.00 — names go 0.00 -> ~0.92 (firstname 0.93, lastname 0.92, middlename 1.00) plus locations/dates/urls — the free-form PII regex can't catch. DEMO.md documents the regex-vs-semantic table. Co-Authored-By: Claude Opus 4.8 <[email protected]>
mode=semantic now selects among the extractors READI already ships instead of a
hand-built pipeline. New config: readi_extractor (default|spacy|hf|presidio),
readi_model, readi_language.
- default -> READI's DetectionType.PII (spaCy en_core_web_trf + identifiers), English.
- spacy/hf -> swap the NER model (e.g. ja_core_news_trf, or a multilingual
pipeline('ner') HF model) via a single SpacyEntityExtractor/HFEntityExtractor.
- presidio -> Microsoft Presidio (MIT) via READI's PresidioEntityExtractor.
Each path passes ONE READI-provided extractor into DetectionType.CUSTOM; no
custom aggregator tuning. Verified default + presidio redact name/email/phone
end-to-end. DEMO.md documents the options.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
- pii_benchmark.py: add a WikiANN loader (tokens + BIO -> PER/ORG/LOC char spans; CJK-safe) and --dataset-format, --dataset-config, plus --readi-extractor / --readi-model / --readi-language passthrough so any extractor+model can be benchmarked. Enables scoring the semantic backend on languages ai4privacy lacks (e.g. Japanese via unimelb-nlp/wikiann config=ja). - DEMO.md: add a "Reviewing this PoC (for agents)" section encoding the verification-first, reproduce-and-adversarially-spot-check approach used to build it. Co-Authored-By: Claude Opus 4.8 <[email protected]>
DEMO.md gains a full-sized results section: - English ai4privacy 43,501 records (~65 min): regex 0.12 vs semantic 0.47 overall recall at ~1.0 precision; names 0.00 -> ~0.9+. - Japanese WikiANN 10,000 records: the English NER scores 0.00, a language- matched ja_core_news_trf scores 0.87 overall (person 0.93) — swapped in via the pluggable readi_extractor/readi_model config. Documents the WikiANN precision caveat and the hybrid takeaway. Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
#275) - DEMO.md: add a learned-guideline before/after (PII removed, reusable knowledge kept); a retention "why each memory was kept/flagged/removed" decision table; and a scenarios table (data minimization, right-to-be-forgotten cascade, memory hygiene, human-in-the-loop). Name the English model (en_core_web_trf) in the semantic columns. - retention_demo.py: emit a per-memory decision + derived "why" (age / unused / provenance cascade), including kept memories. - pii_benchmark.py: map openpii-1.5m labels (TELEPHONENUM->phone, SOCIALNUM->ssn) for the Japanese PII run. Co-Authored-By: Claude Opus 4.8 <[email protected]>
cpex-pii-filter is a Rust regex engine and returns BYTE offsets. On multi-byte text (e.g. Japanese) those don't align with Python character indexing, so any consumer using detect() spans (like the benchmark's span-overlap scorer) breaks — which made regex look like ~0.01 recall on Japanese even though redaction worked. Convert byte->char offsets in detect(); redact() is unchanged (mask() runs Rust-side with byte-consistent offsets). ASCII is a no-op. Adds a multi-byte regression test. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Replace the WikiANN Japanese table with a real-PII 3-way on ai4privacy/pii-masking-openpii-1.5m (ja, 2000 records): regex 0.03 vs English NER 0.15 vs Japanese ja_core_news_trf 0.92 overall recall, all at ~0.99 precision (JA 0.93). Regex/English fail on Japanese-embedded PII; the language-matched model dominates. Notes the byte->char offset fix caught here. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What it does
Two enterprise asks from #275, in the
altk_evolvepackage (PII redaction also mirrored into the evolve-lite plugin):BaseEntityBackend.update_entities), covering every write path (CLI / MCP / API / Phoenix sync). Two pluggable backends:regex(CPEX) andsemantic(IBM READI NER — with a pluggable extractor/model: spaCy / HF / Microsoft Presidio — defaulting to spaCy-English).evolve retention run --policy <file> [--apply](dry-run by default).See it in ~2 minutes
A learned guideline with PII removed — the reusable knowledge is kept, only the PII goes:
How well does it actually remove PII? (measured)
English — ai4privacy, 43,501 records — per-entity recall:
en_core_web_trf)Japanese — ai4privacy openpii-1.5m, 2,000 records — regex and the English NER largely fail on Japanese-embedded PII; a language-matched model (one config line) fixes it:
en_core_web_trf)ja_core_news_trf)Takeaways: regex is high-precision on well-formed structured PII but blind to names and to non-Western formats; semantic NER adds names/locations (~4× recall on English) but is language-bound — a language-matched (or multilingual) model is essential for non-English. Structured PII (cards, phones) is the shared weak spot → a hybrid (regex + language-matched NER) is the production shape. (Benchmarking also caught a real byte-vs-char offset bug in the regex detector — now fixed.)
More
Full walkthrough, all commands, per-entity tables, retention scenarios & "why", limitations, and a reviewer guide:
DEMO.md.🤖 Generated with Claude Code