feat(validators): central registry + sync to Paramify

.gitignore

@@ -17,6 +17,9 @@ venv/
 .env.*
 !.env.example
 
+# --- Customer-side sync state (validator key -> Paramify id); never commit ---
+.paramify/
+
 # --- Generated collect output (runner writes evidence/run-<timestamp>/) ---
 /evidence/
 

CHANGELOG.md

@@ -10,6 +10,27 @@ schemas and the `paramify` CLI — not the internal code.
 
 ## [Unreleased]
 
+### Added
+
+- Central `validators/` registry: validators are now first-class, deduplicated
+  objects (`validators/<category>/<key>.yaml`, one file each), validated against
+  the new `framework/schemas/validator_schema.json`. Each validator carries the
+  Paramify fields as native YAML and owns its fetcher link via an
+  `evidence_sets` list of `reference_id`s, so a validator shared across fetchers
+  is one file — never a copy. See [`docs/validators_design.md`](docs/validators_design.md).
+- Validator sync (`uploaders/paramify_validators/`, `paramify validators sync`,
+  and `paramify upload --with-validators`): pushes registry validators to
+  Paramify (`POST /validators`) and CONNECTs them to evidence sets. **Create-or-skip
+  by default** — a customer's tuned validator is never overwritten unless
+  `--update` is passed; `--dry-run` previews. Per-instance ids live in a
+  gitignored lock, not the shared registry.
+
+### Removed
+
+- The inline `validators` block on `fetcher.yaml` (shipped optional in
+  0.2.0-beta, populated by no fetcher). Validators moved to the registry above;
+  `fetcher.yaml` keeps its `evidence_set` identity and `ksis`.
+
 ## [0.2.1-beta] - 2026-07-10
 
 ### Changed

docs/fetcher_contract.md

@@ -40,7 +40,7 @@ Every fetcher ships a `fetcher.yaml` in its directory. The schema is enforced; s
 | `depends_on` | array | Fetcher names this one depends on (not yet honored by the runner) |
 | `evidence_set` | object | Paramify evidence-set identity: `{reference_id, name, instructions?}`. Carried into envelope metadata and used by the uploader to get-or-create the set. |
 | `ksis` | array | FedRAMP 20x KSIs this fetcher's evidence speaks to (1+). Intrinsic to the fetcher; per-customer control mappings stay Paramify-side. |
-| `validators` | array | Regex checks over the evidence payload that show the control is being implemented. Each entry: `{id, regex, proves?, failure_modes?}` (`id` + `regex` required); each regex matches the whole payload. |
+| _validators_ | — | Not in `fetcher.yaml`. Validators are first-class objects in the central `validators/` registry, each linked to a fetcher by its `evidence_set.reference_id`. See [`validators_design.md`](validators_design.md). |
 
 ---
 

docs/validators_design.md

@@ -0,0 +1,210 @@
+# Validators — Design
+
+**Status:** Schema + registry defined (v0.x, 2026-07-14).
+`framework/schemas/validator_schema.json` + the `validators/` registry;
+`validators/_template/validator.yaml` is the starting point.
+**Date:** 2026-07-14
+**Solves:** validators had only a thin, unused inline sketch in `fetcher.yaml`
+(`{id, regex, proves?, failure_modes?}`, populated by zero fetchers) that could
+not represent the real Paramify validator, and — being per-fetcher — had no way
+to express a validator shared by several fetchers without copying it.
+
+---
+
+## What a validator is
+
+A validator is a check over a fetcher's collected evidence that asserts the
+control is **being implemented** — a collection-side assertion, not a
+per-customer compliance judgment. Two kinds:
+
+- **AUTOMATED** — a `regex` over the evidence envelope whose capture groups are
+  compared by structured `validation_rules` (e.g. "the encrypted count equals
+  the total count").
+- **Attestation / manual** — no regex; the check is expressed as
+  `attestation_rules`.
+
+The fields mirror Paramify's own validator (`name`, `type`, `statement`,
+`regex`, `rules_summary`, `validationRules_json`, `attestationRules_json`,
+`evidence_sets`), stored as native YAML rather than stringified JSON. See the
+field reference in `framework/schemas/validator_schema.json`.
+
+---
+
+## The model: a deduplicated registry, linked on the validator
+
+**Decision: validators are first-class objects in a central `validators/`
+registry — one file per validator — and each validator names every evidence set
+it applies to. They are NOT stored inline in `fetcher.yaml`.**
+
+```
+validators/
+  _template/validator.yaml
+  aws/alb_encryption_in_transit.yaml
+  aws/...
+  okta/...
+```
+
+Each file:
+
+```yaml
+key: alb_encryption_in_transit          # stable id == basename
+name: ALB Encryption In Transit
+type: AUTOMATED
+role: configuration
+statement: Ensures that all application load balancers are encrypting data in transit.
+regex: '"alb_total":\s*(\d+)[\s\S]*?"alb_encrypted":\s*(\d+)'
+rules_summary: MATCH_GROUP[1] EQUALS MATCH_GROUP[2]
+validation_rules:
+  - regexOperation: { type: MATCH_GROUP, groupNumber: 1 }
+    criteria: EQUALS
+    value: { type: MATCH_GROUP, groupNumber: 2 }
+attestation_rules: []
+evidence_sets:                          # <- the link lives HERE
+  - EVD-LB-ENC-STATUS
+  - EVD-LB-ENC-STATUS-MANUAL
+```
+
+### Why the link lives on the validator
+
+The central requirement: **a validator shared by several fetchers must not be
+copied.** Because the fetcher→validator relationship is many-to-many (a fetcher
+has many validators; a validator can serve many fetchers), putting the link on
+the *validator* — a single `evidence_sets` list — means a shared validator is
+one file that simply lists more sets. Putting it on the fetcher instead would
+force either a copy per fetcher or a second layer of references. This is also
+exactly how Paramify models it: the validator owns its evidence-set list (in the
+API export, one validator row carries
+`Load Balancer Encryption Status | Non-automated Load Balancer Encryption Status`).
+
+**A fetcher's validators are a reverse lookup:** every registry file whose
+`evidence_sets` contains that fetcher's `evidence_set.reference_id`. A generator
+or CLI does the walk; nothing verbose lands in `fetcher.yaml`.
+
+### Why a sidecar registry rather than inline
+
+Validators carry a regex, a human `rules_summary`, and a structured
+`validation_rules`/`attestation_rules` block. Inlining all of that in
+`fetcher.yaml` would bury the fetcher's identity under rule blobs and — fatally
+— could not dedupe a shared validator. The old inline `validators` block is
+removed from `fetcher_schema.json`; `fetcher.yaml` keeps only its `evidence_set`
+identity and `ksis`.
+
+---
+
+## Linking by `reference_id`
+
+`evidence_sets` entries are evidence-set **`reference_id`s** (e.g.
+`EVD-LB-ENC-STATUS`), not display names. `reference_id` is the stable
+idempotency key the uploader already get-or-creates sets by; display names
+drift. A future importer that ingests a Paramify validator export (which lists
+sets by name) maps name → `reference_id` via the fetcher registry.
+
+Some `evidence_sets` entries may name a **manual / non-automated** set that has
+no fetcher in this repo (the "Non-automated …" sibling of an automated set).
+That is expected: the registry is validator-centric, so it can reference sets
+that live only Paramify-side.
+
+---
+
+## Cardinality (recap of the confirmed evidence model)
+
+- **1 fetcher = 1 (automated) evidence set** — unchanged. When one collection
+  would feed two sets, split the fetcher, don't fan one artifact out.
+- **Each evidence set carries exactly one `completeness` validator** (does the
+  evidence cover the full population? — often the minimum assessment scope) plus
+  **any number of `configuration` validators** (posture/config checks). `role:`
+  records which is which. This is why multiple validators per fetcher is the
+  norm, not the exception.
+
+---
+
+## Uploading & associating (Paramify REST API v0.6.0)
+
+The API confirms the sync path — it is the script-sync pattern (decision #122),
+just with `subjectType: VALIDATOR` instead of `SCRIPT`:
+
+- **Upsert the validator** — `POST /validators` (create) / `PATCH
+  /validators/{id}` (update). The body is a `oneOf` on `type`, and our registry
+  fields map 1:1:
+  - `AUTOMATED` → `{name, statement, type, regex, validationRules}` — our
+    `validation_rules` is the same object shape (`regexOperation`/`criteria`/`value`).
+  - `ATTESTATION` → `{name, statement, type, attestationRules}` — our
+    `attestation_rules` is the same shape.
+- **Associate to an evidence set** — `POST /evidence/{evidenceId}/associate`
+  with `{associationType: CONNECT, subjectType: VALIDATOR, subjectId:
+  <validatorId>}`. Many-to-many falls out for free: one CONNECT per entry in the
+  validator's `evidence_sets`; `DISCONNECT` reverses it.
+- **The evidence set** is get-or-created by `reference_id` via the existing
+  evidence uploader (`get_or_create_evidence_set`). A manual/non-automated
+  sibling set that has no fetcher is created (`automated: false`) before CONNECT.
+
+This is scoped and manifest-driven: sync only the validators whose
+`evidence_sets` intersect the evidence sets the user's manifest actually
+produces.
+
+### These are templates — create-or-skip, never clobber
+
+The shipped validators are **templates, ~80% correct**. A customer tunes them to
+their environment (regex thresholds, `statement`, attestation questions) *inside
+Paramify*. The sync must never destroy that tuning, so:
+
+- **Create-or-skip.** Sync creates a validator only when the customer's instance
+  doesn't already have it. If it exists, the sync **does not PATCH it** by
+  default — the customer's tuned copy is left exactly as-is.
+- **Update is explicit and loud.** A `--update` flag can PATCH an existing
+  validator from the template (for the not-yet-tuned case), but only opt-in and
+  with a clear warning; `--dry-run` shows the exact calls first. Default runs
+  never update.
+- **Association is always safe.** CONNECT (`/evidence/{id}/associate`) only wires
+  the validator to a set; it never touches validator content, so the sync always
+  ensures the association even when it skips the update.
+
+### Behavior after a partial upload
+
+`upload --with-validators` runs the validator sync even when some evidence files
+in the run failed to upload, scoped to every evidence-set reference_id the run
+*produced*. This is deliberate: a validator attaches to an evidence *set*, which
+is get-or-created independently of whether every artifact uploaded, so a single
+failed target should not block wiring the set's validators. The combined exit
+code still reflects both stages. (If this proves too loose, the alternatives are
+to skip the sync unless the upload was fully OK, or to scope only to sets whose
+artifacts actually uploaded — both easy to add later.)
+
+### Per-instance ids live customer-side, not in the registry
+
+`GET /validators` filters only by `ids`/`type` — no name/reference filter (the
+gap script-sync hit). So a validator's Paramify id is **resolved at sync time and
+cached in customer-side state** (a gitignored lock, alongside the uploader
+config) — never written back into the shared registry file, whose only identity
+is `key`. First sync: list + match by `name` → adopt the existing id (no
+duplicate) or create. This mirrors evidence sets exactly: the shared stable key
+is `reference_id`, and the Paramify id is resolved per instance, never stored in
+the repo. (Known limit: if a customer *renames* a validator before its id is
+cached, a name-match can miss and create a duplicate — documented, and why the
+lock is written on first create.)
+
+---
+
+## What this pass does / does not do
+
+**Does:** define `validator_schema.json`, stand up the `validators/` registry
+with a template, remove the dead inline `validators` block from the fetcher
+schema, and document the model.
+
+**Does not (deferred):**
+
+- **No importer / backfill.** Converting a Paramify validator export into
+  registry files (matching `evidence_sets` → `reference_id`, minting `key`s) is
+  a separate tool. No real validators ship in this pass.
+- ~~**No sync tool yet.**~~ **Built.** `uploaders/paramify_validators/syncer.py`
+  (create-or-skip reconcile + client + customer-side lock) plus `paramify
+  validators sync` and `paramify upload --with-validators`
+  (`framework/api.sync_validators`, scoped by manifest or by the reference_ids a
+  run produced). This intentionally supersedes the "validator linkage stays
+  manual" scope of decision #122.
+- **No contract test yet.** A `validators/**/*.yaml` gate against
+  `validator_schema.json` (mirroring `tests/test_contracts.py`) should land with
+  the first real validators, so it doesn't assert on an empty set.
+- **`suggest-validator` skill** still frames Paramify as the validator's only
+  home; update it to point an accepted suggestion at this registry when the
+  authoring path is built.

framework/api.py

@@ -689,6 +689,80 @@ def _load_paramify_uploader(root: Path):
     return module
 
 
+def _load_paramify_validator_syncer(root: Path):
+    """Load the source-tree validator syncer without requiring uploaders/ to be packaged."""
+    path = Path(root) / "uploaders" / "paramify_validators" / "syncer.py"
+    if not path.exists():
+        raise RuntimeError(f"Paramify validator syncer not found at {path}")
+    spec = importlib.util.spec_from_file_location("paramify_validators_syncer", path)
+    if spec is None or spec.loader is None:
+        raise RuntimeError(f"Could not load Paramify validator syncer from {path}")
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module
+
+
+def reference_ids_from_run(run_dir) -> set:
+    """The evidence-set reference_ids present in a run directory's envelopes.
+
+    Lets `upload --with-validators` scope the sync to exactly the sets this run
+    produced, rather than a manifest.
+    """
+    refs: set = set()
+    run_path = Path(run_dir)
+    if not run_path.is_dir():
+        return refs
+    for p in sorted(run_path.glob("*.json")):
+        if p.name in ("_run_metadata.json", "upload_log.json"):
+            continue
+        try:
+            env = json.loads(p.read_text())
+        except (OSError, json.JSONDecodeError):
+            continue
+        if not isinstance(env, dict):
+            continue
+        meta = env.get("metadata")
+        if not isinstance(meta, dict):
+            continue
+        es = meta.get("evidence_set")
+        if isinstance(es, dict) and es.get("reference_id"):
+            refs.add(es["reference_id"])
+    return refs
+
+
+def sync_validators(
+    root: Path,
+    manifest_path: Optional[Path] = None,
+    config_path: Optional[Path] = None,
+    *,
+    reference_ids=None,
+    dry_run: bool = False,
+    update: bool = False,
+    lock_path: Optional[str] = None,
+    on_event: Optional[Callable[[dict], None]] = None,
+) -> dict:
+    """Sync registry validators to Paramify and associate them to evidence sets.
+
+    Scope precedence: explicit `reference_ids` > `manifest_path`'s fetchers'
+    sets > whole registry. Create-or-skip; associates on create only; `update`
+    opt-in patches existing. Fires sync_start / sync_validator / sync_complete.
+    Raises ValueError for setup errors; returns the syncer summary otherwise.
+    """
+    syncer = _load_paramify_validator_syncer(root)
+    validators = syncer.collect_validators(root, manifest_path, reference_ids)
+    config: dict = {}
+    if config_path:
+        config = yaml.safe_load(Path(config_path).read_text()) or {}
+    return syncer.sync_validators(
+        validators,
+        config=config,
+        dry_run=dry_run,
+        update=update,
+        lock_path=lock_path,
+        on_event=on_event,
+    )
+
+
 def upload_preflight(
     run_dir,
     root: Path,