Skip to content

feat: add optional Semantic Guard plugin for variable-level alignment… - #2

Open
Fengrru wants to merge 1 commit into
amair-lab:mainfrom
Fengrru:feature/semantic-guard
Open

feat: add optional Semantic Guard plugin for variable-level alignment…#2
Fengrru wants to merge 1 commit into
amair-lab:mainfrom
Fengrru:feature/semantic-guard

Conversation

@Fengrru

@Fengrru Fengrru commented Jul 26, 2026

Copy link
Copy Markdown

What

This adds a semantic guard that checks whether two LLM-generated principles actually mean the same thing by the same variable names, before PiEvo throws them into the same Bayesian pot.

Why

Say PiEvo is working on combustion. The Principle Agent generates two candidates:

"Metals contain phlogiston. When burned, phlogiston escapes. Mass remains constant."

"Metals combine with oxygen to form oxides. Mass increases."

Both mention mass_change. PiEvo's Bayesian update treats them as sharing an observational vocabulary, so it computes likelihoods as if they're talking about the same quantity. They're not. Phlogiston's mass_change means "weight of dephlogisticated metal in open air." Oxidation's means "weight of metal oxide in sealed vessel." Same name, different referent, different measurement protocol — the posterior gets corrupted.

This isn't an edge case. It's the Phlogiston → Oxidation paradigm shift, and it also occurs with Caloric → Kinetic (temperature as primitive measurement vs. derived quantity).

How

The hook sits at exactly one place: update_pievo_state(), between _extract_principles() and _handle_new_principles(). A new principle is checked against every existing one. If any shared variable name maps to a different referent, operational definition, or evidential status, that principle is removed from self.principles and never reaches the GP training or posterior update.

When guard is None (the default), nothing changes.

_extract_principles()
       │
       ▼
  [guard hook] ── blocked principles removed
       │
       ▼
_handle_new_principles()   ← only validated principles

Files

New — pievo/guard/ (6 modules, no new dependencies)

  • signature.py — data models
  • alignment.py — 3D alignment vector (referential, operational, evidential; EXACT / TOKEN / EMBEDDING)
  • policy.pyStrictPolicy, PermissivePolicy, ConservativePolicy
  • extractor.py — LLM extraction via PiEvo's existing util_client, plus ManualExtractor for ablation; 4 few-shot examples
  • guard.pySemanticGuard orchestrator
  • __init__.py — public API

Modified (+83 lines, both additive)

  • evolveflow.pyPiEvo.__init__ accepts optional guard parameter; hook in update_pievo_state()
  • main.py--enable-semantic-guard and --guard-policy CLI flags

Tests

tests/test_semantic_guard.py23 tests, all passing.

Six of them replicate the exact guard hook logic from update_pievo_state(): blocked principles removed, validated ones survive, mixed batches handled correctly, guard=None passes everything through.

py -m pytest tests/test_semantic_guard.py -v

Usage

pievo -t task.yaml -m model.yaml -o ./out \
  --enable-semantic-guard --guard-policy strict
  • strict — any misalignment → block (default)
  • permissive — partial operational alignment → restrict
  • conservative — uncertain extraction → abstain

… checks

Integrates the Semantic Guard Protocol as an optional pre-check hook in
PiEvo's Bayesian update pipeline (update_pievo_state).  When enabled, new
LLM-generated principles are validated for variable-level semantic
alignment against all existing principles before entering the Gaussian
Process training and posterior update.

Problem
-------
PiEvo's Bayesian comparison assumes shared observational vocabulary.
Two principles may use the same variable name (e.g. 'mass_change',
'temperature') but embed them in different physical referents,
measurement protocols, or evidential frameworks -- producing corrupted
posteriors.

New files (pievo/guard/)
-------------------------
signature.py    Data models: VariableSignature, Principle, AlignmentReport,
                OperationalDefinition, EvidentialStatus, NO_SHARED_VARIABLES
alignment.py   3D alignment vector: referential, operational, evidential
               (EXACT / TOKEN / EMBEDDING similarity methods)
policy.py      Decision strategies: StrictPolicy, PermissivePolicy,
               ConservativePolicy
extractor.py   LLM-based signature extraction via PiEvo's
               OpenAIChatCompletionClient, plus ManualExtractor for
               ablation; 4 few-shot examples covering both referential
               and evidential misalignment patterns
guard.py       SemanticGuard orchestrator: check_pair(),
               check_against_all(), any_blocked(), signature caching

Modified files
--------------
evolveflow.py  PiEvo accepts optional guard parameter; guard intercepts
               new principles between _extract_principles() and
               _handle_new_principles(), removing blocked entries before
               Bayesian update
main.py        --enable-semantic-guard and --guard-policy CLI flags
               (strict / permissive / conservative)

Tests
-----
tests/test_semantic_guard.py   23 tests (all passing)
  - Alignment vector computation (3 tests)
  - Policy decisions (3 tests)
  - Guard orchestration (11 tests)
  - PiEvo integration simulation (6 tests) -- validates the exact
    guard hook logic from update_pievo_state() in evolveflow.py

Design
------
- Optional: guard=None by default; zero impact on existing workflows
- Non-invasive: single hook point, no API changes to existing code
- No new dependencies: uses PiEvo's existing util_client
- Confidence decoupled from alignment: raw semantic match scores are
  separate from extraction confidence, allowing policies to treat
  'genuine misalignment' and 'low-confidence extraction' as orthogonal

Usage
-----
  pievo -t task.yaml -m model.yaml -o ./out \
    --enable-semantic-guard --guard-policy strict

Related
-------
Standalone implementation with 145 unit tests and 4 paper-reproduction
experiments (Phlogiston vs Oxidation, Caloric vs Kinetic, etc.):
  https://github.com/Fengrru/semantic-guard-pievo
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant