feat: add optional Semantic Guard plugin for variable-level alignment… - #2
Open
Fengrru wants to merge 1 commit into
Open
feat: add optional Semantic Guard plugin for variable-level alignment…#2Fengrru wants to merge 1 commit into
Fengrru wants to merge 1 commit into
Conversation
… checks
Integrates the Semantic Guard Protocol as an optional pre-check hook in
PiEvo's Bayesian update pipeline (update_pievo_state). When enabled, new
LLM-generated principles are validated for variable-level semantic
alignment against all existing principles before entering the Gaussian
Process training and posterior update.
Problem
-------
PiEvo's Bayesian comparison assumes shared observational vocabulary.
Two principles may use the same variable name (e.g. 'mass_change',
'temperature') but embed them in different physical referents,
measurement protocols, or evidential frameworks -- producing corrupted
posteriors.
New files (pievo/guard/)
-------------------------
signature.py Data models: VariableSignature, Principle, AlignmentReport,
OperationalDefinition, EvidentialStatus, NO_SHARED_VARIABLES
alignment.py 3D alignment vector: referential, operational, evidential
(EXACT / TOKEN / EMBEDDING similarity methods)
policy.py Decision strategies: StrictPolicy, PermissivePolicy,
ConservativePolicy
extractor.py LLM-based signature extraction via PiEvo's
OpenAIChatCompletionClient, plus ManualExtractor for
ablation; 4 few-shot examples covering both referential
and evidential misalignment patterns
guard.py SemanticGuard orchestrator: check_pair(),
check_against_all(), any_blocked(), signature caching
Modified files
--------------
evolveflow.py PiEvo accepts optional guard parameter; guard intercepts
new principles between _extract_principles() and
_handle_new_principles(), removing blocked entries before
Bayesian update
main.py --enable-semantic-guard and --guard-policy CLI flags
(strict / permissive / conservative)
Tests
-----
tests/test_semantic_guard.py 23 tests (all passing)
- Alignment vector computation (3 tests)
- Policy decisions (3 tests)
- Guard orchestration (11 tests)
- PiEvo integration simulation (6 tests) -- validates the exact
guard hook logic from update_pievo_state() in evolveflow.py
Design
------
- Optional: guard=None by default; zero impact on existing workflows
- Non-invasive: single hook point, no API changes to existing code
- No new dependencies: uses PiEvo's existing util_client
- Confidence decoupled from alignment: raw semantic match scores are
separate from extraction confidence, allowing policies to treat
'genuine misalignment' and 'low-confidence extraction' as orthogonal
Usage
-----
pievo -t task.yaml -m model.yaml -o ./out \
--enable-semantic-guard --guard-policy strict
Related
-------
Standalone implementation with 145 unit tests and 4 paper-reproduction
experiments (Phlogiston vs Oxidation, Caloric vs Kinetic, etc.):
https://github.com/Fengrru/semantic-guard-pievo
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
This adds a semantic guard that checks whether two LLM-generated principles actually mean the same thing by the same variable names, before PiEvo throws them into the same Bayesian pot.
Why
Say PiEvo is working on combustion. The Principle Agent generates two candidates:
Both mention
mass_change. PiEvo's Bayesian update treats them as sharing an observational vocabulary, so it computes likelihoods as if they're talking about the same quantity. They're not. Phlogiston'smass_changemeans "weight of dephlogisticated metal in open air." Oxidation's means "weight of metal oxide in sealed vessel." Same name, different referent, different measurement protocol — the posterior gets corrupted.This isn't an edge case. It's the Phlogiston → Oxidation paradigm shift, and it also occurs with Caloric → Kinetic (temperature as primitive measurement vs. derived quantity).
How
The hook sits at exactly one place:
update_pievo_state(), between_extract_principles()and_handle_new_principles(). A new principle is checked against every existing one. If any shared variable name maps to a different referent, operational definition, or evidential status, that principle is removed fromself.principlesand never reaches the GP training or posterior update.When
guardisNone(the default), nothing changes.Files
New —
pievo/guard/(6 modules, no new dependencies)signature.py— data modelsalignment.py— 3D alignment vector (referential, operational, evidential; EXACT / TOKEN / EMBEDDING)policy.py—StrictPolicy,PermissivePolicy,ConservativePolicyextractor.py— LLM extraction via PiEvo's existingutil_client, plusManualExtractorfor ablation; 4 few-shot examplesguard.py—SemanticGuardorchestrator__init__.py— public APIModified (+83 lines, both additive)
evolveflow.py—PiEvo.__init__accepts optionalguardparameter; hook inupdate_pievo_state()main.py—--enable-semantic-guardand--guard-policyCLI flagsTests
tests/test_semantic_guard.py— 23 tests, all passing.Six of them replicate the exact guard hook logic from
update_pievo_state(): blocked principles removed, validated ones survive, mixed batches handled correctly, guard=None passes everything through.Usage
strict— any misalignment → block (default)permissive— partial operational alignment → restrictconservative— uncertain extraction → abstain