feat(skills): quality gates, project/global scope boundary, LLM curation - #120
Merged
Conversation
TestStartRunsSweepOnTick waited for the janitor's real 1-minute tick, making internal/maintenance take 63s on every run and blowing past any 60s test bound (it looked like a hang under 'go test'/'go test -race'). Start now honors a package-level tickInterval test hook; the test overrides it to 25ms. Production behavior is unchanged.
Anti-garbage: - derive.go: filter shell operators, flags, and version fragments from derived trigger keywords. - DetectCorrection: fire only on explicit correction phrases and require the corrected commands to share a lead verb (shareLeadVerb), killing the 'corrected-cd' garbage class. - extractTopic/leadVerb: skip shell plumbing (cd, sudo, operators, paths, flags) so topics resolve to real verbs. Project -> global boundary: - Split the reusability gate: NonReusableReason (machine-specific: absolute home paths) drops suggestions entirely; ProjectScopeReason (project-specific: repo-rooted ./scripts/... invocations, hardcoded release version tags) redirects them to ./.odek/skills instead of the global dir. Redirect is refused when the project dir resolves to the same location as the global dir (odek run from $HOME). - Confine micro-curation (MicroCuration/RunAutoCurate) to skills physically stored under the global dir via Skill.Source.Dir, so a project skill can never be merged into a global skill or deleted by the curator (fail-closed on empty source). - Wire projectDir through RunAutoSaveLoop and the CLI/serve/telegram call sites; reload + notifier events account for ProjectSaved. Docs: LEARNING.md design decisions, AGENTS.md security architecture.
Skill bodies are session content written to disk permanently; a command captured with credentials in it (curl -H 'Authorization: ...', export AWS_..., etc.) would be immortalized in ~/.odek/skills. WriteSkill now runs internal/redact over body + description: matches are replaced with [REDACTED] and the skill is pinned to NeedsReview so the operator sees why before it can auto-load. Covers every write path (auto-save, interactive prompt, skill_save, curator merges, imports).
AutoSaveSuggestions kept the first MaxPerRun eligible suggestions in heuristic emission order, so weak patterns could crowd out strong ones. Eligible suggestions are now sorted by a deterministic substance score (verification section, command-log evidence, body length, description, LLM-judged provenance) before the cap is applied.
A single quirky session could previously produce a saved skill. The auto-save pipeline now records each suggestion's fingerprint (heuristic+name, stable across sessions) in ~/.odek/skills/.candidates.json and only saves once the pattern recurs in a later session (auto_save.min_occurrences, default 2; set 1 to disable). Pending suggestions are reported in AutoSaveResult.Pending and verbose output. Candidates unseen for 30 days are pruned.
Dedup was exact BodyHash only (and post-hoc via curation), so light paraphrases accumulated as separate skills. AutoSaveSuggestions now compares each quality-gated suggestion against existing skills in the target dir with stopword-filtered Jaccard word-set similarity (>= 0.85 = duplicate, recorded in AutoSaveResult.DuplicateOf). Same-named saves bypass the gate: re-saving a name is an update, not a duplicate.
Mechanical merges concatenated bodies with a 'Merged from X' separator, producing ever-growing, contradictory skill files. When llm_curate is enabled and an LLM client is available, ExecuteMicroCurationWithLLM now asks the model to synthesize one deduplicated body (sanity-checked for structure and size); any failure or invalid output falls back to the mechanical merge. The worst-of provenance union from MergeSkills is kept either way, so an LLM merge can never launder taint.
…ctive prompt The project/global boundary only existed in the unattended pipeline. skill_save now accepts scope: "project" to save project-specific procedures to ./.odek/skills (refused when no distinct project dir exists, e.g. odek run from $HOME), and the interactive learn prompt gained a 'p' response with the same semantics. Both paths keep the existing NeedsReview pinning.
The field-wise AutoSave merge in internal/config silently dropped the new min_occurrences key. Also pins min_occurrences=1 in TestRunLearn_MultiStepProcedure (it covers the save path; the gate has its own unit tests) and documents the new gates in LEARNING.md, CONFIG.md, and AGENTS.md.
… new gates Interactive prompt examples now show the p=project option, config snippets include auto_save.min_occurrences, the LEARNING.md limitations section reflects save-time dedup, and SECURITY.md's skill provenance section documents the scope gates and save-time hygiene (recurrence, near-dup rejection, secret redaction).
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
odek | 6bb2ea1 | Commit Preview URL Branch Preview URL |
Jul 27 2026, 05:52 PM |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A hardening pass over the skill-learning pipeline with two goals: stop creating garbage skills, and project-related skills must never be promoted to global skills.
Project → global boundary
selfimprove.go):NonReusableReason(machine-specific: absolute home paths) drops suggestions entirely;ProjectScopeReason(project-specific: repo-rooted./scripts/...invocations, hardcoded release version tags) redirects them to./.odek/skillsinstead of the global dir. Redirect is refused when the project dir resolves to the global one (odek run from$HOME).curator.go):MicroCuration/RunAutoCurate/ExecuteMicroCurationare confined viaSkill.Source.Dirto skills physically stored under the global dir — previously a project skill overlapping a global draft would be merged into a global skill, silently promoting project content (fail-closed on empty source).skill_saveacceptsscope: "project"; the interactive learn prompt has apoption. Both refuse when no distinct project dir exists.odek skill promote(project-dir skills are forcedNeedsReviewon scan, as before).Anti-garbage gates
DetectCorrectiontightening: fires only on explicit correction phrases and requires the corrected commands to share a lead verb (shareLeadVerb) — kills thecorrected-cdgarbage class.extractTopicskips shell plumbing (cd,sudo, operators, paths).candidates.go): a pattern must recur inauto_save.min_occurrencesdistinct sessions (default 2,1disables) before auto-save; first sightings are tracked in~/.odek/skills/.candidates.json(30-day pruning) and reported as pending.max_per_runkeeps the strongest.WriteSkill(single choke point for all save paths) runsinternal/redactover body + description — matches become[REDACTED]and the skill is pinned toNeedsReview.LLM-enhanced curation
With
llm_curateenabled, merge bodies are synthesized by the model (structure/size sanity-checked, mechanical concatenation as fallback) instead of accumulating--- Merged from X ---sections. The worst-of provenance union is preserved either way, so a merge can never launder taint.Also in this PR
test(maintenance):TestStartRunsSweepOnTickno longer waits for the real 1-minute janitor tick (63s → ~1s per run; it was blowing past 60s test bounds and looking like ago testhang).fix(config): the field-wiseauto_savemerge now honorsmin_occurrences(was silently dropped).Test plan
go test ./... -count=1— all 29 packages okgo test -race ./internal/skills/ ./internal/config/ ./cmd/odek/ -count=1— okskill_savescope (project/same-dir refusal/invalid), WriteSkill redaction (leaky + clean)LEARNING.md,CONFIG.md,SECURITY.md,AGENTS.md