Add evolving teaching skills with local learning loop#101
Open
octavi42 wants to merge 15 commits into
Open
Conversation
Clicky now learns from successful tutoring sessions, injects matched skills into prompts, and includes a mock-worker E2E test for the read/write loop.
Derive skill names from the primary question, persist cross-session topic history, verify read-path injection via Phase B relaunch, and run the full harness locally and on macos-14 GitHub Actions.
E2E workflow runs teaching-skills.sh without niche-discovery tests.
Hermes-style loop: stable (app, task) identity, target-app resolution from session text, patch-first synthesis, and E2E Phase C — so refinements update one skill instead of spawning duplicates mid-tutorial.
Input Monitoring gates the CGEvent tap instead of Accessibility, and ClickyWorkerBaseURL routes Claude, TTS, and AssemblyAI through the Cloudflare Worker so manual testing matches main.
Ship the full skills library view, LLM curator passes, shared E2E helpers, skills-library regression script, and teaching-only launch hooks without niche or vault dependencies.
Drop niche and full-stack scripts from the upstream PR branch so CI matches the scoped feature set.
|
hi, if you would like to work on a more recent fork of this project, reach out to me |
* Add runtime data-home isolation for parallel app instances. Introduce ClickyPaths/ClickyDefaults/ClickyLaunchArguments so the app's data home (~/.clicky), UserDefaults suite, and global push-to-talk can be redirected per instance via -CLICKY_HOME / -CLICKY_DEFAULTS_SUITE / -CLICKY_DISABLE_GLOBAL_PTT (or the CLICKY_HOME env var). This lets multiple worktree builds run side by side without clobbering each other's skills, preferences, or fighting over the global hotkey. Routes skill and topic stores through ClickyPaths, threads CLICKY_HOME through the E2E harness, and adds unit coverage plus a @mainactor fix so the test target compiles. * Make input-monitoring preflight nonisolated to fix Xcode 16.4 build. GlobalPushToTalkShortcutMonitor.start() runs in a nonisolated context and calls WindowPositionManager.hasInputMonitoringPermission(), which Xcode 16.4 (used by CI) rejects as a cross-actor call. The helper only reads live TCC state via a pure C preflight, so marking it nonisolated is safe and unblocks the CI build on both Xcode 16.4 and 17.
ScreenCaptureKit routes through screencaptureui on macOS Tahoe and shows floating screenshot thumbnails during voice sessions. Switch companion capture to CGWindowListCreateImage (via a runtime dlsym wrapper), suppress and dismiss leftover thumbnail UI for the app lifetime, and harden permission refresh so dev builds reflect live TCC state after granting in System Settings. Also fixes a CGWindowID crash when window numbers are out of UInt32 range.
new-worktree.sh creates a branch worktree with its own CLICKY_HOME, UserDefaults suite, and optional push-to-talk disable, and patches that worktree's Xcode scheme so multiple Clicky instances can run side by side without sharing ~/.clicky or fighting over the global hotkey.
Save session JSON under ~/.clicky/sessions when a voice session ends via confirm phrase, 30s idle, or panel close. Includes 7-day retention cleanup, SessionStore unit tests, and cross-suite test isolation for ClickyPaths.
maybeWriteTeachingSkill's in-flight task ended with an unconditional sessionTrace.removeAll(). Since synthesis is async, it could fire after a new session had already started and wipe its turns before persistence. Only clear the trace when it still matches the synthesized snapshot.
- Keep the in-memory trace and re-arm the idle timer when sessionStore.save throws, so a transient I/O error retries instead of dropping the capture. - Arm the 30s idle boundary after TTS playback completes rather than in recordSessionExchange, so a long spoken reply no longer fires the timer mid-speech and splits one conversation into two persisted sessions.
If the 30s idle timer fires while the assistant is mid-speech, re-arm instead of finalizing so a long spoken reply cannot split one conversation into two persisted sessions.
Persist voice sessions to disk on session boundary
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SKILL.mdfiles under~/.clicky/skills/(inspired by Hermes Agent, adapted for screen-native teaching)teaching-skills.sh,skills-library.sh) with mock worker + GitHub Actions onmacos-15Demo success: "I taught Clicky how to walk me through TextEdit save yesterday — today it remembers and points better on the first try."
Test plan
./tests/e2e/run-all.shpasses locally (write + read-path + patch + library restore)TeachingSkillTestsfrom XcodeE2E Teaching Skillsworkflow passes on this PRNotes
This PR is intentionally scoped to teaching skills only — niche discovery and personal vault are planned as follow-up PRs.