docs: composing-word source-of-truth — design + shipped outcome#89
Closed
SHAWNERZZ wants to merge 31 commits into
Closed
docs: composing-word source-of-truth — design + shipped outcome#89SHAWNERZZ wants to merge 31 commits into
SHAWNERZZ wants to merge 31 commits into
Conversation
Captures the opt-in learned key-geometry feature: one per-user model behind both taps (KeyDetector) and gestures (ProximityInfo sweet spots, Java-only / no native rebuild), the context-prior + learned-geometry layers, hard caps to avoid wrong literals, privacy (content-free, incognito-gated), leantype.db persistence riding the existing backup, a stats page, and a phased build order. Co-Authored-By: Claude Opus 4.8 <[email protected]>
…store) Phase 1 of #1 (no behavior yet; nothing reads the model). Adds the opt-in prefs PREF_ADAPTIVE_KEY_GEOMETRY (+ strength) via the 5-file pattern, and a content-free per-(key, layout, orientation) touch-model table in leantype.db with a cached DAO (EMA running mean/variance + count, restore, clear). DB VERSION 2->3 with an additive onUpgrade, and copyFromDb extended to carry the model across the existing settings backup/restore (guarded so older backups without the table restore fine). Compiles: :app:compileStandardDebugJavaWithJavac. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Phase 2 policy layer for #1: capped, confidence- and strength-scaled landing-offset math (pure functions). The cap (MAX_SHIFT_FRACTION of key size) is the safety bound so a learned bias can never flip a clear press to a neighbor; confidence ramps the bias in with sample count; strength scales it (0 = off). 7 unit tests. Co-Authored-By: Claude Opus 4.8 <[email protected]>
…on + settings Phase 2 of #1 (testable end-to-end). Closes the loop: - PointerTracker records each letter tap's landing offset into the touch model (opt-in + incognito gated; async DB write so typing stays fast). - ProximityInfo shifts per-key sweet spots by the capped learned offset, so gesture recognition AND tap-correction follow where the user actually types. Generated even when a layout has no static touch-position-correction data. Element id passed from Keyboard; computed in Java, crosses the existing JNI - no native change. - Gesture typing -> Advanced: opt-in toggle + strength slider (reload keyboard on change). - @JvmStatic on the Kotlin DAO factory / manager so the Java hot paths can call them. Deferred: literal-tap KeyDetector tie-break (marginal/risky), the completion-derived context prior, and the stats page. Builds: :app:assembleStandardDebug. Co-Authored-By: Claude Opus 4.8 <[email protected]>
…page #1, follow-ups to the adaptive-geometry feature: - Gestures now teach the model (not just consume it) via their clean endpoints: finger-down -> word's first letter, finger-up -> last letter. Interior keys skipped (corner-cutting); fresh single strokes only. InputLogic.maybeRecordGestureEndpoints. - New stats screen (AdaptiveTypingStatsScreen) reachable from Gesture typing -> Advanced when the feature is on: per-key average offset, spread (consistency), sample count, plus a Reset button. Wired via SettingsDestination + a clickable Preference row. Build + SettingsContainerTest green; :app:assembleStandardDebug builds. Co-Authored-By: Claude Opus 4.8 <[email protected]>
…ure lifecycle WordComposer.mExtendBatchInputBase (the multi-part merged-trail base) was only cleared by a normally-completing gesture, so an abnormal end (cancel / empty-top recognition) left it armed, and no deletion path cleared it. A later fresh swipe then merged with the ghost trail - most visibly at the start of a text box. Clear it at the same word-end sites where mLiveStroke is already dropped (handleBackspaceEvent, resetComposingState, commitChosenWord, desync) plus the two gesture-lifecycle origins (onStartBatchInput top, onCancelBatchInput). Does not touch the hot WordComposer.reset() path. Adds 8 tests: base cleared after each backspace mode (character/fragment/whole-word), the delete slider, fresh onStartBatchInput, and onCancelBatchInput; plus 2 guards pinning the (currently dead) static-seed interlock. Co-Authored-By: Claude Opus 4.8 <[email protected]>
… page #1. The raw px-delta list was hard to interpret, so show the data spatially: - Store each key's size (KEY_WIDTH/KEY_HEIGHT) with its offset so the offset can be expressed as a fraction of the key (DB VERSION 3->4; additive ALTER on upgrade; copyFromDb reads by column name so older backups restore fine). - Stats page now renders a mock QWERTY where each key shows a dot at where you tend to land (offset as a fraction of the key) plus a faint spread ring; confident keys in the accent color, still-learning keys faded. The numeric list stays below for exact values. - Recorders (tap + gesture-endpoint) now pass the key's hitbox size. Build + extendBase/manager/settings tests green; :app:assembleStandardDebug builds. Co-Authored-By: Claude Opus 4.8 <[email protected]>
…recreate-on-upgrade step
Completes adaptive typing on the tap side: - AdaptiveKeyContext builds a "likely next key" prior from the top-5 suggestions, weighted equally (averaged, not score-skewed): the next char of the in-progress word's completions, or the first char of next-word predictions for a fresh word. Rebuilt between keystrokes in InputLogic.setSuggestedWords (off the tap hot path), read lock-free per tap. - KeyDetector.detectHitKey now biases the tapped key by the learned per-key landing offset (shifts the effective center) AND the context prior (enlarges likely keys). Both capped; the prior cap (~18% of key) is deliberately below the learned cap (~25%) so it nudges rather than dominates. Only near-boundary taps can flip; clear presses are untouched. - Suppressed during gestures/swipes (PointerTracker.isInGestureOrKeySwipe); gestures are not context-biased since suggestions don't change mid-swipe. Applies to both current words being built and the first key of a new word (predictions). Build + extendBase/manager/settings tests green; :app:assembleStandardDebug builds. Co-Authored-By: Claude Opus 4.8 <[email protected]>
… swipe-start behavior
Split the next-key context prior into its own opt-in setting (adaptive_context_prior, default off) alongside the existing learned key-geometry toggle, and group both under a single "Adaptive typing" section in the Gesture typing settings. - Settings/Defaults/SettingsValues: new mAdaptiveContextPrior flag. - KeyDetector: independently gate learned-offset bias vs prior boost; either alone can bias a near-boundary tap, both share the strength slider. adjustedDistance now takes usePrior and gates the boost. - InputLogic.setSuggestedWords: build the prior only when the prior toggle is on; clear AdaptiveKeyContext otherwise. - GestureTypingScreen: new "Adaptive typing" category holding both toggles; strength slider shown if either is on; stats shown if learning is on. Co-Authored-By: Claude Opus 4.8 <[email protected]>
A debug toggle ("Show adaptive targets on keyboard") that visualizes the
adaptive model directly on the live keyboard, so the feature is visible as
you type:
- Learned geometry: each letter key shows a faint geometric-center ring, an
arrow to its learned landing target, and a dot at that target.
- Context prior: keys the suggestions predict get a translucent halo whose
radius grows with the prior weight; halos morph between keystrokes.
Implemented as AdaptiveTargetsDrawingPreview (an AbstractDrawingPreview, same
mechanism as the gesture-debug overlay), drawn on the DrawingPreviewPlacerView
above the keys. It is purely visual, reads the same live model / prior /
settings the engine uses, and is gated on its pref each frame (zero cost when
off). The halo radius is exaggerated vs the engine's sub-key boost for
legibility; the visible keys are deliberately not reflowed.
- New pref PREF_ADAPTIVE_DEBUG_OVERLAY (5-file), shown under "Adaptive typing"
when either adaptive toggle is on.
- AdaptiveKeyContext gains a change listener fired on update()/clear(); the
overlay repaints on it. MainKeyboardView registers the listener and feeds
the overlay the keyboard + padding so markers align with rendered keys.
- docs/ADAPTIVE_TYPING.md: document the overlay, the independent context-prior
toggle, the grouped settings, the built heatmap, and fix the DB version.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
On-device tracing showed the next-key bias is applied correctly in practice (each tap is biased by the prediction for the prefix typed so far), but two issues made it look/behave wrong: 1. Overlay trailed by ~one key. The prior is rebuilt only when the suggestion strip refreshes, which is debounced ~100 ms; the trace showed a consistent ~113 ms gap between a keystroke and its setSuggested, so for most of the inter-key interval the overlay still showed the previous prediction. When the context prior is enabled, shorten that debounce (PROMPT_PRIOR_UPDATE_DELAY_MS = 30 ms in LatinIME.UIHandler) so the prediction lands before the next tap. 2. Bias silently skipped capital letters. The prior stores lowercase next-characters, but a shifted keyboard reports uppercase key codes, so AdaptiveKeyContext.weight() missed them (priorOnGeo=0.0 on the sentence-initial capital). weight() now folds the queried code to lowercase, so the bias and the overlay halos work on the shifted layout too. Also adds debug-overlay-gated tracing (AdaptivePrior tag) in setSuggestedWords and KeyDetector, plus AdaptiveKeyContext.debugString(), to make this diagnosable on-device, and documents both fixes in docs/ADAPTIVE_TYPING.md. Co-Authored-By: Claude Opus 4.8 <[email protected]>
The suggestion-update debounce (~100 ms) exists because the suggestion compute blocks the UI thread (performUpdateSuggestionStripSync waits on holder.get with a 200 ms timeout); the debounce coalesces that expensive compute so fast typing isn't hit by one block per keystroke. The functional context-prior bias already works at the full debounce (at normal speed the prior is ready before the next tap). Only the debug overlay visibly trails. So instead of shortening the debounce whenever the prior is enabled, shorten it only while the debug overlay is also on — a temporary diagnostic — and raise the value from 30 ms to 50 ms. Ordinary typing (overlay off) keeps the full debounce and is unaffected. Co-Authored-By: Claude Opus 4.8 <[email protected]>
The 50 ms (and even 30 ms) overlay debounce was easy to out-type. A truly non-blocking async refresh isn't safe here: the suggestion compute reads the non-thread-safe WordComposer on a background thread, and the existing design keeps that safe by blocking the UI during the compute (the only existing async compute runs during gestures, when there's no concurrent typing). Doing it async during typing would risk a torn read / crash and would need a composer snapshot — too much for a debug visualization. Instead, drop the suggestion debounce to 0 while the debug overlay + the context prior are both on. The overlay already repaints the instant the prior updates (the AdaptiveKeyContext change listener fires inside setSuggestedWords), so computing immediately is the safe equivalent of "update as soon as the suggestion is made". The only remaining latency is the compute itself (~5-10 ms release, ~20 ms debug build). Ordinary typing (overlay off) keeps the full, smooth debounce. Reverts the earlier PROMPT_PRIOR_UPDATE_DELAY_MS approach. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Casing bugs specific to two-thumb live-converge / multi-part re-recognition, where every extending tap or swipe REPLACES the whole composing word with the recognizer's fresh (lowercase) output: - Dropped auto-cap (#5): "Hello" -> "hellow" on the first extension. - Stuck all-caps (#4): a short ambiguous swipe whose top pick is an all-caps acronym ("CSA") set WordComposer.isAllUpperCase, which forced every later suggestion upper, so the word stuck in caps ("CSA"->"CAN"->"CAME"). - Swipe-extension downcasing: "Was"+swipe -> "wait" (a second swipe re-entered onStartBatchInput and re-captured the now-cleared shift state). Approach: separate a word's casing INTENT from the recognizer's letters. - WordComposer.mCapitalizedMode is the persistent per-word intent: seeded at word start from auto-cap + shift, it survives the setBatchInputWord rebuild and is cleared only at commitWord. Exposed via getCapitalizedMode(). - New InputLogic.applyComposingCase(lemma, capsMode, locale) treats the recognizer output as a casing-NEUTRAL lemma (lowercased first) and re-applies the intent. Lowercasing first dissolves #4 at the source: the composing word is never all-caps, so isAllUpperCase never arms. - onStartBatchInput captures the intent only for a FRESH word (gated on !extendComposingWord), so an extending gesture preserves the first fragment's intent instead of re-capturing the auto-cleared shift state (#5 / "Wait"). - Fresh-word gesture capitalization is unchanged (still mShiftModeAtGestureStart, captured before any state mutates), so plain glide typing is byte-identical and a standalone acronym swipe ("CSA") still stays as-is. Tests: applyComposingCase covered directly (pure, native-free) plus intent lifecycle tests in InputLogicTest. :app:testOfflineDebugUnitTest green except the 3 documented pre-existing failures. Fixes #4, #5. Co-Authored-By: Claude Opus 4.8 <[email protected]>
feat(typing): adaptive learned key geometry (taps + gestures)
Plan of record for a multi-phase refactor of two-thumb / multi-part composition: make the editor text + composing region the single source of truth for the current word, and derive stroke geometry from text on demand (getCoordinatesForCurrentKeyboard) instead of maintaining drift-prone parallel state (mLiveStroke, mInputPointers survives-reset contract, mGestureFragmentBoundaries). Captures the problem (the recurring stale-stroke / desync bug class), the existing foundation already used for tap recorrection, the target design and state inventory, and a 3-phase incremental plan (fragment-pop fix -> live-converge from text -> generalize to mid-word editing) with on-device validation gates. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Clarify scope per review: knowing the current word from the editor is for CONTEXT, not re-typing. Taps stay exact and bypass live-composition entirely (replaces the re-recognize-taps behavior / PREF_MULTIPART_RERECOGNIZE_TAPS); moving the cursor into a word is context-only and must not re-compose; only a swipe added at the cursor (re)composes by building on that word's text. Updated target design, state inventory, and Phase 2/3 accordingly. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Earlier revision wrongly proposed making taps exact and removing tap re-recognition. Re-recognition (combining taps + swipes into one word, re-resolved against context) is the core Nintype behavior and must be preserved. Corrected trigger model: - A swipe always re-recognizes, pulling in the word at the cursor as context (typed / swiped / just-moved-into): "Doc" + swipe "ument" -> "Document". - A tap is exact UNLESS the current word already contains a swipe; then taps before/during/after contribute (the combine feature stays). Tap into a no-swipe word is literal: "Do|c" + tap "g" -> "Dogc". - Cursor move is context-only (inert until a swipe; bare taps stay exact). mLiveStroke is retired as STORED state only — the re-recognition behavior it powered is preserved, re-sourced from the current word's text on demand. Updated state inventory and Phase 2/3 accordingly. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Per review: swiping onto a word the cursor was moved into re-recognizes with that word's context even when it is already committed to the text box (re-entered via a cursor move) — the full 'editor as source of truth' payoff, scoped to Phase 3. Co-Authored-By: Claude Opus 4.8 <[email protected]>
- Mark Phase 1 & 2 landed/validated; reframe Phase 2 from 'retire mLiveStroke' to 'real stroke primary + text-derived fallback' (the feel-test outcome). - Update state inventory: mLiveStroke is KEPT as primary, backed by the text-derived fallback. - Add 'Current behavior (pre-Phase 3)' section documenting that a cursor move + swipe starts a fresh autospaced word, while a tap edits the existing word, with the code mechanism and how Phase 3 would change the swipe half. Co-Authored-By: Claude Opus 4.8 <[email protected]>
…lit) Product decision: swipe at a moved cursor = start fresh word, tap = edit the existing word is the intended behavior, not a stepping stone. Phase 3's swipe-onto-any-existing-word re-recognition is dropped. The fragment-boundary derivation cleanup is noted as an independent optional leftover. Co-Authored-By: Claude Opus 4.8 <[email protected]>
…on-point fix) The doc described the Phase 2 mLiveStroke hybrid/fallback as landed. In reality that was attempted and REVERTED (the fallback was effectively dead code, and the real captured stroke beats synthesized geometry on feel). The bug was instead fixed surgically at the swipe-extend consumption point (mComposingStrokeStale + realignComposerStrokeToText). - Status header: 'partially shipped — record of what shipped and why the plan changed'. - Phases/outcome: Phase 1 shipped; Phase 2 attempted-then-reverted (with the lesson); Phase 3 descoped; new 'what actually fixed the bug' section; Nintype whole-word backspace noted as planned/next. - State inventory: mLiveStroke kept unchanged; mInputPointers drift fixed not retired. - Marked the 'Target design' bullets as the original aspiration, only partially realized. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Design note for the two-thumb stale-stroke work, reconciled to record what actually shipped rather than the original aspiration.
mComposingStrokeStale+realignComposerStrokeToText), not by removing the parallel stroke buffer.mLiveStroke/ derive everything from text' refactor (original Phase 2) was attempted and reverted — the fallback was effectively dead code, and the real captured stroke re-recognizes better than synthesized key-centers. Documented as a lesson.Mergeable as the record of this effort.
🤖 Generated with Claude Code