Fix stale gesture stroke after edits: realign extend-base from text (Phase 1 + consumption-point)#88
Closed
SHAWNERZZ wants to merge 26 commits into
Closed
Fix stale gesture stroke after edits: realign extend-base from text (Phase 1 + consumption-point)#88SHAWNERZZ wants to merge 26 commits into
SHAWNERZZ wants to merge 26 commits into
Conversation
Captures the opt-in learned key-geometry feature: one per-user model behind both taps (KeyDetector) and gestures (ProximityInfo sweet spots, Java-only / no native rebuild), the context-prior + learned-geometry layers, hard caps to avoid wrong literals, privacy (content-free, incognito-gated), leantype.db persistence riding the existing backup, a stats page, and a phased build order. Co-Authored-By: Claude Opus 4.8 <[email protected]>
…store) Phase 1 of #1 (no behavior yet; nothing reads the model). Adds the opt-in prefs PREF_ADAPTIVE_KEY_GEOMETRY (+ strength) via the 5-file pattern, and a content-free per-(key, layout, orientation) touch-model table in leantype.db with a cached DAO (EMA running mean/variance + count, restore, clear). DB VERSION 2->3 with an additive onUpgrade, and copyFromDb extended to carry the model across the existing settings backup/restore (guarded so older backups without the table restore fine). Compiles: :app:compileStandardDebugJavaWithJavac. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Phase 2 policy layer for #1: capped, confidence- and strength-scaled landing-offset math (pure functions). The cap (MAX_SHIFT_FRACTION of key size) is the safety bound so a learned bias can never flip a clear press to a neighbor; confidence ramps the bias in with sample count; strength scales it (0 = off). 7 unit tests. Co-Authored-By: Claude Opus 4.8 <[email protected]>
…on + settings Phase 2 of #1 (testable end-to-end). Closes the loop: - PointerTracker records each letter tap's landing offset into the touch model (opt-in + incognito gated; async DB write so typing stays fast). - ProximityInfo shifts per-key sweet spots by the capped learned offset, so gesture recognition AND tap-correction follow where the user actually types. Generated even when a layout has no static touch-position-correction data. Element id passed from Keyboard; computed in Java, crosses the existing JNI - no native change. - Gesture typing -> Advanced: opt-in toggle + strength slider (reload keyboard on change). - @JvmStatic on the Kotlin DAO factory / manager so the Java hot paths can call them. Deferred: literal-tap KeyDetector tie-break (marginal/risky), the completion-derived context prior, and the stats page. Builds: :app:assembleStandardDebug. Co-Authored-By: Claude Opus 4.8 <[email protected]>
…page #1, follow-ups to the adaptive-geometry feature: - Gestures now teach the model (not just consume it) via their clean endpoints: finger-down -> word's first letter, finger-up -> last letter. Interior keys skipped (corner-cutting); fresh single strokes only. InputLogic.maybeRecordGestureEndpoints. - New stats screen (AdaptiveTypingStatsScreen) reachable from Gesture typing -> Advanced when the feature is on: per-key average offset, spread (consistency), sample count, plus a Reset button. Wired via SettingsDestination + a clickable Preference row. Build + SettingsContainerTest green; :app:assembleStandardDebug builds. Co-Authored-By: Claude Opus 4.8 <[email protected]>
…ure lifecycle WordComposer.mExtendBatchInputBase (the multi-part merged-trail base) was only cleared by a normally-completing gesture, so an abnormal end (cancel / empty-top recognition) left it armed, and no deletion path cleared it. A later fresh swipe then merged with the ghost trail - most visibly at the start of a text box. Clear it at the same word-end sites where mLiveStroke is already dropped (handleBackspaceEvent, resetComposingState, commitChosenWord, desync) plus the two gesture-lifecycle origins (onStartBatchInput top, onCancelBatchInput). Does not touch the hot WordComposer.reset() path. Adds 8 tests: base cleared after each backspace mode (character/fragment/whole-word), the delete slider, fresh onStartBatchInput, and onCancelBatchInput; plus 2 guards pinning the (currently dead) static-seed interlock. Co-Authored-By: Claude Opus 4.8 <[email protected]>
… page #1. The raw px-delta list was hard to interpret, so show the data spatially: - Store each key's size (KEY_WIDTH/KEY_HEIGHT) with its offset so the offset can be expressed as a fraction of the key (DB VERSION 3->4; additive ALTER on upgrade; copyFromDb reads by column name so older backups restore fine). - Stats page now renders a mock QWERTY where each key shows a dot at where you tend to land (offset as a fraction of the key) plus a faint spread ring; confident keys in the accent color, still-learning keys faded. The numeric list stays below for exact values. - Recorders (tap + gesture-endpoint) now pass the key's hitbox size. Build + extendBase/manager/settings tests green; :app:assembleStandardDebug builds. Co-Authored-By: Claude Opus 4.8 <[email protected]>
…recreate-on-upgrade step
Completes adaptive typing on the tap side: - AdaptiveKeyContext builds a "likely next key" prior from the top-5 suggestions, weighted equally (averaged, not score-skewed): the next char of the in-progress word's completions, or the first char of next-word predictions for a fresh word. Rebuilt between keystrokes in InputLogic.setSuggestedWords (off the tap hot path), read lock-free per tap. - KeyDetector.detectHitKey now biases the tapped key by the learned per-key landing offset (shifts the effective center) AND the context prior (enlarges likely keys). Both capped; the prior cap (~18% of key) is deliberately below the learned cap (~25%) so it nudges rather than dominates. Only near-boundary taps can flip; clear presses are untouched. - Suppressed during gestures/swipes (PointerTracker.isInGestureOrKeySwipe); gestures are not context-biased since suggestions don't change mid-swipe. Applies to both current words being built and the first key of a new word (predictions). Build + extendBase/manager/settings tests green; :app:assembleStandardDebug builds. Co-Authored-By: Claude Opus 4.8 <[email protected]>
… swipe-start behavior
Split the next-key context prior into its own opt-in setting (adaptive_context_prior, default off) alongside the existing learned key-geometry toggle, and group both under a single "Adaptive typing" section in the Gesture typing settings. - Settings/Defaults/SettingsValues: new mAdaptiveContextPrior flag. - KeyDetector: independently gate learned-offset bias vs prior boost; either alone can bias a near-boundary tap, both share the strength slider. adjustedDistance now takes usePrior and gates the boost. - InputLogic.setSuggestedWords: build the prior only when the prior toggle is on; clear AdaptiveKeyContext otherwise. - GestureTypingScreen: new "Adaptive typing" category holding both toggles; strength slider shown if either is on; stats shown if learning is on. Co-Authored-By: Claude Opus 4.8 <[email protected]>
A debug toggle ("Show adaptive targets on keyboard") that visualizes the
adaptive model directly on the live keyboard, so the feature is visible as
you type:
- Learned geometry: each letter key shows a faint geometric-center ring, an
arrow to its learned landing target, and a dot at that target.
- Context prior: keys the suggestions predict get a translucent halo whose
radius grows with the prior weight; halos morph between keystrokes.
Implemented as AdaptiveTargetsDrawingPreview (an AbstractDrawingPreview, same
mechanism as the gesture-debug overlay), drawn on the DrawingPreviewPlacerView
above the keys. It is purely visual, reads the same live model / prior /
settings the engine uses, and is gated on its pref each frame (zero cost when
off). The halo radius is exaggerated vs the engine's sub-key boost for
legibility; the visible keys are deliberately not reflowed.
- New pref PREF_ADAPTIVE_DEBUG_OVERLAY (5-file), shown under "Adaptive typing"
when either adaptive toggle is on.
- AdaptiveKeyContext gains a change listener fired on update()/clear(); the
overlay repaints on it. MainKeyboardView registers the listener and feeds
the overlay the keyboard + padding so markers align with rendered keys.
- docs/ADAPTIVE_TYPING.md: document the overlay, the independent context-prior
toggle, the grouped settings, the built heatmap, and fix the DB version.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
On-device tracing showed the next-key bias is applied correctly in practice (each tap is biased by the prediction for the prefix typed so far), but two issues made it look/behave wrong: 1. Overlay trailed by ~one key. The prior is rebuilt only when the suggestion strip refreshes, which is debounced ~100 ms; the trace showed a consistent ~113 ms gap between a keystroke and its setSuggested, so for most of the inter-key interval the overlay still showed the previous prediction. When the context prior is enabled, shorten that debounce (PROMPT_PRIOR_UPDATE_DELAY_MS = 30 ms in LatinIME.UIHandler) so the prediction lands before the next tap. 2. Bias silently skipped capital letters. The prior stores lowercase next-characters, but a shifted keyboard reports uppercase key codes, so AdaptiveKeyContext.weight() missed them (priorOnGeo=0.0 on the sentence-initial capital). weight() now folds the queried code to lowercase, so the bias and the overlay halos work on the shifted layout too. Also adds debug-overlay-gated tracing (AdaptivePrior tag) in setSuggestedWords and KeyDetector, plus AdaptiveKeyContext.debugString(), to make this diagnosable on-device, and documents both fixes in docs/ADAPTIVE_TYPING.md. Co-Authored-By: Claude Opus 4.8 <[email protected]>
The suggestion-update debounce (~100 ms) exists because the suggestion compute blocks the UI thread (performUpdateSuggestionStripSync waits on holder.get with a 200 ms timeout); the debounce coalesces that expensive compute so fast typing isn't hit by one block per keystroke. The functional context-prior bias already works at the full debounce (at normal speed the prior is ready before the next tap). Only the debug overlay visibly trails. So instead of shortening the debounce whenever the prior is enabled, shorten it only while the debug overlay is also on — a temporary diagnostic — and raise the value from 30 ms to 50 ms. Ordinary typing (overlay off) keeps the full debounce and is unaffected. Co-Authored-By: Claude Opus 4.8 <[email protected]>
The 50 ms (and even 30 ms) overlay debounce was easy to out-type. A truly non-blocking async refresh isn't safe here: the suggestion compute reads the non-thread-safe WordComposer on a background thread, and the existing design keeps that safe by blocking the UI during the compute (the only existing async compute runs during gestures, when there's no concurrent typing). Doing it async during typing would risk a torn read / crash and would need a composer snapshot — too much for a debug visualization. Instead, drop the suggestion debounce to 0 while the debug overlay + the context prior are both on. The overlay already repaints the instant the prior updates (the AdaptiveKeyContext change listener fires inside setSuggestedWords), so computing immediately is the safe equivalent of "update as soon as the suggestion is made". The only remaining latency is the compute itself (~5-10 ms release, ~20 ms debug build). Ordinary typing (overlay off) keeps the full, smooth debounce. Reverts the earlier PROMPT_PRIOR_UPDATE_DELAY_MS approach. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Casing bugs specific to two-thumb live-converge / multi-part re-recognition, where every extending tap or swipe REPLACES the whole composing word with the recognizer's fresh (lowercase) output: - Dropped auto-cap (#5): "Hello" -> "hellow" on the first extension. - Stuck all-caps (#4): a short ambiguous swipe whose top pick is an all-caps acronym ("CSA") set WordComposer.isAllUpperCase, which forced every later suggestion upper, so the word stuck in caps ("CSA"->"CAN"->"CAME"). - Swipe-extension downcasing: "Was"+swipe -> "wait" (a second swipe re-entered onStartBatchInput and re-captured the now-cleared shift state). Approach: separate a word's casing INTENT from the recognizer's letters. - WordComposer.mCapitalizedMode is the persistent per-word intent: seeded at word start from auto-cap + shift, it survives the setBatchInputWord rebuild and is cleared only at commitWord. Exposed via getCapitalizedMode(). - New InputLogic.applyComposingCase(lemma, capsMode, locale) treats the recognizer output as a casing-NEUTRAL lemma (lowercased first) and re-applies the intent. Lowercasing first dissolves #4 at the source: the composing word is never all-caps, so isAllUpperCase never arms. - onStartBatchInput captures the intent only for a FRESH word (gated on !extendComposingWord), so an extending gesture preserves the first fragment's intent instead of re-capturing the auto-cleared shift state (#5 / "Wait"). - Fresh-word gesture capitalization is unchanged (still mShiftModeAtGestureStart, captured before any state mutates), so plain glide typing is byte-identical and a standalone acronym swipe ("CSA") still stays as-is. Tests: applyComposingCase covered directly (pure, native-free) plus intent lifecycle tests in InputLogicTest. :app:testOfflineDebugUnitTest green except the 3 documented pre-existing failures. Fixes #4, #5. Co-Authored-By: Claude Opus 4.8 <[email protected]>
feat(typing): adaptive learned key geometry (taps + gestures)
After a fragment-pop backspace (swipe "th" + swipe "ing", then backspace
to drop "ing"), the raw stroke buffer still held the longer pre-pop
geometry. A following swipe-extend snapshotted that stale buffer as its
merged-trail base and rebuilt an ever-longer word ("thinking" instead of
"thing"). Phase 1 of COMPOSING_WORD_SOURCE_OF_TRUTH.md.
- tryFragmentBackspace now lowercases the truncated word before key-center
lookup (Keyboard#getCoordinates is an exact code-point match and layouts
store lowercase, so "Th" would resolve to NOT_A_COORDINATE and drop out).
- seedInputPointersFromKeyCenters skips NOT_A_COORDINATE points so an
unresolvable key cannot warp the stroke toward (-1,-1).
- Added getExtendBatchInputBaseSize() plus diagnostic log lines at the
extend-arm site and in the batch-candidates dump (gated on
mGestureDebugDrawPoints) to make the buffer state observable on-device.
- New tests: end-to-end fragment-pop -> reswipe simulation asserting the
re-armed base is the 2-point seed (27 merged pts, not ~51), and the
NOT_A_COORDINATE-skip case.
Validated on-device: th + ing + backspace + reswipe ing now yields "thing".
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Log proved the prior char-delete fix sat on the wrong path: deleting 'ng' from
'Thing' went through a selection / multi-char delete ('deleting 2 characters
before cursor'), not the single-char backspace branch. The next swipe armed its
extend base from the stale 27-point 'Thing' stroke (extend arm baseSize=27
composing='Thi') -> 'Whining'.
Rather than patch each of the many edit paths (single-char, fragment-pop,
whole-word, selection/multi-char delete, cursor re-compose), fix it at the single
consumption point: when a swipe arms its extend base in onStartBatchInput. Track
mComposingStrokeStale (set on ANY backspace, cleared when a gesture rebuilds the
buffer or the word ends). If stale at arm time, realign the base from the
authoritative composing text; otherwise keep the real captured stroke. This
preserves the validated continuous swipe+swipe and tap+swipe behavior exactly
(no delete -> not stale -> real stroke), while fixing every edit path at once.
realignComposerStrokeToText is now defensive: if the layout resolves no key
coordinates it leaves the buffer untouched rather than emptying it (which would
disarm the extend). The fragment-pop seed routes through the same helper.
Offline tests at the 12-failure baseline (no new failures; the 4 two-thumb
regressions from the first attempt are resolved by gating realign on an actual
edit). APK built.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
This was referenced Jun 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the long-standing two-thumb stale-stroke class where editing a gesture-built word and then swiping rebuilds an ever-longer word. Validated on-device.
The bug
After building a word with swipes/taps, editing it (backspace, fragment-pop, or selection/multi-char delete), then swiping to extend, the recognizer merged the stale pre-edit stroke geometry with the new swipe:
th+ swipeing→ "Thing", backspace theingfragment, re-swipeing→ "Thinking"ng→ "Thi" → swipeng→ "Whining"Root cause:
WordComposer.mInputPointers(the raw stroke buffer) is parallel state that a delete never shrinks, so it drifts out of sync with the composing text. A following swipe-extend armed its merged-trail base from that stale buffer.The fix (two commits)
tryFragmentBackspacerealign (19e445ed, was PR Fix fragment-pop stale stroke: realign raw buffer to truncated word #73): on a fragment-pop, rebuild the stroke buffer from the truncated word's key centers. Lowercases for the exact key lookup; skips unresolvable keys.5cac76ed): the general fix. TrackmComposingStrokeStale(set on any backspace, cleared when a gesture rebuilds the buffer or the word ends). At the single point the stroke is consumed as a swipe-extend base (onStartBatchInput), realign from the authoritative composing text only when stale — so continuous swipe+swipe and tap+swipe keep their real captured stroke, while every edit path (single-char, fragment-pop, selection / multi-char delete, cursor re-compose) is covered at once.realignComposerStrokeToTextis defensive: if the layout resolves no key coordinates it leaves the buffer untouched rather than disarming the extend.Validation
th+ing→backspace→reswipe → "thing"; "Thing"→deleteng→swipeng→ "thing". Theextend arm … realigned=true baseSize=3diagnostic confirms the base is rebuilt from text.ParserTestenv failures ontestOfflineRunTestsUnitTest; verified identical onmain).Supersedes #73 (its Phase 1 commit is included here).
🤖 Generated with Claude Code