From bdda7139090b0853089b87b9b0354d7faa105cba Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Sat, 6 Jun 2026 14:53:48 -0700 Subject: [PATCH 01/26] docs: adaptive typing design note (learned per-user key geometry) Captures the opt-in learned key-geometry feature: one per-user model behind both taps (KeyDetector) and gestures (ProximityInfo sweet spots, Java-only / no native rebuild), the context-prior + learned-geometry layers, hard caps to avoid wrong literals, privacy (content-free, incognito-gated), leantype.db persistence riding the existing backup, a stats page, and a phased build order. Co-Authored-By: Claude Opus 4.8 --- docs/ADAPTIVE_TYPING.md | 199 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 199 insertions(+) create mode 100644 docs/ADAPTIVE_TYPING.md diff --git a/docs/ADAPTIVE_TYPING.md b/docs/ADAPTIVE_TYPING.md new file mode 100644 index 000000000..895d1930a --- /dev/null +++ b/docs/ADAPTIVE_TYPING.md @@ -0,0 +1,199 @@ +# Adaptive Typing — learned per-user key geometry (taps + gestures) + +> Status: **design / in progress** (opt-in feature). Tracking issue: see the +> "Adaptive learned key geometry" issue on the fork. This note is the source of +> truth for the design; update it as the implementation evolves. + +## Goal + +Make the keyboard *feel like it learns how you type* — for both **tapping** and +**gesture/glide** input — by adapting each key's effective touch geometry to where +your finger actually lands and how the current context makes some keys more likely. +This is the technique popular keyboards use ("internally resize keys"); the academic +framing is a spatial model with a context prior (Bayesian touch). + +Explicitly **not** this feature: making autocorrect pick a better *word* after the +fact. We want it to feel like "the keyboard adapted to me," not "I still miss keys +but autocorrect cleans up." (That word-ranking idea is captured separately in +`SUGGESTION_RANKING.md`.) + +## How the engine works today (why this is feasible) + +Two independent geometry systems decide what your input becomes: + +1. **Literal tap → key** — `KeyDetector.detectHitKey` (`keyboard/KeyDetector.java`) + picks the key with the smallest edge-distance (`Key.squaredDistanceToEdge`). Pure + geometry, no weighting. The exact tapped letter is committed immediately; + mistakes are fixed downstream by autocorrect. +2. **Spatial model (gestures + tap-correction)** — the native recognizer scores + candidates using per-key **sweet spots** (effective center + radius). Those sweet + spots are **computed in Java** in `ProximityInfo.createNativeProximityInfo` + (`com/android/inputmethod/keyboard/ProximityInfo.java:168-220`) from key hit-box + centers + the static per-row `TouchPositionCorrection`, then pushed across the + existing JNI (`setProximityInfoNative`). + +**Key consequence:** because the sweet spots are produced in Java and passed through +the existing JNI, we can change what a swipe resolves to **without a native (C++) +rebuild** — we just feed adjusted centers/radii. This is what lets the feature reach +gesture compose, which is a hard requirement. + +There is no dynamic or learned key resizing today. A *static* per-row +`TouchPositionCorrection` exists (sweet-spot Y offset + radius per row); its +X-correction is disabled/obsolete. + +## The model: one learned model, two consumers + +A single per-user model, keyed by **(key, layout, orientation)**, stores content-free +geometry: + +``` +TOUCH_MODEL( + key_code, layout, orientation, + mean_dx, mean_dy, -- where you land relative to the key center (EMA) + var_dx, var_dy, -- how consistent you are (for confidence + radius scaling) + count, -- samples seen (gates confidence; powers the stats page) + updated_at +) +``` + +No characters, words, or sequences are ever stored — only aggregate geometry per key. + +Two parts read it: + +- **Taps** → `KeyDetector.detectHitKey`: measure distance to each key's **learned** + center (center + mean offset) instead of the raw center → borderline taps resolve + the way *you* type. +- **Gestures** → `ProximityInfo` sweet spots: shift each key's center by its learned + offset and scale its radius by your consistency → the recognizer matches your swipe + against keys positioned where your hand actually goes. + +Same model behind both ⇒ coherent "this is how I type." + +### Layer A — dynamic context prior (tap-only, ephemeral, stores nothing) + +Each keystroke, read the in-progress word's top-N completion candidates (the engine +already produces ~18 internally; the visible 3 is only a display limit) and project +them to a *next-character* distribution: for each candidate, take the char at the +current position weighted by the candidate's score; sum per letter. Example: typed +`H`, candidates `Hello`/`Hey`/`He` → `e` dominates → **E's tap target grows slightly +for the next tap.** Recomputed live; never persisted. + +For **gestures** there is no single "next key" to enlarge mid-stroke, so the context +prior is tap-only. The contextual-likelihood part for gestures is already handled by +the native language model (word-level). The new gesture win is Layer B (learned +geometry). + +### Layer B — learned per-user touch model (persisted, the "learning") + +- **Record:** on a *confident* tap (committed and not immediately backspaced / + autocorrected away), compute `dx = touchX - keyCenterX`, `dy = touchY - keyCenterY` + and fold into that key's running mean/variance via an exponential moving average + (recent behavior weighted more; old data decays). +- **Apply:** effective center = `center + mean_offset`; effective radius scales with + consistency (tighter for keys you nail, more forgiving for scattered keys). + +## Safety: never "I pressed W but it typed E" + +The literal tap is committed as-is, so the bias is **hard-capped** and confidence-gated: + +- Max center shift ≤ ~25% of key width/height (tunable via a strength slider). +- Radius scale bounded (e.g. 0.7–1.4×). +- A key's bias only ramps in after enough samples (`count`) and low-enough variance; + the applied magnitude scales with confidence — no sudden jumps. +- The context prior only breaks *ambiguous* taps within the cap band. **Beyond the cap + into a neighbor's territory, the neighbor always wins.** Strength = 0 ⇒ pure learned + geometry, no context flipping. + +## Privacy & security + +- Layer A persists nothing. +- Layer B persists only per-key geometry + counts — **no text content.** The only weak + signal is per-key `count` (letter-usage frequency), which never leaves the device. +- **Respect existing learning gates:** do not record while `mIncognitoModeEnabled` + (always-incognito pref, framework no-learning fields, or password fields) — the same + gate user-history learning uses. Opt-in master toggle on top. +- The settings backup already contains clipboard text + user dictionaries, so adding + content-free geometry does not widen the backup's sensitivity. We may still + quantize/round counts for extra caution. + +## Persistence, export/import, backup compatibility + +Local data lives in a single SQLite DB, `leantype.db` +(`latin/database/Database.kt`, raw `SQLiteOpenHelper`, currently VERSION 2; clipboard +lives there). The settings backup (`settings/preferences/BackupRestorePreference.kt`, +Advanced tab) **already zips the entire `leantype.db` and restores it**, and restore +is lenient (unknown zip entries skipped; missing columns handled by schema checks in +`Database.copyFromDb` + `onUpgrade`). + +Therefore: + +- **Single export/import path (satisfied for free):** add the touch model as a **new + table in `leantype.db`** → it is automatically part of the existing Advanced-tab + backup and restored the same way. No second mechanism. +- **Move behaviors to another device:** backup → restore. +- **Delete:** a "Reset learned typing model" button clears the table. +- **Don't break existing setting files:** purely additive — bump `Database.VERSION`, + add an `onUpgrade` that `CREATE`s the table; old code reads only tables it knows, so + an old backup (no table → created empty) and a new backup on an older app (unknown + table ignored) both restore safely. The format is unversioned but tolerant by design. + +## Stats / "your learned keyboard" page (Settings) + +A visualization page (trust + the reset control live here): + +- **Heatmap**: each key with its learned offset (arrow) and a variance/confidence + ellipse — literally "what your learned keyboard looks like." +- **Per-key accuracy stats**: + - *Consistency* = how tightly you cluster on the key (low variance). + - *Correction rate* = how often a tap on the key was immediately backspaced / + autocorrected away — an **approximate** accuracy proxy (we never know true intent). + e.g. "Z corrected ~18% vs E ~2%." +- **Reset** button. + +## Configurability + +- Master toggle (opt-in, default off). +- Strength slider (off → gentle tie-break → aggressive). +- Reset learned model. +- Honors incognito / no-learning fields. + +## Implementation footprint + +- **Gestures:** `ProximityInfo.createNativeProximityInfo` (`:185-198`) — add the + learned per-key offset to `sweetSpotCenterXs/Ys` and scale `sweetSpotRadii`; when the + feature is on, generate sweet spots from key centers even for layouts lacking + `TouchPositionCorrection` data (so it doesn't depend on the layout shipping it). + Re-push on keyboard reload / model update (learning is slow → no per-keystroke native + churn). **No C++ rebuild.** +- **Taps:** `KeyDetector.detectHitKey` — distance to the learned center + the capped + context prior tie-break. +- **Store:** new table in `leantype.db` + a DAO (follow `ClipboardDao`) + a + `TouchModelManager` that computes capped effective geometry and applies the EMA + update. +- **Learning hook:** record confident-tap offsets (incognito-gated) from the input + path. +- **Settings:** 5-file pref pattern (toggle + strength), reset, stats page. + +## Caveat to validate on-device + +The default sweet spots are tuned. We must confirm the learned shifts *improve* gesture +recognition rather than destabilize it — hence confidence-gating, caps, opt-in, and the +stats page to keep it honest. Validate the gesture path explicitly (it's the priority). + +## Phased build order + +1. **Foundation:** opt-in pref (5-file) + `leantype.db` table + DAO + `TouchModelManager` + (EMA update, capped effective-geometry API). Compiles; no behavior yet. +2. **Learning + gesture injection:** record confident taps; feed learned geometry into + `ProximityInfo` sweet spots (the gesture priority) + the capped `KeyDetector` + tie-break for taps. Testable on-device. +3. **Context prior (Layer A):** completion-derived next-char boost for taps. +4. **Stats page + reset UI + strength tuning.** + +## Open questions (tracked) + +1. Counts in backup: keep raw (slightly richer stats, faint usage signal) vs + quantize/omit. Current plan: keep (needed for the stats page), local-only + opt-in. +2. Learning scope confirmed: per layout + orientation (not per size/one-handed/floating + for v1). +3. Strength default and cap magnitude — tune on-device. From 056c0407a3a1dd202d48ec6287b24edde7a16f3f Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Sat, 6 Jun 2026 15:00:32 -0700 Subject: [PATCH 02/26] feat(typing): adaptive key geometry - foundation (pref + touch-model store) Phase 1 of #1 (no behavior yet; nothing reads the model). Adds the opt-in prefs PREF_ADAPTIVE_KEY_GEOMETRY (+ strength) via the 5-file pattern, and a content-free per-(key, layout, orientation) touch-model table in leantype.db with a cached DAO (EMA running mean/variance + count, restore, clear). DB VERSION 2->3 with an additive onUpgrade, and copyFromDb extended to carry the model across the existing settings backup/restore (guarded so older backups without the table restore fine). Compiles: :app:compileStandardDebugJavaWithJavac. Co-Authored-By: Claude Opus 4.8 --- .../keyboard/latin/database/Database.kt | 23 ++- .../keyboard/latin/database/TouchModelDao.kt | 169 ++++++++++++++++++ .../keyboard/latin/settings/Defaults.kt | 3 + .../keyboard/latin/settings/Settings.java | 5 + .../latin/settings/SettingsValues.java | 9 + 5 files changed, 208 insertions(+), 1 deletion(-) create mode 100644 app/src/main/java/helium314/keyboard/latin/database/TouchModelDao.kt diff --git a/app/src/main/java/helium314/keyboard/latin/database/Database.kt b/app/src/main/java/helium314/keyboard/latin/database/Database.kt index 0e9c785cd..15b1c8f5b 100644 --- a/app/src/main/java/helium314/keyboard/latin/database/Database.kt +++ b/app/src/main/java/helium314/keyboard/latin/database/Database.kt @@ -10,17 +10,21 @@ import java.io.File class Database private constructor(context: Context, name: String = NAME) : SQLiteOpenHelper(context, name, null, VERSION) { override fun onCreate(db: SQLiteDatabase) { db.execSQL(ClipboardDao.CREATE_TABLE) + db.execSQL(TouchModelDao.CREATE_TABLE) } override fun onUpgrade(db: SQLiteDatabase, oldVersion: Int, newVersion: Int) { if (oldVersion < 2) { db.execSQL("ALTER TABLE CLIPBOARD ADD COLUMN IMAGE_URI TEXT") } + if (oldVersion < 3) { + db.execSQL(TouchModelDao.CREATE_TABLE) + } } companion object { private val TAG = Database::class.java.simpleName - private const val VERSION = 2 + private const val VERSION = 3 const val NAME = "leantype.db" private var instance: Database? = null fun getInstance(context: Context): Database { @@ -55,6 +59,23 @@ class Database private constructor(context: Context, name: String = NAME) : SQLi clipDao.addClip(it.getLong(0), it.getInt(1) != 0, it.getString(2) ?: "", imageUri) } } + // Touch model (adaptive typing): present only in backups from versions that have it. + val touchDao = TouchModelDao.getInstance(context) + if (touchDao != null) { + val hasTouchModel = otherDb.readableDatabase.rawQuery( + "SELECT name FROM sqlite_master WHERE type='table' AND name='${TouchModelDao.TABLE}'", null + ).use { it.moveToNext() } + if (hasTouchModel) { + touchDao.clear() + otherDb.readableDatabase.rawQuery(TouchModelDao.SELECT_ALL, null).use { + while (it.moveToNext()) { + touchDao.restore(TouchModelDao.Stat( + it.getInt(0), it.getString(1), it.getInt(2), it.getFloat(3), it.getFloat(4), + it.getFloat(5), it.getFloat(6), it.getInt(7), it.getLong(8))) + } + } + } + } otherDb.close() file.delete() } diff --git a/app/src/main/java/helium314/keyboard/latin/database/TouchModelDao.kt b/app/src/main/java/helium314/keyboard/latin/database/TouchModelDao.kt new file mode 100644 index 000000000..d53f3f2ca --- /dev/null +++ b/app/src/main/java/helium314/keyboard/latin/database/TouchModelDao.kt @@ -0,0 +1,169 @@ +// SPDX-License-Identifier: GPL-3.0-only +package helium314.keyboard.latin.database + +import android.content.ContentValues +import android.content.Context +import android.database.sqlite.SQLiteDatabase +import helium314.keyboard.latin.utils.Log + +/** + * Cached access to the learned per-user touch-model table (see docs/ADAPTIVE_TYPING.md). + * + * Stores, per (key code, layout, orientation), the running mean landing offset relative to the + * key center and its variance, plus a sample count. **Content-free**: no characters/words are + * stored, only aggregate touch geometry. Used to bias tap resolution (KeyDetector) and gesture + * sweet spots (ProximityInfo) toward where the user actually types. + * + * Lives in the shared [Database] ("leantype.db"), so it rides the existing settings + * backup/restore. Lookups are O(1) from an in-memory cache for the input hot path. + */ +class TouchModelDao private constructor(private val db: Database) { + + /** One key's learned stats. Offsets/variance are in pixels (relative to the key center). */ + data class Stat( + val keyCode: Int, + val layout: String, + val orientation: Int, + var meanDx: Float, + var meanDy: Float, + var varDx: Float, + var varDy: Float, + var count: Int, + var updatedAt: Long, + ) + + private val cache = HashMap() + + init { + db.readableDatabase.query( + TABLE, + arrayOf(COLUMN_KEY_CODE, COLUMN_LAYOUT, COLUMN_ORIENTATION, COLUMN_MEAN_DX, COLUMN_MEAN_DY, + COLUMN_VAR_DX, COLUMN_VAR_DY, COLUMN_COUNT, COLUMN_UPDATED_AT), + null, null, null, null, null + ).use { + while (it.moveToNext()) { + val s = Stat(it.getInt(0), it.getString(1), it.getInt(2), it.getFloat(3), it.getFloat(4), + it.getFloat(5), it.getFloat(6), it.getInt(7), it.getLong(8)) + cache[key(s.keyCode, s.layout, s.orientation)] = s + } + } + } + + /** + * Fold a new landing offset (touch position minus key center, in px) into the running model + * for this key, using an exponential moving average so recent behavior is weighted more and + * old data decays. Persists immediately. Callers must gate on incognito + the opt-in pref. + */ + @Synchronized + fun record(keyCode: Int, layout: String, orientation: Int, dx: Float, dy: Float, now: Long) { + val k = key(keyCode, layout, orientation) + val s = cache[k] + if (s == null) { + val ns = Stat(keyCode, layout, orientation, dx, dy, 0f, 0f, 1, now) + cache[k] = ns + write(ns) + return + } + val a = EMA_ALPHA + val oldMeanDx = s.meanDx + val oldMeanDy = s.meanDy + s.meanDx = (1 - a) * oldMeanDx + a * dx + s.meanDy = (1 - a) * oldMeanDy + a * dy + // Exponentially-weighted variance around the pre-update mean. + s.varDx = (1 - a) * (s.varDx + a * (dx - oldMeanDx) * (dx - oldMeanDx)) + s.varDy = (1 - a) * (s.varDy + a * (dy - oldMeanDy) * (dy - oldMeanDy)) + if (s.count < Int.MAX_VALUE) s.count++ + s.updatedAt = now + write(s) + } + + /** Learned stats for a key, or null if none recorded yet. */ + @Synchronized + fun get(keyCode: Int, layout: String, orientation: Int): Stat? = + cache[key(keyCode, layout, orientation)] + + /** Snapshot of all stats (for the stats page / debugging). */ + @Synchronized + fun all(): List = cache.values.map { it.copy() } + + /** Insert a full stat verbatim (no EMA) — used when restoring from a backup. */ + @Synchronized + fun restore(s: Stat) { + cache[key(s.keyCode, s.layout, s.orientation)] = s + write(s) + } + + /** Forget everything (the "reset learned typing model" action). */ + @Synchronized + fun clear() { + if (cache.isEmpty()) return + cache.clear() + db.writableDatabase.delete(TABLE, null, null) + } + + private fun write(s: Stat) { + val cv = ContentValues(9) + cv.put(COLUMN_KEY_CODE, s.keyCode) + cv.put(COLUMN_LAYOUT, s.layout) + cv.put(COLUMN_ORIENTATION, s.orientation) + cv.put(COLUMN_MEAN_DX, s.meanDx) + cv.put(COLUMN_MEAN_DY, s.meanDy) + cv.put(COLUMN_VAR_DX, s.varDx) + cv.put(COLUMN_VAR_DY, s.varDy) + cv.put(COLUMN_COUNT, s.count) + cv.put(COLUMN_UPDATED_AT, s.updatedAt) + db.writableDatabase.insertWithOnConflict(TABLE, null, cv, SQLiteDatabase.CONFLICT_REPLACE) + } + + companion object { + private const val TAG = "TouchModelDao" + // ~ last 20 samples dominate; tuned later on-device. + private const val EMA_ALPHA = 0.05f + /** Below this many samples a key's learned bias should not be applied (confidence gate). */ + const val MIN_CONFIDENT_SAMPLES = 20 + + const val TABLE = "TOUCH_MODEL" + private const val COLUMN_KEY_CODE = "KEY_CODE" + private const val COLUMN_LAYOUT = "LAYOUT" + private const val COLUMN_ORIENTATION = "ORIENTATION" + private const val COLUMN_MEAN_DX = "MEAN_DX" + private const val COLUMN_MEAN_DY = "MEAN_DY" + private const val COLUMN_VAR_DX = "VAR_DX" + private const val COLUMN_VAR_DY = "VAR_DY" + private const val COLUMN_COUNT = "COUNT" + private const val COLUMN_UPDATED_AT = "UPDATED_AT" + const val CREATE_TABLE = """ + CREATE TABLE $TABLE ( + $COLUMN_KEY_CODE INTEGER NOT NULL, + $COLUMN_LAYOUT TEXT NOT NULL, + $COLUMN_ORIENTATION INTEGER NOT NULL, + $COLUMN_MEAN_DX REAL NOT NULL, + $COLUMN_MEAN_DY REAL NOT NULL, + $COLUMN_VAR_DX REAL NOT NULL, + $COLUMN_VAR_DY REAL NOT NULL, + $COLUMN_COUNT INTEGER NOT NULL, + $COLUMN_UPDATED_AT INTEGER NOT NULL, + PRIMARY KEY ($COLUMN_KEY_CODE, $COLUMN_LAYOUT, $COLUMN_ORIENTATION) + ) + """ + const val SELECT_ALL = + "SELECT $COLUMN_KEY_CODE, $COLUMN_LAYOUT, $COLUMN_ORIENTATION, $COLUMN_MEAN_DX, $COLUMN_MEAN_DY, " + + "$COLUMN_VAR_DX, $COLUMN_VAR_DY, $COLUMN_COUNT, $COLUMN_UPDATED_AT FROM $TABLE" + + private fun key(keyCode: Int, layout: String, orientation: Int) = "$keyCode|$layout|$orientation" + + private var instance: TouchModelDao? = null + + /** Returns the instance, or null if it can't be created (e.g. device locked). */ + @Synchronized + fun getInstance(context: Context): TouchModelDao? { + if (instance == null) + try { + instance = TouchModelDao(Database.getInstance(context)) + } catch (e: Throwable) { + Log.e(TAG, "can't create TouchModelDao", e) + } + return instance + } + } +} diff --git a/app/src/main/java/helium314/keyboard/latin/settings/Defaults.kt b/app/src/main/java/helium314/keyboard/latin/settings/Defaults.kt index b948de0b1..344c0b422 100644 --- a/app/src/main/java/helium314/keyboard/latin/settings/Defaults.kt +++ b/app/src/main/java/helium314/keyboard/latin/settings/Defaults.kt @@ -157,6 +157,9 @@ object Defaults { const val PREF_MULTIPART_FULL_WORD_SUGGESTIONS = true const val PREF_MULTIPART_TAP_SEED_GESTURE = true const val PREF_MULTIPART_RERECOGNIZE_TAPS = false + // Adaptive typing (opt-in). Off by default; strength is a 0..100 percentage of the cap. + const val PREF_ADAPTIVE_KEY_GEOMETRY = false + const val PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH = 50 const val PREF_SHOW_SETUP_WIZARD_ICON = true const val PREF_USE_CONTACTS = false const val PREF_USE_APPS = false diff --git a/app/src/main/java/helium314/keyboard/latin/settings/Settings.java b/app/src/main/java/helium314/keyboard/latin/settings/Settings.java index 1b07fb3e7..29d1830a7 100644 --- a/app/src/main/java/helium314/keyboard/latin/settings/Settings.java +++ b/app/src/main/java/helium314/keyboard/latin/settings/Settings.java @@ -183,6 +183,11 @@ public final class Settings implements SharedPreferences.OnSharedPreferenceChang // re-recognize the whole word, instead of literally appending it to a (possibly // mis-resolved) fragment. Makes a slow tap-after-swipe behave like a fast one. Default off. public static final String PREF_MULTIPART_RERECOGNIZE_TAPS = "multipart_rerecognize_taps"; + // Adaptive typing (opt-in): learn per-user key landing offsets and bias both tap resolution + // and gesture sweet-spots toward where the user actually types. Content-free (geometry only), + // incognito-gated. Strength scales the cap (0 = off). See docs/ADAPTIVE_TYPING.md. + public static final String PREF_ADAPTIVE_KEY_GEOMETRY = "adaptive_key_geometry"; + public static final String PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH = "adaptive_key_geometry_strength"; public static final String PREF_SHOW_SETUP_WIZARD_ICON = "show_setup_wizard_icon"; public static final String PREF_USE_CONTACTS = "use_contacts"; public static final String PREF_USE_APPS = "use_apps"; diff --git a/app/src/main/java/helium314/keyboard/latin/settings/SettingsValues.java b/app/src/main/java/helium314/keyboard/latin/settings/SettingsValues.java index e0081fac6..563929a6d 100644 --- a/app/src/main/java/helium314/keyboard/latin/settings/SettingsValues.java +++ b/app/src/main/java/helium314/keyboard/latin/settings/SettingsValues.java @@ -145,6 +145,9 @@ public class SettingsValues { public final boolean mMultipartFullWordSuggestions; public final boolean mMultipartTapSeedGesture; public final boolean mMultipartRerecognizeTaps; + // Adaptive typing (opt-in): learn where the user lands and bias tap/gesture key geometry. + public final boolean mAdaptiveKeyGeometry; + public final int mAdaptiveKeyGeometryStrength; public final boolean mSlidingKeyInputPreviewEnabled; public final int mKeyLongpressTimeout; public final boolean mEnableEmojiAltPhysicalKey; @@ -395,6 +398,12 @@ public SettingsValues(final Context context, final SharedPreferences prefs, fina mMultipartRerecognizeTaps = prefs.getBoolean( Settings.PREF_MULTIPART_RERECOGNIZE_TAPS, Defaults.PREF_MULTIPART_RERECOGNIZE_TAPS); + mAdaptiveKeyGeometry = prefs.getBoolean( + Settings.PREF_ADAPTIVE_KEY_GEOMETRY, + Defaults.PREF_ADAPTIVE_KEY_GEOMETRY); + mAdaptiveKeyGeometryStrength = prefs.getInt( + Settings.PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH, + Defaults.PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH); mSuggestionStripHiddenPerUserSettings = mToolbarMode == ToolbarMode.HIDDEN || mToolbarMode == ToolbarMode.TOOLBAR_KEYS; mOverrideShowingSuggestions = mInputAttributes.mMayOverrideShowingSuggestions From 73e4baf7090cc9bb109f950d61efd67b6cbe161c Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Sat, 6 Jun 2026 15:03:07 -0700 Subject: [PATCH 03/26] feat(typing): adaptive key geometry - TouchModelManager policy + tests Phase 2 policy layer for #1: capped, confidence- and strength-scaled landing-offset math (pure functions). The cap (MAX_SHIFT_FRACTION of key size) is the safety bound so a learned bias can never flip a clear press to a neighbor; confidence ramps the bias in with sample count; strength scales it (0 = off). 7 unit tests. Co-Authored-By: Claude Opus 4.8 --- .../latin/database/TouchModelManager.kt | 46 ++++++++++++++ .../latin/database/TouchModelManagerTest.kt | 60 +++++++++++++++++++ 2 files changed, 106 insertions(+) create mode 100644 app/src/main/java/helium314/keyboard/latin/database/TouchModelManager.kt create mode 100644 app/src/test/java/helium314/keyboard/latin/database/TouchModelManagerTest.kt diff --git a/app/src/main/java/helium314/keyboard/latin/database/TouchModelManager.kt b/app/src/main/java/helium314/keyboard/latin/database/TouchModelManager.kt new file mode 100644 index 000000000..62b814a33 --- /dev/null +++ b/app/src/main/java/helium314/keyboard/latin/database/TouchModelManager.kt @@ -0,0 +1,46 @@ +// SPDX-License-Identifier: GPL-3.0-only +package helium314.keyboard.latin.database + +/** + * Policy layer over [TouchModelDao]: turns a key's raw learned stats into a *capped*, + * confidence- and strength-scaled landing offset that the input paths apply + * (see docs/ADAPTIVE_TYPING.md). Pure functions — no Android / DB / threading — so the + * caps and ramp are unit-testable and the input hot paths just consume the result. + * + * The cap is the safety guarantee: a learned bias can shift a key's effective center by at + * most [MAX_SHIFT_FRACTION] of the key dimension, so a clearly-on-target press can never flip + * to a neighbor. The bias also ramps in with sample count (no sudden jumps from sparse data) + * and scales with the user's strength setting (0 = off). + */ +object TouchModelManager { + /** Max center shift as a fraction of the key's width/height. */ + const val MAX_SHIFT_FRACTION = 0.25f + /** Confidence reaches full strength at this many samples; it is 0 at/below MIN_CONFIDENT_SAMPLES. */ + const val FULL_CONFIDENCE_SAMPLES = 60 + + /** + * Capped, confidence- and strength-scaled landing offset for a key, in pixels. + * @return a fresh {dx, dy}; {0, 0} when there is not enough data or strength is 0. + */ + fun adjustedOffset(stat: TouchModelDao.Stat?, keyWidth: Int, keyHeight: Int, + strengthPercent: Int): FloatArray { + if (stat == null || strengthPercent <= 0 || keyWidth <= 0 || keyHeight <= 0) { + return floatArrayOf(0f, 0f) + } + val scale = confidence(stat.count) * (strengthPercent.coerceIn(0, 100) / 100f) + if (scale <= 0f) return floatArrayOf(0f, 0f) + val maxX = MAX_SHIFT_FRACTION * keyWidth + val maxY = MAX_SHIFT_FRACTION * keyHeight + val dx = (stat.meanDx * scale).coerceIn(-maxX, maxX) + val dy = (stat.meanDy * scale).coerceIn(-maxY, maxY) + return floatArrayOf(dx, dy) + } + + /** 0 below [TouchModelDao.MIN_CONFIDENT_SAMPLES], ramps linearly to 1 at [FULL_CONFIDENCE_SAMPLES]. */ + fun confidence(count: Int): Float { + val min = TouchModelDao.MIN_CONFIDENT_SAMPLES + if (count <= min) return 0f + if (count >= FULL_CONFIDENCE_SAMPLES) return 1f + return (count - min).toFloat() / (FULL_CONFIDENCE_SAMPLES - min) + } +} diff --git a/app/src/test/java/helium314/keyboard/latin/database/TouchModelManagerTest.kt b/app/src/test/java/helium314/keyboard/latin/database/TouchModelManagerTest.kt new file mode 100644 index 000000000..b815e83f2 --- /dev/null +++ b/app/src/test/java/helium314/keyboard/latin/database/TouchModelManagerTest.kt @@ -0,0 +1,60 @@ +// SPDX-License-Identifier: GPL-3.0-only +package helium314.keyboard.latin.database + +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlin.test.assertTrue + +/** Unit tests for the capped/confidence/strength policy in [TouchModelManager]. */ +class TouchModelManagerTest { + + private fun stat(meanDx: Float, meanDy: Float, count: Int) = + TouchModelDao.Stat(0x65 /* 'e' */, "qwerty", 0, meanDx, meanDy, 0f, 0f, count, 0L) + + @Test fun nullStatGivesNoOffset() { + val o = TouchModelManager.adjustedOffset(null, 100, 120, 100) + assertEquals(0f, o[0]); assertEquals(0f, o[1]) + } + + @Test fun belowConfidenceThresholdGivesNoOffset() { + // count <= MIN_CONFIDENT_SAMPLES => confidence 0 => no bias even with a large mean + val o = TouchModelManager.adjustedOffset(stat(40f, -30f, TouchModelDao.MIN_CONFIDENT_SAMPLES), 100, 120, 100) + assertEquals(0f, o[0]); assertEquals(0f, o[1]) + } + + @Test fun zeroStrengthGivesNoOffset() { + val o = TouchModelManager.adjustedOffset(stat(40f, -30f, 1000), 100, 120, 0) + assertEquals(0f, o[0]); assertEquals(0f, o[1]) + } + + @Test fun largeOffsetIsCappedToKeyFraction() { + // Full confidence + full strength, but a huge mean must be clamped to +/-25% of the key. + val o = TouchModelManager.adjustedOffset(stat(9999f, -9999f, 1000), 100, 120, 100) + assertEquals(TouchModelManager.MAX_SHIFT_FRACTION * 100, o[0], 0.001f) // +25 + assertEquals(-TouchModelManager.MAX_SHIFT_FRACTION * 120, o[1], 0.001f) // -30 + } + + @Test fun moderateOffsetPassesThroughWithinCap() { + // count=60 => confidence 1; strength 100% => full mean, well within the cap. + val o = TouchModelManager.adjustedOffset(stat(5f, -4f, TouchModelManager.FULL_CONFIDENCE_SAMPLES), 100, 120, 100) + assertEquals(5f, o[0], 0.001f) + assertEquals(-4f, o[1], 0.001f) + } + + @Test fun strengthScalesTheOffset() { + val full = TouchModelManager.adjustedOffset(stat(10f, 0f, 1000), 100, 120, 100)[0] + val half = TouchModelManager.adjustedOffset(stat(10f, 0f, 1000), 100, 120, 50)[0] + assertEquals(full / 2f, half, 0.001f) + } + + @Test fun confidenceRampIsMonotonic() { + val c20 = TouchModelManager.confidence(20) + val c40 = TouchModelManager.confidence(40) + val c60 = TouchModelManager.confidence(60) + val c100 = TouchModelManager.confidence(100) + assertEquals(0f, c20, 0.001f) + assertTrue(c40 > c20 && c40 < c60) + assertEquals(1f, c60, 0.001f) + assertEquals(1f, c100, 0.001f) + } +} From 094f484f973ef1ae439e18039f7fa9738eaf8641 Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Sat, 6 Jun 2026 19:00:09 -0700 Subject: [PATCH 04/26] feat(typing): adaptive key geometry - learning hook + gesture injection + settings Phase 2 of #1 (testable end-to-end). Closes the loop: - PointerTracker records each letter tap's landing offset into the touch model (opt-in + incognito gated; async DB write so typing stays fast). - ProximityInfo shifts per-key sweet spots by the capped learned offset, so gesture recognition AND tap-correction follow where the user actually types. Generated even when a layout has no static touch-position-correction data. Element id passed from Keyboard; computed in Java, crosses the existing JNI - no native change. - Gesture typing -> Advanced: opt-in toggle + strength slider (reload keyboard on change). - @JvmStatic on the Kotlin DAO factory / manager so the Java hot paths can call them. Deferred: literal-tap KeyDetector tie-break (marginal/risky), the completion-derived context prior, and the stats page. Builds: :app:assembleStandardDebug. Co-Authored-By: Claude Opus 4.8 --- .../inputmethod/keyboard/ProximityInfo.java | 52 ++++++++++++++++--- .../helium314/keyboard/keyboard/Keyboard.java | 2 +- .../keyboard/keyboard/PointerTracker.java | 22 ++++++++ .../keyboard/latin/database/TouchModelDao.kt | 19 ++++++- .../latin/database/TouchModelManager.kt | 1 + .../settings/screens/GestureTypingScreen.kt | 18 +++++++ app/src/main/res/values/strings.xml | 4 ++ 7 files changed, 108 insertions(+), 10 deletions(-) diff --git a/app/src/main/java/com/android/inputmethod/keyboard/ProximityInfo.java b/app/src/main/java/com/android/inputmethod/keyboard/ProximityInfo.java index 9015f3e6d..04a974753 100644 --- a/app/src/main/java/com/android/inputmethod/keyboard/ProximityInfo.java +++ b/app/src/main/java/com/android/inputmethod/keyboard/ProximityInfo.java @@ -6,6 +6,7 @@ package com.android.inputmethod.keyboard; +import android.content.Context; import android.graphics.Rect; import helium314.keyboard.latin.utils.Log; @@ -14,6 +15,10 @@ import helium314.keyboard.keyboard.Key; import helium314.keyboard.keyboard.internal.TouchPositionCorrection; import helium314.keyboard.latin.common.Constants; +import helium314.keyboard.latin.database.TouchModelDao; +import helium314.keyboard.latin.database.TouchModelManager; +import helium314.keyboard.latin.settings.Settings; +import helium314.keyboard.latin.settings.SettingsValues; import helium314.keyboard.latin.utils.JniUtils; import java.util.ArrayList; @@ -48,12 +53,17 @@ public class ProximityInfo { private final List mSortedKeys; @NonNull private final List[] mGridNeighbors; + // Adaptive typing: the keyboard element this proximity info is for (e.g. alphabet vs symbols). + // Keys the learned touch model by (key, this element, orientation) so layouts stay separate. + private final int mLayoutElementId; @SuppressWarnings("unchecked") public ProximityInfo(final int gridWidth, final int gridHeight, final int minWidth, final int height, final int mostCommonKeyWidth, final int mostCommonKeyHeight, @NonNull final List sortedKeys, - @NonNull final TouchPositionCorrection touchPositionCorrection) { + @NonNull final TouchPositionCorrection touchPositionCorrection, + final int layoutElementId) { + mLayoutElementId = layoutElementId; mGridWidth = gridWidth; mGridHeight = gridHeight; mGridSize = mGridWidth * mGridHeight; @@ -165,14 +175,34 @@ private long createNativeProximityInfo(@NonNull final TouchPositionCorrection to infoIndex++; } - if (touchPositionCorrection.isValid()) { + // Adaptive typing (opt-in): when enabled, shift each key's sweet-spot center by the + // learned per-user landing offset (capped) so the native recognizer matches swipes — and + // tap-correction candidates — against keys where the user's hand actually goes. This + // path runs even when the layout ships no static touch-position-correction data, so we + // still generate sweet spots in that case. The shift is computed in Java and crosses the + // existing JNI; no native change. See docs/ADAPTIVE_TYPING.md. + final boolean tpcValid = touchPositionCorrection.isValid(); + final SettingsValues sv = Settings.getValues(); + final boolean adaptiveOn = sv != null && sv.mAdaptiveKeyGeometry; + final int adaptiveStrength = sv != null ? sv.mAdaptiveKeyGeometryStrength : 0; + TouchModelDao adaptiveDao = null; + int adaptiveOrientation = 0; + if (adaptiveOn && adaptiveStrength > 0) { + final Context ctx = Settings.getCurrentContext(); + if (ctx != null) { + adaptiveDao = TouchModelDao.getInstance(ctx); + adaptiveOrientation = ctx.getResources().getConfiguration().orientation; + } + } + final boolean applyAdaptive = adaptiveDao != null; + if (tpcValid || applyAdaptive) { if (DEBUG) { - Log.d(TAG, "touchPositionCorrection: ON"); + Log.d(TAG, "sweet spots: ON (tpc=" + tpcValid + " adaptive=" + applyAdaptive + ")"); } sweetSpotCenterXs = new float[keyCount]; sweetSpotCenterYs = new float[keyCount]; sweetSpotRadii = new float[keyCount]; - final int rows = touchPositionCorrection.getRows(); + final int rows = tpcValid ? touchPositionCorrection.getRows() : 0; final float defaultRadius = DEFAULT_TOUCH_POSITION_CORRECTION_RADIUS * (float)Math.hypot(mMostCommonKeyWidth, mMostCommonKeyHeight); for (int infoIndex = 0, keyIndex = 0; keyIndex < sortedKeys.size(); keyIndex++) { @@ -186,7 +216,7 @@ private long createNativeProximityInfo(@NonNull final TouchPositionCorrection to sweetSpotCenterYs[infoIndex] = hitBox.exactCenterY(); sweetSpotRadii[infoIndex] = defaultRadius; final int row = hitBox.top / mMostCommonKeyHeight; - if (row < rows) { + if (tpcValid && row < rows) { final int hitBoxWidth = hitBox.width(); final int hitBoxHeight = hitBox.height(); final float hitBoxDiagonal = (float)Math.hypot(hitBoxWidth, hitBoxHeight); @@ -197,11 +227,19 @@ private long createNativeProximityInfo(@NonNull final TouchPositionCorrection to sweetSpotRadii[infoIndex] = touchPositionCorrection.getRadius(row) * hitBoxDiagonal; } + if (applyAdaptive) { + final TouchModelDao.Stat stat = adaptiveDao.get(key.getCode(), + Integer.toString(mLayoutElementId), adaptiveOrientation); + final float[] off = TouchModelManager.adjustedOffset( + stat, key.getWidth(), key.getHeight(), adaptiveStrength); + sweetSpotCenterXs[infoIndex] += off[0]; + sweetSpotCenterYs[infoIndex] += off[1]; + } if (DEBUG) { Log.d(TAG, String.format(Locale.US, " [%2d] row=%d x/y/r=%7.2f/%7.2f/%5.2f %s code=%s", infoIndex, row, sweetSpotCenterXs[infoIndex], sweetSpotCenterYs[infoIndex], - sweetSpotRadii[infoIndex], (row < rows ? "correct" : "default"), + sweetSpotRadii[infoIndex], (tpcValid && row < rows ? "correct" : "default"), Constants.printableCode(key.getCode()))); } infoIndex++; @@ -209,7 +247,7 @@ private long createNativeProximityInfo(@NonNull final TouchPositionCorrection to } else { sweetSpotCenterXs = sweetSpotCenterYs = sweetSpotRadii = null; if (DEBUG) { - Log.d(TAG, "touchPositionCorrection: OFF"); + Log.d(TAG, "sweet spots: OFF"); } } diff --git a/app/src/main/java/helium314/keyboard/keyboard/Keyboard.java b/app/src/main/java/helium314/keyboard/keyboard/Keyboard.java index c89423538..807c33828 100644 --- a/app/src/main/java/helium314/keyboard/keyboard/Keyboard.java +++ b/app/src/main/java/helium314/keyboard/keyboard/Keyboard.java @@ -113,7 +113,7 @@ public Keyboard(@NonNull final KeyboardParams params) { mProximityInfo = new ProximityInfo(params.GRID_WIDTH, params.GRID_HEIGHT, mOccupiedWidth, mOccupiedHeight, mMostCommonKeyWidth, mMostCommonKeyHeight, - mSortedKeys, params.mTouchPositionCorrection); + mSortedKeys, params.mTouchPositionCorrection, mId.mElementId); mProximityCharsCorrectionEnabled = params.mProximityCharsCorrectionEnabled; } diff --git a/app/src/main/java/helium314/keyboard/keyboard/PointerTracker.java b/app/src/main/java/helium314/keyboard/keyboard/PointerTracker.java index 427b88e6b..23943c6b9 100644 --- a/app/src/main/java/helium314/keyboard/keyboard/PointerTracker.java +++ b/app/src/main/java/helium314/keyboard/keyboard/PointerTracker.java @@ -1591,6 +1591,7 @@ eventTime, getActivePointerTrackerCount(), graceMs, this, sLastLetterTapTime = eventTime; sLastLetterTapCodepoint = code; pushTapDebugPoint(mKeyX, mKeyY, mPointerId, eventTime); + recordAdaptiveTouchSample(currentKey, mKeyX, mKeyY); } } if (isInSlidingKeyInput) { @@ -1598,6 +1599,27 @@ eventTime, getActivePointerTrackerCount(), graceMs, this, } } + // Adaptive typing (opt-in, see docs/ADAPTIVE_TYPING.md): fold this letter tap's landing + // offset (touch minus key center) into the learned per-(key, layout, orientation) model so + // the spatial model can later bias gesture sweet-spots toward where the user actually types. + // Content-free (geometry only); gated on the opt-in pref and incognito. The DAO write is async. + private void recordAdaptiveTouchSample(final Key key, final int x, final int y) { + if (key == null || x < 0 || y < 0 || mKeyboard == null) return; + final SettingsValues sv = Settings.getValues(); + if (sv == null || !sv.mAdaptiveKeyGeometry || sv.mIncognitoModeEnabled) return; + final android.content.Context context = Settings.getCurrentContext(); + if (context == null) return; + final helium314.keyboard.latin.database.TouchModelDao dao = + helium314.keyboard.latin.database.TouchModelDao.getInstance(context); + if (dao == null) return; + final android.graphics.Rect hitBox = key.getHitBox(); + final float dx = x - hitBox.exactCenterX(); + final float dy = y - hitBox.exactCenterY(); + dao.record(key.getCode(), Integer.toString(mKeyboard.mId.mElementId), + context.getResources().getConfiguration().orientation, dx, dy, + System.currentTimeMillis()); + } + @Override public void cancelTrackingForAction() { if (isShowingPopupKeysPanel()) { diff --git a/app/src/main/java/helium314/keyboard/latin/database/TouchModelDao.kt b/app/src/main/java/helium314/keyboard/latin/database/TouchModelDao.kt index d53f3f2ca..951acf5fc 100644 --- a/app/src/main/java/helium314/keyboard/latin/database/TouchModelDao.kt +++ b/app/src/main/java/helium314/keyboard/latin/database/TouchModelDao.kt @@ -5,6 +5,7 @@ import android.content.ContentValues import android.content.Context import android.database.sqlite.SQLiteDatabase import helium314.keyboard.latin.utils.Log +import java.util.concurrent.Executors /** * Cached access to the learned per-user touch-model table (see docs/ADAPTIVE_TYPING.md). @@ -33,6 +34,11 @@ class TouchModelDao private constructor(private val db: Database) { ) private val cache = HashMap() + // Persist off the input thread: record() runs on every letter tap, so the DB write must not + // block typing. The cache (source of truth at runtime) is updated synchronously; the disk + // write is serialized on this single thread (order preserved). Losing the last sample or two + // on a crash is acceptable for a learning model. + private val writeExecutor = Executors.newSingleThreadExecutor() init { db.readableDatabase.query( @@ -61,7 +67,7 @@ class TouchModelDao private constructor(private val db: Database) { if (s == null) { val ns = Stat(keyCode, layout, orientation, dx, dy, 0f, 0f, 1, now) cache[k] = ns - write(ns) + persistAsync(ns.copy()) return } val a = EMA_ALPHA @@ -74,7 +80,15 @@ class TouchModelDao private constructor(private val db: Database) { s.varDy = (1 - a) * (s.varDy + a * (dy - oldMeanDy) * (dy - oldMeanDy)) if (s.count < Int.MAX_VALUE) s.count++ s.updatedAt = now - write(s) + persistAsync(s.copy()) + } + + private fun persistAsync(snapshot: Stat) { + try { + writeExecutor.execute { write(snapshot) } + } catch (e: Throwable) { + Log.e(TAG, "touch model write rejected", e) + } } /** Learned stats for a key, or null if none recorded yet. */ @@ -155,6 +169,7 @@ class TouchModelDao private constructor(private val db: Database) { private var instance: TouchModelDao? = null /** Returns the instance, or null if it can't be created (e.g. device locked). */ + @JvmStatic @Synchronized fun getInstance(context: Context): TouchModelDao? { if (instance == null) diff --git a/app/src/main/java/helium314/keyboard/latin/database/TouchModelManager.kt b/app/src/main/java/helium314/keyboard/latin/database/TouchModelManager.kt index 62b814a33..1918f91b3 100644 --- a/app/src/main/java/helium314/keyboard/latin/database/TouchModelManager.kt +++ b/app/src/main/java/helium314/keyboard/latin/database/TouchModelManager.kt @@ -22,6 +22,7 @@ object TouchModelManager { * Capped, confidence- and strength-scaled landing offset for a key, in pixels. * @return a fresh {dx, dy}; {0, 0} when there is not enough data or strength is 0. */ + @JvmStatic fun adjustedOffset(stat: TouchModelDao.Stat?, keyWidth: Int, keyHeight: Int, strengthPercent: Int): FloatArray { if (stat == null || strengthPercent <= 0 || keyWidth <= 0 || keyHeight <= 0) { diff --git a/app/src/main/java/helium314/keyboard/settings/screens/GestureTypingScreen.kt b/app/src/main/java/helium314/keyboard/settings/screens/GestureTypingScreen.kt index 489b036a4..b2930a362 100644 --- a/app/src/main/java/helium314/keyboard/settings/screens/GestureTypingScreen.kt +++ b/app/src/main/java/helium314/keyboard/settings/screens/GestureTypingScreen.kt @@ -73,6 +73,9 @@ fun GestureTypingScreen( add(Settings.PREF_SPACE_VERTICAL_SWIPE) add(Settings.PREF_TOUCHPAD_SENSITIVITY) add(Settings.PREF_DELETE_SWIPE) + add(Settings.PREF_ADAPTIVE_KEY_GEOMETRY) + if (prefs.getBoolean(Settings.PREF_ADAPTIVE_KEY_GEOMETRY, Defaults.PREF_ADAPTIVE_KEY_GEOMETRY)) + add(Settings.PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH) add(Settings.PREF_SHORTCUT_ROWS) if (prefs.getBoolean(Settings.PREF_SHORTCUT_ROWS, Defaults.PREF_SHORTCUT_ROWS)) { add(Settings.PREF_SHORTCUT_TOP_ROW) @@ -172,6 +175,21 @@ fun createGestureTypingSettings(context: Context) = listOf( Setting(context, Settings.PREF_DELETE_SWIPE, R.string.delete_swipe, R.string.delete_swipe_summary) { SwitchPreference(it, Defaults.PREF_DELETE_SWIPE) }, + Setting(context, Settings.PREF_ADAPTIVE_KEY_GEOMETRY, + R.string.adaptive_key_geometry, R.string.adaptive_key_geometry_summary) { + SwitchPreference(it, Defaults.PREF_ADAPTIVE_KEY_GEOMETRY) { + KeyboardSwitcher.getInstance().setThemeNeedsReload() // rebuild keyboard so sweet spots refresh + } + }, + Setting(context, Settings.PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH, R.string.adaptive_key_geometry_strength) { def -> + SliderPreference( + name = def.title, + key = def.key, + default = Defaults.PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH, + range = 0f..100f, + description = { it.toString() } + ) { KeyboardSwitcher.getInstance().setThemeNeedsReload() } + }, Setting(context, Settings.PREF_SHORTCUT_ROWS, R.string.shortcut_rows, R.string.shortcut_rows_summary) { SwitchPreference(it, Defaults.PREF_SHORTCUT_ROWS) }, diff --git a/app/src/main/res/values/strings.xml b/app/src/main/res/values/strings.xml index da85fb934..b613493d6 100644 --- a/app/src/main/res/values/strings.xml +++ b/app/src/main/res/values/strings.xml @@ -367,6 +367,10 @@ Delete swipe Perform a swipe from the delete key to select and remove bigger portions of text at once + + Adaptive key geometry (experimental) + Learn where you actually tap each key and nudge gesture and tap recognition toward your style. Stored on-device only, no text is saved. + Adaptive geometry strength Shortcut rows Enable temporary layout-defined shortcut rows opened by vertical swipes Top shortcut row swipe From 00ede4ff7236dbf0a4cea140fb620bf0fddda2fc Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Sun, 7 Jun 2026 02:07:43 +0000 Subject: [PATCH 05/26] chore: update README badges [skip ci] --- docs/badges/downloads.svg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/badges/downloads.svg b/docs/badges/downloads.svg index 59a40ea91..044abfbad 100644 --- a/docs/badges/downloads.svg +++ b/docs/badges/downloads.svg @@ -1 +1 @@ -DownloadsDownloads2817928179 +DownloadsDownloads2886228862 From 2ef0d0f1b79a3e7cf1b72790495e4053fa1624f0 Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Sat, 6 Jun 2026 20:16:25 -0700 Subject: [PATCH 06/26] feat(typing): gesture-endpoint learning + learned-typing-model stats page #1, follow-ups to the adaptive-geometry feature: - Gestures now teach the model (not just consume it) via their clean endpoints: finger-down -> word's first letter, finger-up -> last letter. Interior keys skipped (corner-cutting); fresh single strokes only. InputLogic.maybeRecordGestureEndpoints. - New stats screen (AdaptiveTypingStatsScreen) reachable from Gesture typing -> Advanced when the feature is on: per-key average offset, spread (consistency), sample count, plus a Reset button. Wired via SettingsDestination + a clickable Preference row. Build + SettingsContainerTest green; :app:assembleStandardDebug builds. Co-Authored-By: Claude Opus 4.8 --- .../keyboard/latin/inputlogic/InputLogic.java | 40 +++++++ .../keyboard/settings/SettingsContainer.kt | 2 + .../keyboard/settings/SettingsNavHost.kt | 5 + .../screens/AdaptiveTypingStatsScreen.kt | 100 ++++++++++++++++++ .../settings/screens/GestureTypingScreen.kt | 17 ++- app/src/main/res/values/strings.xml | 5 + docs/ADAPTIVE_TYPING.md | 30 ++++-- 7 files changed, 187 insertions(+), 12 deletions(-) create mode 100644 app/src/main/java/helium314/keyboard/settings/screens/AdaptiveTypingStatsScreen.kt diff --git a/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java b/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java index af91356e7..0faf3cb7e 100644 --- a/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java +++ b/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java @@ -3909,6 +3909,9 @@ public void onUpdateTailBatchInputCompleted(final SettingsValues settingsValues, if (settingsValues.mMultipartRerecognizeTaps) { mLiveStroke.set(mWordComposer.getInputPointers()); } + // Adaptive typing: learn this gesture's clean endpoints so swipes teach the model too. + maybeRecordGestureEndpoints(settingsValues, composedText, extendExistingCompose, + usedMergedTrail, keyboardSwitcher); mWordComposer.setBatchInputWord(composedText); setComposingTextInternal(composedText, 1); if (extendExistingCompose) { @@ -3973,6 +3976,43 @@ public void onUpdateTailBatchInputCompleted(final SettingsValues settingsValues, enterCombiningMode(settingsValues, false /* fromTap, unused — kept for clarity */); } + // Adaptive typing (opt-in, see docs/ADAPTIVE_TYPING.md): a gesture's first point (finger-down) + // and last point (finger-up) are clean "I aimed here" samples for the word's first and last + // letters — unlike interior keys, which suffer corner-cutting. Fold those two offsets into the + // learned model so swipes teach it too. Only for fresh single strokes (merged/extended trails + // have ambiguous endpoints). Gated on the opt-in pref + incognito; the DAO write is async. + private void maybeRecordGestureEndpoints(final SettingsValues sv, final String word, + final boolean extend, final boolean usedMergedTrail, + final KeyboardSwitcher keyboardSwitcher) { + if (sv == null || !sv.mAdaptiveKeyGeometry || sv.mIncognitoModeEnabled) return; + if (extend || usedMergedTrail || word == null || word.isEmpty()) return; + final InputPointers pts = mWordComposer.getInputPointers(); + final int n = pts.getPointerSize(); + if (n < 2) return; // need a real stroke; taps are handled in PointerTracker + final Keyboard keyboard = keyboardSwitcher.getKeyboard(); + if (keyboard == null) return; + final int[] xs = pts.getXCoordinates(); + final int[] ys = pts.getYCoordinates(); + recordGestureEndpoint(keyboard, Character.toLowerCase(word.codePointAt(0)), xs[0], ys[0]); + recordGestureEndpoint(keyboard, Character.toLowerCase(word.codePointBefore(word.length())), + xs[n - 1], ys[n - 1]); + } + + private void recordGestureEndpoint(final Keyboard keyboard, final int codePoint, + final int x, final int y) { + if (!Character.isLetter(codePoint) || x < 0 || y < 0) return; + final helium314.keyboard.keyboard.Key key = keyboard.getKey(codePoint); + if (key == null) return; + final helium314.keyboard.latin.database.TouchModelDao dao = + helium314.keyboard.latin.database.TouchModelDao.getInstance(mLatinIME); + if (dao == null) return; + final android.graphics.Rect hitBox = key.getHitBox(); + dao.record(codePoint, Integer.toString(keyboard.mId.mElementId), + mLatinIME.getResources().getConfiguration().orientation, + x - hitBox.exactCenterX(), y - hitBox.exactCenterY(), + System.currentTimeMillis()); + } + /** * Commit the typed string to the editor. *

diff --git a/app/src/main/java/helium314/keyboard/settings/SettingsContainer.kt b/app/src/main/java/helium314/keyboard/settings/SettingsContainer.kt index 5dc52e6a5..6f4097e32 100644 --- a/app/src/main/java/helium314/keyboard/settings/SettingsContainer.kt +++ b/app/src/main/java/helium314/keyboard/settings/SettingsContainer.kt @@ -96,6 +96,8 @@ object SettingsWithoutKey { const val LOAD_GESTURE_LIB = "load_gesture_library" const val TWO_THUMB_SPACING_MODE = "two_thumb_spacing_mode" const val TWO_THUMB_BACKSPACE_BEHAVIOR = "two_thumb_backspace_behavior" + const val ADAPTIVE_TYPING_STATS = "adaptive_typing_stats" // entry row (navigates) + const val ADAPTIVE_TYPING_STATS_CONTENT = "adaptive_typing_stats_content" // the stats body const val BACKGROUND_IMAGE = "background_image" const val BACKGROUND_IMAGE_LANDSCAPE = "background_image_landscape" const val CUSTOM_FONT = "custom_font" diff --git a/app/src/main/java/helium314/keyboard/settings/SettingsNavHost.kt b/app/src/main/java/helium314/keyboard/settings/SettingsNavHost.kt index 9b8b9952a..eea1a388d 100644 --- a/app/src/main/java/helium314/keyboard/settings/SettingsNavHost.kt +++ b/app/src/main/java/helium314/keyboard/settings/SettingsNavHost.kt @@ -18,6 +18,7 @@ import helium314.keyboard.latin.settings.SettingsSubtype.Companion.toSettingsSub import helium314.keyboard.latin.settings.getTransitionAnimationScale import helium314.keyboard.settings.screens.AIIntegrationScreen import helium314.keyboard.settings.screens.AboutScreen +import helium314.keyboard.settings.screens.AdaptiveTypingStatsScreen import helium314.keyboard.settings.screens.AdvancedSettingsScreen import helium314.keyboard.settings.screens.AppearanceScreen import helium314.keyboard.settings.screens.ColorsScreen @@ -106,6 +107,9 @@ fun SettingsNavHost( composable(SettingsDestination.TwoThumbTyping) { TwoThumbTypingScreen(onClickBack = ::goBack) } + composable(SettingsDestination.AdaptiveTypingStats) { + AdaptiveTypingStatsScreen(onClickBack = ::goBack) + } composable(SettingsDestination.Advanced) { AdvancedSettingsScreen(onClickBack = ::goBack) } @@ -183,6 +187,7 @@ object SettingsDestination { const val Toolbar = "toolbar" const val GestureTyping = "gesture_typing" const val TwoThumbTyping = "two_thumb_typing" + const val AdaptiveTypingStats = "adaptive_typing_stats_screen" const val Advanced = "advanced" const val Libraries = "libraries_hub" const val AIIntegration = "ai_integration" diff --git a/app/src/main/java/helium314/keyboard/settings/screens/AdaptiveTypingStatsScreen.kt b/app/src/main/java/helium314/keyboard/settings/screens/AdaptiveTypingStatsScreen.kt new file mode 100644 index 000000000..7aa78442b --- /dev/null +++ b/app/src/main/java/helium314/keyboard/settings/screens/AdaptiveTypingStatsScreen.kt @@ -0,0 +1,100 @@ +// SPDX-License-Identifier: GPL-3.0-only +package helium314.keyboard.settings.screens + +import android.content.Context +import androidx.compose.foundation.layout.Arrangement +import androidx.compose.foundation.layout.Column +import androidx.compose.foundation.layout.Row +import androidx.compose.foundation.layout.fillMaxWidth +import androidx.compose.foundation.layout.padding +import androidx.compose.material3.Button +import androidx.compose.material3.MaterialTheme +import androidx.compose.material3.Text +import androidx.compose.runtime.Composable +import androidx.compose.runtime.getValue +import androidx.compose.runtime.mutableStateOf +import androidx.compose.runtime.remember +import androidx.compose.runtime.setValue +import androidx.compose.ui.Modifier +import androidx.compose.ui.platform.LocalContext +import androidx.compose.ui.res.stringResource +import androidx.compose.ui.text.font.FontFamily +import androidx.compose.ui.unit.dp +import helium314.keyboard.latin.R +import helium314.keyboard.latin.database.TouchModelDao +import helium314.keyboard.settings.SearchSettingsScreen +import helium314.keyboard.settings.SettingsWithoutKey +import kotlin.math.sqrt + +/** + * "Learned typing model" page: shows what the adaptive-key-geometry feature has learned per key + * (mean landing offset, spread/consistency, sample count) and lets the user reset it. Reuses the + * standard settings scaffold by rendering one content [Setting]; see [AdaptiveTypingStatsContent]. + */ +@Composable +fun AdaptiveTypingStatsScreen(onClickBack: () -> Unit) { + SearchSettingsScreen( + onClickBack = onClickBack, + title = stringResource(R.string.adaptive_key_geometry_stats_title), + settings = listOf(SettingsWithoutKey.ADAPTIVE_TYPING_STATS_CONTENT) + ) +} + +private data class StatRow(val label: String, val dx: Int, val dy: Int, val spread: Int, val count: Int) + +private fun loadStatRows(context: Context): List { + val dao = TouchModelDao.getInstance(context) ?: return emptyList() + return dao.all() + .sortedByDescending { it.count } + .map { s -> + val letter = try { String(Character.toChars(s.keyCode)) } catch (e: Throwable) { "?" } + val orient = if (s.orientation == 2) "L" else "P" // 2 == landscape + val spread = sqrt(((s.varDx + s.varDy) / 2f).coerceAtLeast(0f)).toInt() + StatRow("$letter ($orient)", s.meanDx.toInt(), s.meanDy.toInt(), spread, s.count) + } +} + +/** The dynamic stats body. Rendered as the content of a single registered Setting. */ +@Composable +fun AdaptiveTypingStatsContent() { + val context = LocalContext.current + var rows by remember { mutableStateOf(loadStatRows(context)) } + Column(Modifier.fillMaxWidth().padding(horizontal = 12.dp, vertical = 8.dp)) { + Text( + stringResource(R.string.adaptive_key_geometry_stats_explanation), + style = MaterialTheme.typography.bodySmall, + color = MaterialTheme.colorScheme.onSurfaceVariant, + modifier = Modifier.padding(bottom = 12.dp) + ) + if (rows.isEmpty()) { + Text( + stringResource(R.string.adaptive_key_geometry_stats_empty), + modifier = Modifier.padding(vertical = 8.dp) + ) + } else { + rows.forEach { r -> + Row( + modifier = Modifier.fillMaxWidth().padding(vertical = 3.dp), + horizontalArrangement = Arrangement.SpaceBetween + ) { + Text(r.label, fontFamily = FontFamily.Monospace, style = MaterialTheme.typography.bodyLarge) + Text( + "Δ(${r.dx}, ${r.dy})px ±${r.spread}px n=${r.count}", + fontFamily = FontFamily.Monospace, + style = MaterialTheme.typography.bodyMedium, + color = MaterialTheme.colorScheme.onSurfaceVariant + ) + } + } + } + Button( + onClick = { + TouchModelDao.getInstance(context)?.clear() + rows = emptyList() + }, + modifier = Modifier.padding(top = 16.dp) + ) { + Text(stringResource(R.string.adaptive_key_geometry_reset)) + } + } +} diff --git a/app/src/main/java/helium314/keyboard/settings/screens/GestureTypingScreen.kt b/app/src/main/java/helium314/keyboard/settings/screens/GestureTypingScreen.kt index b2930a362..ab7e3912a 100644 --- a/app/src/main/java/helium314/keyboard/settings/screens/GestureTypingScreen.kt +++ b/app/src/main/java/helium314/keyboard/settings/screens/GestureTypingScreen.kt @@ -19,6 +19,8 @@ import helium314.keyboard.latin.utils.prefs import helium314.keyboard.settings.Setting import helium314.keyboard.settings.SearchSettingsScreen import helium314.keyboard.settings.SettingsActivity +import helium314.keyboard.settings.SettingsDestination +import helium314.keyboard.settings.preferences.Preference import helium314.keyboard.settings.preferences.SliderPreference import helium314.keyboard.settings.preferences.SwitchPreference import helium314.keyboard.settings.Theme @@ -74,8 +76,10 @@ fun GestureTypingScreen( add(Settings.PREF_TOUCHPAD_SENSITIVITY) add(Settings.PREF_DELETE_SWIPE) add(Settings.PREF_ADAPTIVE_KEY_GEOMETRY) - if (prefs.getBoolean(Settings.PREF_ADAPTIVE_KEY_GEOMETRY, Defaults.PREF_ADAPTIVE_KEY_GEOMETRY)) + if (prefs.getBoolean(Settings.PREF_ADAPTIVE_KEY_GEOMETRY, Defaults.PREF_ADAPTIVE_KEY_GEOMETRY)) { add(Settings.PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH) + add(SettingsWithoutKey.ADAPTIVE_TYPING_STATS) + } add(Settings.PREF_SHORTCUT_ROWS) if (prefs.getBoolean(Settings.PREF_SHORTCUT_ROWS, Defaults.PREF_SHORTCUT_ROWS)) { add(Settings.PREF_SHORTCUT_TOP_ROW) @@ -190,6 +194,17 @@ fun createGestureTypingSettings(context: Context) = listOf( description = { it.toString() } ) { KeyboardSwitcher.getInstance().setThemeNeedsReload() } }, + Setting(context, SettingsWithoutKey.ADAPTIVE_TYPING_STATS, + R.string.adaptive_key_geometry_stats_title, R.string.adaptive_key_geometry_stats_summary) { + Preference( + name = it.title, + description = stringResource(R.string.adaptive_key_geometry_stats_summary), + onClick = { SettingsDestination.navigateTo(SettingsDestination.AdaptiveTypingStats) } + ) + }, + Setting(context, SettingsWithoutKey.ADAPTIVE_TYPING_STATS_CONTENT, R.string.adaptive_key_geometry_stats_title) { + AdaptiveTypingStatsContent() + }, Setting(context, Settings.PREF_SHORTCUT_ROWS, R.string.shortcut_rows, R.string.shortcut_rows_summary) { SwitchPreference(it, Defaults.PREF_SHORTCUT_ROWS) }, diff --git a/app/src/main/res/values/strings.xml b/app/src/main/res/values/strings.xml index b613493d6..26e09d80c 100644 --- a/app/src/main/res/values/strings.xml +++ b/app/src/main/res/values/strings.xml @@ -371,6 +371,11 @@ Adaptive key geometry (experimental) Learn where you actually tap each key and nudge gesture and tap recognition toward your style. Stored on-device only, no text is saved. Adaptive geometry strength + Learned typing model + See what the keyboard has learned about your typing + Per key: average offset from the key center (where you land), spread (how consistent you are), and number of samples. Higher samples and lower spread mean a stronger, more confident adjustment. Nothing here ever leaves your device. + Nothing learned yet. Turn on adaptive key geometry and type for a while, then come back. + Reset learned model Shortcut rows Enable temporary layout-defined shortcut rows opened by vertical swipes Top shortcut row swipe diff --git a/docs/ADAPTIVE_TYPING.md b/docs/ADAPTIVE_TYPING.md index 895d1930a..ef6024021 100644 --- a/docs/ADAPTIVE_TYPING.md +++ b/docs/ADAPTIVE_TYPING.md @@ -85,10 +85,15 @@ geometry). ### Layer B — learned per-user touch model (persisted, the "learning") -- **Record:** on a *confident* tap (committed and not immediately backspaced / - autocorrected away), compute `dx = touchX - keyCenterX`, `dy = touchY - keyCenterY` - and fold into that key's running mean/variance via an exponential moving average - (recent behavior weighted more; old data decays). +- **Record (taps):** on a letter tap, compute `dx = touchX - keyCenterX`, + `dy = touchY - keyCenterY` and fold into that key's running mean/variance via an + exponential moving average (recent behavior weighted more; old data decays). + Implemented in `PointerTracker.recordAdaptiveTouchSample`. +- **Record (gestures):** a swipe also teaches the model, but only via its clean + **endpoints**: finger-down ≈ the word's first letter, finger-up ≈ its last letter. + Interior keys are skipped (corner-cutting makes them unreliable) and only fresh single + strokes count (merged/extended trails have ambiguous ends). Implemented in + `InputLogic.maybeRecordGestureEndpoints`. - **Apply:** effective center = `center + mean_offset`; effective radius scales with consistency (tighter for keys you nail, more forgiving for scattered keys). @@ -182,13 +187,16 @@ stats page to keep it honest. Validate the gesture path explicitly (it's the pri ## Phased build order -1. **Foundation:** opt-in pref (5-file) + `leantype.db` table + DAO + `TouchModelManager` - (EMA update, capped effective-geometry API). Compiles; no behavior yet. -2. **Learning + gesture injection:** record confident taps; feed learned geometry into - `ProximityInfo` sweet spots (the gesture priority) + the capped `KeyDetector` - tie-break for taps. Testable on-device. -3. **Context prior (Layer A):** completion-derived next-char boost for taps. -4. **Stats page + reset UI + strength tuning.** +1. ✅ **Foundation:** opt-in pref (5-file) + `leantype.db` table + DAO + `TouchModelManager` + (EMA update, capped effective-geometry API). +2. ✅ **Learning + gesture injection:** record letter taps; feed learned geometry into + `ProximityInfo` sweet spots (gestures + tap-correction). The capped literal-tap + `KeyDetector` tie-break is deferred (marginal/risky — hitbox-gated). +3. ✅ **Gesture-endpoint learning:** swipes teach the model via their start/end keys. +4. ✅ **Stats / "learned typing model" page** (`AdaptiveTypingStatsScreen`) + reset. +5. ⬜ **Context prior (Layer A):** completion-derived next-char boost for taps. +6. ⬜ **Strength/cap tuning** + optional heatmap visualization + interior-key gesture + learning (needs corner-cutting handling or native alignment). ## Open questions (tracked) From 8eda439b92ae65ff4c4881767e9f8a4ddf7fd19a Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Sat, 6 Jun 2026 08:26:26 -0700 Subject: [PATCH 07/26] fix(two-thumb): clear stale merged-trail extend-base on delete + gesture lifecycle WordComposer.mExtendBatchInputBase (the multi-part merged-trail base) was only cleared by a normally-completing gesture, so an abnormal end (cancel / empty-top recognition) left it armed, and no deletion path cleared it. A later fresh swipe then merged with the ghost trail - most visibly at the start of a text box. Clear it at the same word-end sites where mLiveStroke is already dropped (handleBackspaceEvent, resetComposingState, commitChosenWord, desync) plus the two gesture-lifecycle origins (onStartBatchInput top, onCancelBatchInput). Does not touch the hot WordComposer.reset() path. Adds 8 tests: base cleared after each backspace mode (character/fragment/whole-word), the delete slider, fresh onStartBatchInput, and onCancelBatchInput; plus 2 guards pinning the (currently dead) static-seed interlock. Co-Authored-By: Claude Opus 4.8 --- .../keyboard/latin/inputlogic/InputLogic.java | 26 ++++ .../keyboard/latin/InputLogicTest.kt | 115 ++++++++++++++++++ dev-log.md | 89 ++++++++++++++ 3 files changed, 230 insertions(+) diff --git a/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java b/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java index 0faf3cb7e..69bd9f087 100644 --- a/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java +++ b/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java @@ -745,6 +745,15 @@ public void onStartBatchInput(final SettingsValues settingsValues, // Combining mode: snapshot before cancel; the gesture will re-arm the timer on completion. final boolean wasInCombiningMode = mInCombiningMode; cancelCombiningMode(); + // Two-thumb typing: defensively drop any stale merged-trail extend-base before this + // gesture's extend decision. The base is meant to live for exactly one gesture (set in + // the extend branch below, cleared in onUpdateTailBatchInputCompleted), but an abnormal + // prior gesture end — a cancel, or an empty-top recognition that early-returns before + // that clear — can leave it armed. A fresh gesture would then merge with the ghost + // trail; most visibly at the start of a text box, where no composing word exists to + // re-set it. Clearing here guarantees every gesture begins from a clean base; the + // extend branch re-arms it from the real composing word when appropriate. + mWordComposer.setExtendBatchInputBase(null); mConnection.beginBatchEdit(); // Two-thumb typing (#1.1 + combining-mode): two ways the gesture can EXTEND an existing // composing word instead of replacing it: @@ -895,6 +904,10 @@ public void onCancelBatchInput(final LatinIME.UIHandler handler) { // Drop any seed codepoint stashed by PointerTracker so the next gesture doesn't // strip its first letter against a stale seed. helium314.keyboard.keyboard.PointerTracker.consumeGestureSeedCodepoint(); + // Two-thumb typing: a cancelled gesture never reaches onUpdateTailBatchInputCompleted, + // which is the only routine site that clears the merged-trail extend-base. Drop it here + // so this cancelled gesture's trail can't leak into the next one. + mWordComposer.setExtendBatchInputBase(null); // Tap-promotion-extend was a per-gesture decision; clear on cancel so the NEXT // gesture re-evaluates against current timing. mGestureExtendsByTapPromotion = false; @@ -2360,6 +2373,12 @@ private void handleBackspaceEvent(final Event event, final InputTransaction inpu // stroke. The whole-word and batch delete branches below call mWordComposer.reset() // directly rather than resetComposingState(), so clear here to cover every path. mLiveStroke.reset(); + // Two-thumb typing: a backspace invalidates the merged-trail extend-base for the same + // reason it invalidates mLiveStroke above — re-doing the word must start from a clean + // stroke, not merge with the geometry we just partially deleted. This single clear + // covers all three backspace modes (character / fragment / whole-word), since it runs + // before any of their branches. + mWordComposer.setExtendBatchInputBase(null); // Typing-insight overlay: a backspace edits/clears the gesture word, so its trail is now // stale. Drop it so it doesn't linger. final MainKeyboardView backspaceKv = KeyboardSwitcher.getInstance().getMainKeyboardView(); @@ -3547,6 +3566,9 @@ private void resetComposingState(final boolean alsoResetLastComposedWord) { mCombiningWordHasGestureFragment = false; // Live-converge (#1.7): the word is gone, so its accumulated stroke is stale. mLiveStroke.reset(); + // Two-thumb typing: likewise drop the merged-trail extend-base. This path also covers + // the delete slider (finishInput -> resetComposingState) and cursor-move resets. + mWordComposer.setExtendBatchInputBase(null); // Combining mode is keyed on a composing word existing; if we're wiping it, the // pending timer would commit nothing useful, so cancel. cancelCombiningMode(); @@ -4217,6 +4239,8 @@ private void commitChosenWord(final SettingsValues settingsValues, final String mCombiningWordHasGestureFragment = false; // Live-converge (#1.7): word committed — its accumulated stroke no longer applies. mLiveStroke.reset(); + // Two-thumb typing: the merged-trail extend-base belongs to the just-committed word too. + mWordComposer.setExtendBatchInputBase(null); if (DebugFlags.DEBUG_ENABLED) { long runTimeMillis = System.currentTimeMillis() - startTimeMillis; Log.d(TAG, "commitChosenWord() : " + runTimeMillis + " ms to run " @@ -4379,6 +4403,8 @@ private void setComposingTextInternalWithBackgroundColor(final CharSequence newC // Live-converge (#1.7): composing was force-cancelled due to a desync — the stored // stroke no longer matches anything on screen, so drop it. mLiveStroke.reset(); + // Two-thumb typing: the merged-trail extend-base is equally stale after a desync. + mWordComposer.setExtendBatchInputBase(null); } } diff --git a/app/src/test/java/helium314/keyboard/latin/InputLogicTest.kt b/app/src/test/java/helium314/keyboard/latin/InputLogicTest.kt index 3ef7bb4fc..b73ab10fe 100644 --- a/app/src/test/java/helium314/keyboard/latin/InputLogicTest.kt +++ b/app/src/test/java/helium314/keyboard/latin/InputLogicTest.kt @@ -39,9 +39,12 @@ import org.robolectric.shadows.ShadowLog import java.util.* import kotlin.math.min import kotlin.streams.asSequence +import helium314.keyboard.latin.common.InputPointers import kotlin.test.BeforeTest import kotlin.test.Test import kotlin.test.assertEquals +import kotlin.test.assertFalse +import kotlin.test.assertTrue @RunWith(RobolectricTestRunner::class) @Config(shadows = [ @@ -607,6 +610,107 @@ class InputLogicTest { assertEquals("", composingText) } + // --- Two-thumb typing: the merged-trail extend-base (WordComposer.mExtendBatchInputBase) + // must never outlive the word it belonged to. Before the fix it was only cleared on a + // normally-completing gesture, so an abnormal end (cancel / empty-top recognition) left it + // armed, and NO deletion path cleared it. A later fresh swipe — most visibly at the start of + // a text box — then merged with that ghost trail. Each test below arms the base, performs one + // user action, and asserts the base is dropped. (We arm the base directly because the JVM + // harness has no native recognizer to produce a real merged trail.) Each fails pre-fix. --- + + @Test fun extendBaseClearedByCharacterBackspace() { + reset() + latinIME.prefs().edit { + putBoolean(Settings.PREF_GESTURE_MANUAL_SPACING, true) + // Character mode = neither fragment-pop nor whole-word delete. Both default ON + // (Defaults.kt), so they must be explicitly disabled to exercise the per-char path. + putBoolean(Settings.PREF_GESTURE_FRAGMENT_BACKSPACE, false) + putBoolean(Settings.PREF_COMBINING_BACKSPACE_DELETES_GESTURE_WORD, false) + } + chainInput("hello") + armExtendBase() + functionalKeyPress(KeyCode.DELETE) // character delete; word stays composing as "hell" + assertEquals("hell", composingText) + assertFalse(composer.isExtendBatchInputBaseSet) + } + + @Test fun extendBaseClearedByFragmentBackspace() { + reset() + latinIME.prefs().edit { + putBoolean(Settings.PREF_GESTURE_MANUAL_SPACING, true) + putBoolean(Settings.PREF_GESTURE_FRAGMENT_BACKSPACE, true) + putBoolean(Settings.PREF_COMBINING_BACKSPACE_DELETES_GESTURE_WORD, false) + } + chainInput("hello") // each tap records a fragment boundary under manual spacing + armExtendBase() + functionalKeyPress(KeyCode.DELETE) // fragment pop + assertFalse(composer.isExtendBatchInputBaseSet) + } + + @Test fun extendBaseClearedByWholeWordBackspace() { + reset() + latinIME.prefs().edit { + putBoolean(Settings.PREF_GESTURE_MANUAL_SPACING, true) + putBoolean(Settings.PREF_COMBINING_BACKSPACE_DELETES_GESTURE_WORD, true) + putBoolean(Settings.PREF_COMBINING_BACKSPACE_DELETES_COMPOSING_TEXT, true) + } + chainInput("hello") + armExtendBase() + functionalKeyPress(KeyCode.DELETE) // whole-word delete of the composing word + assertEquals("", composingText) + assertFalse(composer.isExtendBatchInputBaseSet) + } + + @Test fun extendBaseClearedByDeleteSlider() { + reset() + latinIME.prefs().edit { putBoolean(Settings.PREF_GESTURE_MANUAL_SPACING, true) } + chainInput("hello") + armExtendBase() + // The delete slider (swipe-from-backspace) routes through inputLogic.finishInput() in + // KeyboardActionListenerImpl.onMoveDeletePointer / onUpWithDeletePointerActive. + inputLogic.finishInput() + assertFalse(composer.isExtendBatchInputBaseSet) + } + + @Test fun extendBaseClearedByFreshGestureStart() { + reset() + latinIME.prefs().edit { putBoolean(Settings.PREF_GESTURE_MANUAL_SPACING, true) } + // No composing word: the next gesture is fresh and must not inherit a leaked base. + armExtendBase() + inputLogic.onStartBatchInput(settingsValues, KeyboardSwitcher.getInstance(), latinIME.mHandler) + handleMessages() + assertFalse(composer.isExtendBatchInputBaseSet) + } + + @Test fun extendBaseClearedByGestureCancel() { + reset() + latinIME.prefs().edit { putBoolean(Settings.PREF_GESTURE_MANUAL_SPACING, true) } + armExtendBase() + inputLogic.onCancelBatchInput(latinIME.mHandler) + handleMessages() + assertFalse(composer.isExtendBatchInputBaseSet) + } + + // Static-seed reachability guard. PointerTracker's tap-seed path (sLastLetterTap*) is gated + // on (!isMultipartComposeActive() && mCombiningGraceMs > 0). But grace > 0 forces multi-part + // composition active, so that conjunction is unsatisfiable and the seed is currently + // unreachable dead code. These pin the interlock: if a future settings refactor decouples + // them and re-arms the seed, it must first add the stale-static cleanup (the seed statics are + // process-global and never reset on delete / commit / field switch). + @Test fun graceImpliesMultipartComposeActive_keepsSeedPathDead() { + reset() + latinIME.prefs().edit { putInt(Settings.PREF_COMBINING_GRACE_MS, 1000) } + setText("") // force a settings reload + assertTrue(settingsValues.isMultipartComposeActive) + } + + @Test fun manualSpacingImpliesMultipartComposeActive() { + reset() + latinIME.prefs().edit { putBoolean(Settings.PREF_GESTURE_MANUAL_SPACING, true) } + setText("") + assertTrue(settingsValues.isMultipartComposeActive) + } + @Test fun forceAutoCapWorksWhenAutoCapIsOff() { reset() latinIME.prefs().edit { @@ -1309,6 +1413,17 @@ class InputLogicTest { checkConnectionConsistency() } + // Arm the merged-trail extend-base with a non-empty trail, simulating the state left by a + // prior gesture fragment. (A single empty InputPointers is treated as "clear", so two real + // points are required for isExtendBatchInputBaseSet to become true.) + private fun armExtendBase() { + val base = InputPointers(8) + base.addPointer(10, 20, 0, 0) + base.addPointer(30, 40, 0, 0) + composer.setExtendBatchInputBase(base) + assertTrue(composer.isExtendBatchInputBaseSet) + } + private fun checkConnectionConsistency() { // RichInputConnection only has composing text up to cursor, but InputConnection has full composing text val expectedConnectionComposingText = if (composingStart == -1 || composingEnd == -1) "" diff --git a/dev-log.md b/dev-log.md index 35da9bbfa..5a976bcda 100644 --- a/dev-log.md +++ b/dev-log.md @@ -825,3 +825,92 @@ The whole-word branch for a *composing* word used `mConnection.deleteTextBeforeC - Whole-word delete through a sentence deletes word-by-word with NO mashing/desync. - Re-doing a word after deletion recognizes cleanly (no ever-growing `th…`). - Swipe-on-backspace bulk delete leaves no stale stroke for the next word. + +--- + +## 2026-06-06 — Fix stale merged-trail extend-base leaking into the next gesture + +### Context +Bug report: gestures occasionally get "stored" after a deletion and bleed into the +next swipe — most visibly when swiping a fresh word at the start of a text box, +which produced a word fused with previously-deleted gesture geometry. Branch: +`fix/extend-base-leak-on-delete` off `main`. + +### Root cause +There are two per-word gesture-geometry stores. `mLiveStroke` (live-converge, in +`InputLogic`) was already cleared on every word-end path. But its WordComposer-side +counterpart — `mExtendBatchInputBase` (the multi-part merged-trail base consumed by +`WordComposer.setBatchInputPointers`) — was only ever cleared by a *normally +completing* gesture (`onUpdateTailBatchInputCompleted`, the lone routine clear, +which sits AFTER two empty-recognition early-returns). An abnormal gesture end left +the `mExtendBatchInputBaseSet` flag armed: +- `onCancelBatchInput` never cleared it, +- the empty-top-word early-returns in `onUpdateTailBatchInputCompleted` skip the clear, +- `WordComposer.reset()` does not touch it, so NO deletion path cleared it either. + +Once leaked, the flag survived every backspace / selection-delete / commit. A later +fresh gesture (no composing word ⇒ the `onStartBatchInput` extend branch never runs, +so it neither set nor cleared the base) hit the merge guard in +`setBatchInputPointers` and prepended the ghost trail. Start-of-text-box made it +maximally visible because there is no composing word there to legitimately re-set +the base. + +### Actions Taken +- `latin/inputlogic/InputLogic.java`: clear the merged-trail base + (`mWordComposer.setExtendBatchInputBase(null)`) at the same word-end sites where + `mLiveStroke` is already dropped, plus the two gesture-lifecycle origins: + - `onStartBatchInput` (top, before the extend decision — guarantees every gesture + starts from a clean base; the extend branch re-arms it from the real composing + word when appropriate), + - `onCancelBatchInput` (a cancelled gesture never reaches the routine clear), + - `handleBackspaceEvent` (next to the existing `mLiveStroke.reset()`, before any + mode branch — covers all three backspace modes in one place), + - `resetComposingState` (covers the delete slider's `finishInput` and cursor-move + resets), + - `commitChosenWord` and the composing-desync recovery path. + Deliberately did NOT add the clear to the hot `WordComposer.reset()` to avoid any + interaction with the mid-gesture merge timing; mirroring the reviewed `mLiveStroke` + sites is equivalent and lower-risk. +- `app/src/test/.../InputLogicTest.kt`: 6 base-clear regression tests (one per + backspace mode — character / fragment / whole-word — plus delete slider, fresh + `onStartBatchInput`, and `onCancelBatchInput`), each arming the base then asserting + it is dropped (each fails pre-fix). Added `armExtendBase()` helper and `InputPointers` + / `assertTrue` / `assertFalse` imports. +- Also added two reachability-guard tests pinning the (currently dead) PointerTracker + static-seed interlock: `grace > 0 ⇒ isMultipartComposeActive()` and the + manual-spacing equivalent. If a future settings refactor decouples them and re-arms + the seed, these fail and force adding the missing stale-static cleanup first. + +### Decisions Made +- Threading was verified safe: every `mExtendBatchInputBase` mutation runs on the UI + thread (the recognizer runs on the suggestions HandlerThread but only reads a + snapshot), so the new clears need no locking and can't race the merge. +- Scoped to dropping the base. The *related* "re-extend after a partial (character / + fragment) delete still merges the un-trimmed `mInputPointers`" issue is left for a + later ticket per the established "drop, don't trim" philosophy — the fix neither + causes nor worsens it. + +### Testing status +- Built and ran on this machine (newly installed JDK 21 + Android SDK platform-35 / + build-tools 35.0.0). `:app:testStandardDebugUnitTest` filtered to the 8 new tests + + `WordComposerTest`: **BUILD SUCCESSFUL, 11/11 passing.** Full-suite still carries the + 3 pre-existing known failures (`insertLetterIntoWordHangulFails`, + `revert autocorrect on delete`, `tapOnlyCombiningWordDoesNotShowAutospaceIndicator…`), + unrelated to this change. +- On-device validation still recommended: reproduce the start-of-text-box ghost-merge + (swipe a word, trigger an abnormal gesture end, delete, swipe a fresh word) and + confirm it no longer fuses; re-check all three backspace modes + the delete slider. + +### Manual Tests — Extend-base leak +| # | Steps | Expected Result | +|---|---|---| +| 1 | Two-thumb on. Swipe a word; cancel/abort a follow-up gesture; delete everything; swipe a fresh word at the start of the box. | Fresh word recognizes alone — no fusion with the deleted gesture. | +| 2 | Repeat #1 with backspace mode = Delete one character. | Same; no ghost merge. | +| 3 | Repeat #1 with backspace mode = Delete last fragment. | Same; no ghost merge. | +| 4 | Repeat #1 with backspace mode = Delete whole word. | Same; no ghost merge. | +| 5 | Repeat #1 using swipe-from-backspace (delete slider) to clear. | Same; no ghost merge. | + +### Open Questions / Next Steps +- Build the standard debug APK and install for on-device validation. +- Decide whether to also tackle the re-extend-after-partial-delete (stale + `mInputPointers`) follow-up. From 2acee3314e947873f79968b74608778990c74b2a Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Sat, 6 Jun 2026 23:46:51 -0700 Subject: [PATCH 08/26] feat(typing): mock-keyboard heatmap on the learned-typing-model stats page #1. The raw px-delta list was hard to interpret, so show the data spatially: - Store each key's size (KEY_WIDTH/KEY_HEIGHT) with its offset so the offset can be expressed as a fraction of the key (DB VERSION 3->4; additive ALTER on upgrade; copyFromDb reads by column name so older backups restore fine). - Stats page now renders a mock QWERTY where each key shows a dot at where you tend to land (offset as a fraction of the key) plus a faint spread ring; confident keys in the accent color, still-learning keys faded. The numeric list stays below for exact values. - Recorders (tap + gesture-endpoint) now pass the key's hitbox size. Build + extendBase/manager/settings tests green; :app:assembleStandardDebug builds. Co-Authored-By: Claude Opus 4.8 --- .../keyboard/keyboard/PointerTracker.java | 2 +- .../keyboard/latin/database/Database.kt | 34 +++++- .../keyboard/latin/database/TouchModelDao.kt | 28 +++-- .../keyboard/latin/inputlogic/InputLogic.java | 2 +- .../screens/AdaptiveTypingStatsScreen.kt | 110 +++++++++++++++--- app/src/main/res/values/strings.xml | 1 + 6 files changed, 144 insertions(+), 33 deletions(-) diff --git a/app/src/main/java/helium314/keyboard/keyboard/PointerTracker.java b/app/src/main/java/helium314/keyboard/keyboard/PointerTracker.java index 23943c6b9..dff85db69 100644 --- a/app/src/main/java/helium314/keyboard/keyboard/PointerTracker.java +++ b/app/src/main/java/helium314/keyboard/keyboard/PointerTracker.java @@ -1617,7 +1617,7 @@ private void recordAdaptiveTouchSample(final Key key, final int x, final int y) final float dy = y - hitBox.exactCenterY(); dao.record(key.getCode(), Integer.toString(mKeyboard.mId.mElementId), context.getResources().getConfiguration().orientation, dx, dy, - System.currentTimeMillis()); + hitBox.width(), hitBox.height(), System.currentTimeMillis()); } @Override diff --git a/app/src/main/java/helium314/keyboard/latin/database/Database.kt b/app/src/main/java/helium314/keyboard/latin/database/Database.kt index 15b1c8f5b..60059bfd3 100644 --- a/app/src/main/java/helium314/keyboard/latin/database/Database.kt +++ b/app/src/main/java/helium314/keyboard/latin/database/Database.kt @@ -18,13 +18,18 @@ class Database private constructor(context: Context, name: String = NAME) : SQLi db.execSQL("ALTER TABLE CLIPBOARD ADD COLUMN IMAGE_URI TEXT") } if (oldVersion < 3) { + // Created fresh with the current schema (already includes the key-size columns). db.execSQL(TouchModelDao.CREATE_TABLE) + } else if (oldVersion < 4) { + // Upgrading from the v3 table, which lacked the per-key size columns. + db.execSQL("ALTER TABLE ${TouchModelDao.TABLE} ADD COLUMN ${TouchModelDao.COLUMN_KEY_WIDTH} INTEGER NOT NULL DEFAULT 0") + db.execSQL("ALTER TABLE ${TouchModelDao.TABLE} ADD COLUMN ${TouchModelDao.COLUMN_KEY_HEIGHT} INTEGER NOT NULL DEFAULT 0") } } companion object { private val TAG = Database::class.java.simpleName - private const val VERSION = 3 + private const val VERSION = 4 const val NAME = "leantype.db" private var instance: Database? = null fun getInstance(context: Context): Database { @@ -67,11 +72,28 @@ class Database private constructor(context: Context, name: String = NAME) : SQLi ).use { it.moveToNext() } if (hasTouchModel) { touchDao.clear() - otherDb.readableDatabase.rawQuery(TouchModelDao.SELECT_ALL, null).use { - while (it.moveToNext()) { - touchDao.restore(TouchModelDao.Stat( - it.getInt(0), it.getString(1), it.getInt(2), it.getFloat(3), it.getFloat(4), - it.getFloat(5), it.getFloat(6), it.getInt(7), it.getLong(8))) + // Read by column name so an older backup that lacks the key-size columns + // restores fine (those default to 0). + otherDb.readableDatabase.rawQuery("SELECT * FROM ${TouchModelDao.TABLE}", null).use { c -> + val iCode = c.getColumnIndex("KEY_CODE") + val iLayout = c.getColumnIndex("LAYOUT") + val iOrient = c.getColumnIndex("ORIENTATION") + val iMdx = c.getColumnIndex("MEAN_DX") + val iMdy = c.getColumnIndex("MEAN_DY") + val iVdx = c.getColumnIndex("VAR_DX") + val iVdy = c.getColumnIndex("VAR_DY") + val iCount = c.getColumnIndex("COUNT") + val iUpd = c.getColumnIndex("UPDATED_AT") + val iW = c.getColumnIndex(TouchModelDao.COLUMN_KEY_WIDTH) + val iH = c.getColumnIndex(TouchModelDao.COLUMN_KEY_HEIGHT) + if (iCode >= 0 && iLayout >= 0 && iOrient >= 0) { + while (c.moveToNext()) { + touchDao.restore(TouchModelDao.Stat( + c.getInt(iCode), c.getString(iLayout), c.getInt(iOrient), + c.getFloat(iMdx), c.getFloat(iMdy), c.getFloat(iVdx), c.getFloat(iVdy), + c.getInt(iCount), c.getLong(iUpd), + if (iW >= 0) c.getInt(iW) else 0, if (iH >= 0) c.getInt(iH) else 0)) + } } } } diff --git a/app/src/main/java/helium314/keyboard/latin/database/TouchModelDao.kt b/app/src/main/java/helium314/keyboard/latin/database/TouchModelDao.kt index 951acf5fc..9717c67f5 100644 --- a/app/src/main/java/helium314/keyboard/latin/database/TouchModelDao.kt +++ b/app/src/main/java/helium314/keyboard/latin/database/TouchModelDao.kt @@ -20,7 +20,9 @@ import java.util.concurrent.Executors */ class TouchModelDao private constructor(private val db: Database) { - /** One key's learned stats. Offsets/variance are in pixels (relative to the key center). */ + /** One key's learned stats. Offsets/variance are in pixels (relative to the key center). + * keyWidth/keyHeight are the key's size in px at record time, so a viewer can express the + * offset as a fraction of the key (e.g. "18px = 20% toward the lower-left of E"). */ data class Stat( val keyCode: Int, val layout: String, @@ -31,6 +33,8 @@ class TouchModelDao private constructor(private val db: Database) { var varDy: Float, var count: Int, var updatedAt: Long, + var keyWidth: Int = 0, + var keyHeight: Int = 0, ) private val cache = HashMap() @@ -44,12 +48,12 @@ class TouchModelDao private constructor(private val db: Database) { db.readableDatabase.query( TABLE, arrayOf(COLUMN_KEY_CODE, COLUMN_LAYOUT, COLUMN_ORIENTATION, COLUMN_MEAN_DX, COLUMN_MEAN_DY, - COLUMN_VAR_DX, COLUMN_VAR_DY, COLUMN_COUNT, COLUMN_UPDATED_AT), + COLUMN_VAR_DX, COLUMN_VAR_DY, COLUMN_COUNT, COLUMN_UPDATED_AT, COLUMN_KEY_WIDTH, COLUMN_KEY_HEIGHT), null, null, null, null, null ).use { while (it.moveToNext()) { val s = Stat(it.getInt(0), it.getString(1), it.getInt(2), it.getFloat(3), it.getFloat(4), - it.getFloat(5), it.getFloat(6), it.getInt(7), it.getLong(8)) + it.getFloat(5), it.getFloat(6), it.getInt(7), it.getLong(8), it.getInt(9), it.getInt(10)) cache[key(s.keyCode, s.layout, s.orientation)] = s } } @@ -61,11 +65,12 @@ class TouchModelDao private constructor(private val db: Database) { * old data decays. Persists immediately. Callers must gate on incognito + the opt-in pref. */ @Synchronized - fun record(keyCode: Int, layout: String, orientation: Int, dx: Float, dy: Float, now: Long) { + fun record(keyCode: Int, layout: String, orientation: Int, dx: Float, dy: Float, + keyWidth: Int, keyHeight: Int, now: Long) { val k = key(keyCode, layout, orientation) val s = cache[k] if (s == null) { - val ns = Stat(keyCode, layout, orientation, dx, dy, 0f, 0f, 1, now) + val ns = Stat(keyCode, layout, orientation, dx, dy, 0f, 0f, 1, now, keyWidth, keyHeight) cache[k] = ns persistAsync(ns.copy()) return @@ -80,6 +85,8 @@ class TouchModelDao private constructor(private val db: Database) { s.varDy = (1 - a) * (s.varDy + a * (dy - oldMeanDy) * (dy - oldMeanDy)) if (s.count < Int.MAX_VALUE) s.count++ s.updatedAt = now + if (keyWidth > 0) s.keyWidth = keyWidth + if (keyHeight > 0) s.keyHeight = keyHeight persistAsync(s.copy()) } @@ -116,7 +123,7 @@ class TouchModelDao private constructor(private val db: Database) { } private fun write(s: Stat) { - val cv = ContentValues(9) + val cv = ContentValues(11) cv.put(COLUMN_KEY_CODE, s.keyCode) cv.put(COLUMN_LAYOUT, s.layout) cv.put(COLUMN_ORIENTATION, s.orientation) @@ -126,6 +133,8 @@ class TouchModelDao private constructor(private val db: Database) { cv.put(COLUMN_VAR_DY, s.varDy) cv.put(COLUMN_COUNT, s.count) cv.put(COLUMN_UPDATED_AT, s.updatedAt) + cv.put(COLUMN_KEY_WIDTH, s.keyWidth) + cv.put(COLUMN_KEY_HEIGHT, s.keyHeight) db.writableDatabase.insertWithOnConflict(TABLE, null, cv, SQLiteDatabase.CONFLICT_REPLACE) } @@ -146,6 +155,8 @@ class TouchModelDao private constructor(private val db: Database) { private const val COLUMN_VAR_DY = "VAR_DY" private const val COLUMN_COUNT = "COUNT" private const val COLUMN_UPDATED_AT = "UPDATED_AT" + const val COLUMN_KEY_WIDTH = "KEY_WIDTH" + const val COLUMN_KEY_HEIGHT = "KEY_HEIGHT" const val CREATE_TABLE = """ CREATE TABLE $TABLE ( $COLUMN_KEY_CODE INTEGER NOT NULL, @@ -157,12 +168,11 @@ class TouchModelDao private constructor(private val db: Database) { $COLUMN_VAR_DY REAL NOT NULL, $COLUMN_COUNT INTEGER NOT NULL, $COLUMN_UPDATED_AT INTEGER NOT NULL, + $COLUMN_KEY_WIDTH INTEGER NOT NULL DEFAULT 0, + $COLUMN_KEY_HEIGHT INTEGER NOT NULL DEFAULT 0, PRIMARY KEY ($COLUMN_KEY_CODE, $COLUMN_LAYOUT, $COLUMN_ORIENTATION) ) """ - const val SELECT_ALL = - "SELECT $COLUMN_KEY_CODE, $COLUMN_LAYOUT, $COLUMN_ORIENTATION, $COLUMN_MEAN_DX, $COLUMN_MEAN_DY, " + - "$COLUMN_VAR_DX, $COLUMN_VAR_DY, $COLUMN_COUNT, $COLUMN_UPDATED_AT FROM $TABLE" private fun key(keyCode: Int, layout: String, orientation: Int) = "$keyCode|$layout|$orientation" diff --git a/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java b/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java index 69bd9f087..4413319df 100644 --- a/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java +++ b/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java @@ -4032,7 +4032,7 @@ private void recordGestureEndpoint(final Keyboard keyboard, final int codePoint, dao.record(codePoint, Integer.toString(keyboard.mId.mElementId), mLatinIME.getResources().getConfiguration().orientation, x - hitBox.exactCenterX(), y - hitBox.exactCenterY(), - System.currentTimeMillis()); + hitBox.width(), hitBox.height(), System.currentTimeMillis()); } /** diff --git a/app/src/main/java/helium314/keyboard/settings/screens/AdaptiveTypingStatsScreen.kt b/app/src/main/java/helium314/keyboard/settings/screens/AdaptiveTypingStatsScreen.kt index 7aa78442b..e59120b34 100644 --- a/app/src/main/java/helium314/keyboard/settings/screens/AdaptiveTypingStatsScreen.kt +++ b/app/src/main/java/helium314/keyboard/settings/screens/AdaptiveTypingStatsScreen.kt @@ -2,9 +2,11 @@ package helium314.keyboard.settings.screens import android.content.Context +import androidx.compose.foundation.Canvas import androidx.compose.foundation.layout.Arrangement import androidx.compose.foundation.layout.Column import androidx.compose.foundation.layout.Row +import androidx.compose.foundation.layout.aspectRatio import androidx.compose.foundation.layout.fillMaxWidth import androidx.compose.foundation.layout.padding import androidx.compose.material3.Button @@ -16,6 +18,12 @@ import androidx.compose.runtime.mutableStateOf import androidx.compose.runtime.remember import androidx.compose.runtime.setValue import androidx.compose.ui.Modifier +import androidx.compose.ui.geometry.CornerRadius +import androidx.compose.ui.geometry.Offset +import androidx.compose.ui.geometry.Size +import androidx.compose.ui.graphics.nativeCanvas +import androidx.compose.ui.graphics.toArgb +import androidx.compose.ui.platform.LocalConfiguration import androidx.compose.ui.platform.LocalContext import androidx.compose.ui.res.stringResource import androidx.compose.ui.text.font.FontFamily @@ -40,25 +48,85 @@ fun AdaptiveTypingStatsScreen(onClickBack: () -> Unit) { ) } -private data class StatRow(val label: String, val dx: Int, val dy: Int, val spread: Int, val count: Int) - -private fun loadStatRows(context: Context): List { +private fun loadStats(context: Context): List { val dao = TouchModelDao.getInstance(context) ?: return emptyList() - return dao.all() - .sortedByDescending { it.count } - .map { s -> - val letter = try { String(Character.toChars(s.keyCode)) } catch (e: Throwable) { "?" } - val orient = if (s.orientation == 2) "L" else "P" // 2 == landscape - val spread = sqrt(((s.varDx + s.varDy) / 2f).coerceAtLeast(0f)).toInt() - StatRow("$letter ($orient)", s.meanDx.toInt(), s.meanDy.toInt(), spread, s.count) + return dao.all().sortedByDescending { it.count } +} + +/** A mock QWERTY keyboard with, on each key, a dot showing where the user tends to land + * (offset from the key center, as a fraction of the key) and a faint ring for the spread. + * Confident keys (enough samples) are drawn in the accent color, still-learning keys faded. */ +@Composable +private fun MockKeyboardHeatmap(stats: List) { + val orientation = LocalConfiguration.current.orientation + val pref = stats.filter { it.orientation == orientation } + val byCode = (if (pref.isNotEmpty()) pref else stats).associateBy { it.keyCode } + + val keyBg = MaterialTheme.colorScheme.surfaceVariant + val labelColor = MaterialTheme.colorScheme.onSurfaceVariant + val accent = MaterialTheme.colorScheme.primary + val faded = MaterialTheme.colorScheme.outline + val rows = listOf("qwertyuiop", "asdfghjkl", "zxcvbnm") + val labelPaint = remember { + android.graphics.Paint().apply { + isAntiAlias = true + textAlign = android.graphics.Paint.Align.CENTER + } + } + + Canvas( + modifier = Modifier + .fillMaxWidth() + .aspectRatio(10f / (3f * 1.35f)) + .padding(vertical = 8.dp) + ) { + val cellW = size.width / 10f + val cellH = cellW * 1.35f + labelPaint.color = labelColor.toArgb() + labelPaint.textSize = cellW * 0.42f + rows.forEachIndexed { r, row -> + val startX = (size.width - row.length * cellW) / 2f + val top = r * cellH + row.forEachIndexed { i, ch -> + val left = startX + i * cellW + val cx = left + cellW / 2f + val cy = top + cellH / 2f + drawRoundRect( + color = keyBg, + topLeft = Offset(left + 3f, top + 3f), + size = Size(cellW - 6f, cellH - 6f), + cornerRadius = CornerRadius(8f, 8f) + ) + drawContext.canvas.nativeCanvas.drawText( + ch.uppercase(), cx, + cy - (labelPaint.ascent() + labelPaint.descent()) / 2f, labelPaint + ) + val s = byCode[ch.code] + if (s != null && s.keyWidth > 0 && s.keyHeight > 0) { + val fx = (s.meanDx / s.keyWidth).coerceIn(-0.5f, 0.5f) + val fy = (s.meanDy / s.keyHeight).coerceIn(-0.5f, 0.5f) + val dotX = cx + fx * cellW + val dotY = cy + fy * cellH + val col = if (s.count >= TouchModelDao.MIN_CONFIDENT_SAMPLES) accent else faded + val spreadFrac = sqrt( + (((s.varDx / (s.keyWidth.toFloat() * s.keyWidth)) + + (s.varDy / (s.keyHeight.toFloat() * s.keyHeight))) / 2f).coerceAtLeast(0f) + ).coerceIn(0f, 0.5f) + if (spreadFrac > 0f) + drawCircle(col.copy(alpha = 0.15f), spreadFrac * cellW, Offset(dotX, dotY)) + drawLine(col, Offset(cx, cy), Offset(dotX, dotY), strokeWidth = 2.5f) + drawCircle(col, cellW * 0.07f, Offset(dotX, dotY)) + } + } } + } } /** The dynamic stats body. Rendered as the content of a single registered Setting. */ @Composable fun AdaptiveTypingStatsContent() { val context = LocalContext.current - var rows by remember { mutableStateOf(loadStatRows(context)) } + var stats by remember { mutableStateOf(loadStats(context)) } Column(Modifier.fillMaxWidth().padding(horizontal = 12.dp, vertical = 8.dp)) { Text( stringResource(R.string.adaptive_key_geometry_stats_explanation), @@ -66,20 +134,30 @@ fun AdaptiveTypingStatsContent() { color = MaterialTheme.colorScheme.onSurfaceVariant, modifier = Modifier.padding(bottom = 12.dp) ) - if (rows.isEmpty()) { + if (stats.isEmpty()) { Text( stringResource(R.string.adaptive_key_geometry_stats_empty), modifier = Modifier.padding(vertical = 8.dp) ) } else { - rows.forEach { r -> + MockKeyboardHeatmap(stats) + Text( + stringResource(R.string.adaptive_key_geometry_stats_legend), + style = MaterialTheme.typography.bodySmall, + color = MaterialTheme.colorScheme.onSurfaceVariant, + modifier = Modifier.padding(vertical = 8.dp) + ) + stats.forEach { s -> + val letter = try { String(Character.toChars(s.keyCode)) } catch (e: Throwable) { "?" } + val orient = if (s.orientation == 2) "L" else "P" // 2 == landscape + val spread = sqrt(((s.varDx + s.varDy) / 2f).coerceAtLeast(0f)).toInt() Row( modifier = Modifier.fillMaxWidth().padding(vertical = 3.dp), horizontalArrangement = Arrangement.SpaceBetween ) { - Text(r.label, fontFamily = FontFamily.Monospace, style = MaterialTheme.typography.bodyLarge) + Text("$letter ($orient)", fontFamily = FontFamily.Monospace, style = MaterialTheme.typography.bodyLarge) Text( - "Δ(${r.dx}, ${r.dy})px ±${r.spread}px n=${r.count}", + "Δ(${s.meanDx.toInt()}, ${s.meanDy.toInt()})px ±${spread}px n=${s.count}", fontFamily = FontFamily.Monospace, style = MaterialTheme.typography.bodyMedium, color = MaterialTheme.colorScheme.onSurfaceVariant @@ -90,7 +168,7 @@ fun AdaptiveTypingStatsContent() { Button( onClick = { TouchModelDao.getInstance(context)?.clear() - rows = emptyList() + stats = emptyList() }, modifier = Modifier.padding(top = 16.dp) ) { diff --git a/app/src/main/res/values/strings.xml b/app/src/main/res/values/strings.xml index 26e09d80c..0de94eba0 100644 --- a/app/src/main/res/values/strings.xml +++ b/app/src/main/res/values/strings.xml @@ -375,6 +375,7 @@ See what the keyboard has learned about your typing Per key: average offset from the key center (where you land), spread (how consistent you are), and number of samples. Higher samples and lower spread mean a stronger, more confident adjustment. Nothing here ever leaves your device. Nothing learned yet. Turn on adaptive key geometry and type for a while, then come back. + On each key, the dot shows where you tend to land relative to the center, and the ring shows your spread. Solid dots are confident; faded dots are still learning. Reset learned model Shortcut rows Enable temporary layout-defined shortcut rows opened by vertical swipes From 320cbe17e99e20610d80bda23569ae259d08e36c Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Sun, 7 Jun 2026 00:04:17 -0700 Subject: [PATCH 09/26] chore(db): collapse experimental touch-model migration into a single recreate-on-upgrade step --- .../helium314/keyboard/latin/database/Database.kt | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/app/src/main/java/helium314/keyboard/latin/database/Database.kt b/app/src/main/java/helium314/keyboard/latin/database/Database.kt index 60059bfd3..f1ec827af 100644 --- a/app/src/main/java/helium314/keyboard/latin/database/Database.kt +++ b/app/src/main/java/helium314/keyboard/latin/database/Database.kt @@ -17,13 +17,13 @@ class Database private constructor(context: Context, name: String = NAME) : SQLi if (oldVersion < 2) { db.execSQL("ALTER TABLE CLIPBOARD ADD COLUMN IMAGE_URI TEXT") } - if (oldVersion < 3) { - // Created fresh with the current schema (already includes the key-size columns). + if (oldVersion < 4) { + // The learned touch model is experimental and disposable, so on any upgrade from + // before it stabilized we just recreate it with the current schema instead of + // carrying per-version column migrations. No real release shipped the table, so + // this loses nothing in practice. + db.execSQL("DROP TABLE IF EXISTS ${TouchModelDao.TABLE}") db.execSQL(TouchModelDao.CREATE_TABLE) - } else if (oldVersion < 4) { - // Upgrading from the v3 table, which lacked the per-key size columns. - db.execSQL("ALTER TABLE ${TouchModelDao.TABLE} ADD COLUMN ${TouchModelDao.COLUMN_KEY_WIDTH} INTEGER NOT NULL DEFAULT 0") - db.execSQL("ALTER TABLE ${TouchModelDao.TABLE} ADD COLUMN ${TouchModelDao.COLUMN_KEY_HEIGHT} INTEGER NOT NULL DEFAULT 0") } } From d43311043ebca0f2a40a666e43dcbc42340b9dd0 Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Sun, 7 Jun 2026 06:51:17 -0700 Subject: [PATCH 10/26] feat(typing): next-key context prior + adaptive tap biasing (#1) Completes adaptive typing on the tap side: - AdaptiveKeyContext builds a "likely next key" prior from the top-5 suggestions, weighted equally (averaged, not score-skewed): the next char of the in-progress word's completions, or the first char of next-word predictions for a fresh word. Rebuilt between keystrokes in InputLogic.setSuggestedWords (off the tap hot path), read lock-free per tap. - KeyDetector.detectHitKey now biases the tapped key by the learned per-key landing offset (shifts the effective center) AND the context prior (enlarges likely keys). Both capped; the prior cap (~18% of key) is deliberately below the learned cap (~25%) so it nudges rather than dominates. Only near-boundary taps can flip; clear presses are untouched. - Suppressed during gestures/swipes (PointerTracker.isInGestureOrKeySwipe); gestures are not context-biased since suggestions don't change mid-swipe. Applies to both current words being built and the first key of a new word (predictions). Build + extendBase/manager/settings tests green; :app:assembleStandardDebug builds. Co-Authored-By: Claude Opus 4.8 --- .../keyboard/keyboard/AdaptiveKeyContext.java | 94 +++++++++++++++++++ .../keyboard/keyboard/KeyDetector.java | 81 ++++++++++++++++ .../keyboard/keyboard/PointerTracker.java | 6 ++ .../keyboard/latin/inputlogic/InputLogic.java | 10 ++ docs/ADAPTIVE_TYPING.md | 14 ++- 5 files changed, 202 insertions(+), 3 deletions(-) create mode 100644 app/src/main/java/helium314/keyboard/keyboard/AdaptiveKeyContext.java diff --git a/app/src/main/java/helium314/keyboard/keyboard/AdaptiveKeyContext.java b/app/src/main/java/helium314/keyboard/keyboard/AdaptiveKeyContext.java new file mode 100644 index 000000000..e61d14076 --- /dev/null +++ b/app/src/main/java/helium314/keyboard/keyboard/AdaptiveKeyContext.java @@ -0,0 +1,94 @@ +/* + * SPDX-License-Identifier: GPL-3.0-only + */ +package helium314.keyboard.keyboard; + +import helium314.keyboard.latin.SuggestedWords; + +import java.util.HashMap; +import java.util.Map; + +/** + * Holds a transient "likely next key" prior derived from the current suggestion strip, used to + * gently enlarge the touch target of likely next keys (adaptive typing, see + * docs/ADAPTIVE_TYPING.md). + * + *

It is rebuilt between keystrokes — whenever the suggestions change — on the UI thread, and + * read per tap in {@link KeyDetector}. Reads are lock-free via {@code volatile} parallel arrays, + * which are tiny (at most a handful of distinct next-characters), so the tap hot path stays fast + * even for very fast typists. Because suggestions are computed asynchronously, the prior may lag + * the latest keystroke by one tap under very fast typing; that is harmless, as the prior is only + * a soft, capped bias. + * + *

Suggestions are weighted EQUALLY (averaged) rather than by score, so the result is not skewed + * toward the single top suggestion. + */ +public final class AdaptiveKeyContext { + /** How many suggestions to average over. */ + private static final int TOP_N = 5; + + private static volatile int[] sCodes; + private static volatile float[] sWeights; + + private AdaptiveKeyContext() {} + + /** + * Rebuild the prior from the top suggestions. + * + * @param words the current suggestion strip contents. + * @param position index of the NEXT character within each suggestion — the current + * composing-word length while a word is being built, or 0 for a new word + * (using next-word predictions, whose first letter is the likely next key). + */ + public static void update(final SuggestedWords words, final int position) { + if (words == null || words.isEmpty() || position < 0) { + clear(); + return; + } + final int n = Math.min(TOP_N, words.size()); + final HashMap tally = new HashMap<>(); + int considered = 0; + for (int i = 0; i < n; i++) { + final String w = words.getWord(i); + if (w == null || position >= w.length()) continue; // typed word itself / too short + final int cp = Character.toLowerCase(w.charAt(position)); + if (!Character.isLetter(cp)) continue; + tally.merge(cp, 1, Integer::sum); + considered++; + } + if (considered == 0) { + clear(); + return; + } + final int[] codes = new int[tally.size()]; + final float[] weights = new float[tally.size()]; + int j = 0; + for (final Map.Entry e : tally.entrySet()) { + codes[j] = e.getKey(); + weights[j] = (float) e.getValue() / considered; // 0..1, equal-weight average + j++; + } + sCodes = codes; + sWeights = weights; + } + + public static void clear() { + sCodes = null; + sWeights = null; + } + + /** Prior weight in [0, 1] for the given key code (0 if none / no prior). */ + public static float weight(final int code) { + final int[] c = sCodes; + final float[] w = sWeights; + if (c == null) return 0f; + for (int i = 0; i < c.length; i++) { + if (c[i] == code) return w[i]; + } + return 0f; + } + + public static boolean hasPrior() { + return sCodes != null; + } +} diff --git a/app/src/main/java/helium314/keyboard/keyboard/KeyDetector.java b/app/src/main/java/helium314/keyboard/keyboard/KeyDetector.java index 400511cda..caf093e25 100644 --- a/app/src/main/java/helium314/keyboard/keyboard/KeyDetector.java +++ b/app/src/main/java/helium314/keyboard/keyboard/KeyDetector.java @@ -6,10 +6,25 @@ package helium314.keyboard.keyboard; +import android.content.Context; +import android.graphics.Rect; + +import helium314.keyboard.latin.database.TouchModelDao; +import helium314.keyboard.latin.database.TouchModelManager; +import helium314.keyboard.latin.settings.Settings; +import helium314.keyboard.latin.settings.SettingsValues; + /** * This class handles key detection. */ public class KeyDetector { + // Adaptive typing: the context prior may enlarge a likely key's effective target by at most + // this fraction of the key (deliberately a bit less than the learned-geometry cap, so the + // prior nudges rather than dominates). See docs/ADAPTIVE_TYPING.md. + private static final float PRIOR_MAX_FRACTION = 0.18f; + // A neighbor is only allowed to win a tap if the touch is within this fraction of its hitbox. + private static final float CONSIDER_MARGIN_FRACTION = 0.40f; + private final int mKeyHysteresisDistanceSquared; private final int mKeyHysteresisDistanceForSlidingModifierSquared; @@ -101,6 +116,72 @@ public Key detectHitKey(final int x, final int y) { primaryKey = key; } } + // Adaptive typing (opt-in): for a plain tap (not a gesture/swipe), let the learned + // per-key landing offset and the current next-key prior gently bias which key wins — + // bounded so only genuinely ambiguous, near-boundary taps can flip. + if (primaryKey != null && !PointerTracker.isInGestureOrKeySwipe()) { + final Key biased = applyAdaptiveBias(touchX, touchY, primaryKey); + if (biased != null) return biased; + } return primaryKey; } + + /** Returns a key that should win this tap instead of {@code geo} due to learned/prior bias, + * or {@code null} to keep the plain geometric result. */ + private Key applyAdaptiveBias(final int touchX, final int touchY, final Key geo) { + final SettingsValues sv = Settings.getValues(); + if (sv == null || !sv.mAdaptiveKeyGeometry || sv.mAdaptiveKeyGeometryStrength <= 0) return null; + final Context ctx = Settings.getCurrentContext(); + if (ctx == null || mKeyboard == null) return null; + final TouchModelDao dao = TouchModelDao.getInstance(ctx); + final boolean hasPrior = AdaptiveKeyContext.hasPrior(); + if (dao == null && !hasPrior) return null; // nothing to bias with + final String layout = Integer.toString(mKeyboard.mId.mElementId); + final int orientation = ctx.getResources().getConfiguration().orientation; + final int strength = sv.mAdaptiveKeyGeometryStrength; + + Key best = geo; + float bestScore = adjustedDistance(geo, touchX, touchY, dao, layout, orientation, strength); + for (final Key k : mKeyboard.getNearestKeys(touchX, touchY)) { + if (k == geo) continue; + final int code = k.getCode(); + if (code <= 0 || !Character.isLetter(code)) continue; + // Only a genuinely-favored neighbor (a confident learned offset and/or a next-key + // prior) may steal a near-boundary tap; otherwise leave the geometric result alone. + final float prior = AdaptiveKeyContext.weight(code); + final TouchModelDao.Stat st = (dao == null) ? null : dao.get(code, layout, orientation); + final boolean hasLearned = st != null && st.getCount() >= TouchModelDao.MIN_CONFIDENT_SAMPLES; + if (prior <= 0f && !hasLearned) continue; + // Bound: the touch must be within a margin of the neighbor's hitbox. + final float margin = CONSIDER_MARGIN_FRACTION * k.getWidth(); + if (k.squaredDistanceToEdge(touchX, touchY) > margin * margin) continue; + final float s = adjustedDistance(k, touchX, touchY, dao, layout, orientation, strength); + if (s < bestScore) { + bestScore = s; + best = k; + } + } + return best == geo ? null : best; + } + + /** Distance from the touch to the key's effective center (center shifted by the learned + * landing offset) minus the capped next-key prior boost. Smaller wins. */ + private float adjustedDistance(final Key k, final int touchX, final int touchY, + final TouchModelDao dao, final String layout, final int orientation, final int strength) { + final Rect hb = k.getHitBox(); + float cx = hb.exactCenterX(); + float cy = hb.exactCenterY(); + if (dao != null) { + final TouchModelDao.Stat st = dao.get(k.getCode(), layout, orientation); + final float[] off = TouchModelManager.adjustedOffset(st, k.getWidth(), k.getHeight(), strength); + cx += off[0]; + cy += off[1]; + } + final float dx = touchX - cx; + final float dy = touchY - cy; + final float dist = (float) Math.sqrt(dx * dx + dy * dy); + final float boost = AdaptiveKeyContext.weight(k.getCode()) + * PRIOR_MAX_FRACTION * k.getWidth() * (strength / 100f); + return dist - boost; + } } diff --git a/app/src/main/java/helium314/keyboard/keyboard/PointerTracker.java b/app/src/main/java/helium314/keyboard/keyboard/PointerTracker.java index dff85db69..cb81b315a 100644 --- a/app/src/main/java/helium314/keyboard/keyboard/PointerTracker.java +++ b/app/src/main/java/helium314/keyboard/keyboard/PointerTracker.java @@ -162,6 +162,12 @@ public static int consumeGestureSeedCodepoint() { return seed; } + /** True while a gesture, key-swipe (space/delete), or shortcut-row swipe is in progress. + * Adaptive tap biasing must be suppressed in these states (it only applies to plain taps). */ + public static boolean isInGestureOrKeySwipe() { + return sInGesture || sInKeySwipe || sInShortcutRowSwipe; + } + private static TypingTimeRecorder sTypingTimeRecorder; // The position and time at which first down event occurred. diff --git a/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java b/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java index 4413319df..449a0a12b 100644 --- a/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java +++ b/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java @@ -1376,6 +1376,16 @@ public void setSuggestedWords(final SuggestedWords suggestedWords) { mWordComposer.setAutoCorrection(suggestedWordInfo); } mSuggestedWords = suggestedWords; + // Adaptive typing: refresh the "likely next key" prior from the strip so the next tap can + // bias toward likely keys. The next-character index is the current composing-word length + // (or 0 for a new word, where the predictions' first letters are the likely next keys). + // Cheap, and runs on suggestion updates rather than the tap hot path. + final SettingsValues svAdaptive = Settings.getValues(); + if (svAdaptive != null && svAdaptive.mAdaptiveKeyGeometry) { + final int nextCharIndex = mWordComposer.isComposingWord() + ? mWordComposer.getTypedWord().length() : 0; + helium314.keyboard.keyboard.AdaptiveKeyContext.update(suggestedWords, nextCharIndex); + } final boolean newAutoCorrectionIndicator = suggestedWords.mWillAutoCorrect; // Put a blue underline to a word in TextView which will be auto-corrected. diff --git a/docs/ADAPTIVE_TYPING.md b/docs/ADAPTIVE_TYPING.md index ef6024021..aa84f11d9 100644 --- a/docs/ADAPTIVE_TYPING.md +++ b/docs/ADAPTIVE_TYPING.md @@ -190,11 +190,19 @@ stats page to keep it honest. Validate the gesture path explicitly (it's the pri 1. ✅ **Foundation:** opt-in pref (5-file) + `leantype.db` table + DAO + `TouchModelManager` (EMA update, capped effective-geometry API). 2. ✅ **Learning + gesture injection:** record letter taps; feed learned geometry into - `ProximityInfo` sweet spots (gestures + tap-correction). The capped literal-tap - `KeyDetector` tie-break is deferred (marginal/risky — hitbox-gated). + `ProximityInfo` sweet spots (gestures + tap-correction). 3. ✅ **Gesture-endpoint learning:** swipes teach the model via their start/end keys. 4. ✅ **Stats / "learned typing model" page** (`AdaptiveTypingStatsScreen`) + reset. -5. ⬜ **Context prior (Layer A):** completion-derived next-char boost for taps. +5. ✅ **Tap biasing + context prior (Layer A):** `KeyDetector` now biases the tapped key by + the learned per-key offset AND a next-key prior. The prior is built in + `AdaptiveKeyContext` from the top-5 suggestions, weighted **equally** (averaged, not + score-skewed): the next char of the in-progress word's completions, or the first char of + the next-word predictions for a fresh word. It is rebuilt between keystrokes (in + `InputLogic.setSuggestedWords`, off the tap path) and read lock-free per tap. The prior's + cap (`PRIOR_MAX_FRACTION` ≈ 18% of key) is deliberately a bit **below** the learned cap + (≈ 25%), so it nudges rather than dominates. Bias is suppressed during gestures/swipes + (`PointerTracker.isInGestureOrKeySwipe`) and only flips near-boundary taps. Gestures are + intentionally NOT context-biased (suggestions don't change mid-swipe). 6. ⬜ **Strength/cap tuning** + optional heatmap visualization + interior-key gesture learning (needs corner-cutting handling or native alignment). From d2a5235dc59b66712780e6cf26cb80fc49dee1f9 Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Sun, 7 Jun 2026 07:02:52 -0700 Subject: [PATCH 11/26] docs(adaptive): correct rationale for tap-only context prior; clarify swipe-start behavior --- docs/ADAPTIVE_TYPING.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/docs/ADAPTIVE_TYPING.md b/docs/ADAPTIVE_TYPING.md index aa84f11d9..9aeee09a4 100644 --- a/docs/ADAPTIVE_TYPING.md +++ b/docs/ADAPTIVE_TYPING.md @@ -201,8 +201,20 @@ stats page to keep it honest. Validate the gesture path explicitly (it's the pri `InputLogic.setSuggestedWords`, off the tap path) and read lock-free per tap. The prior's cap (`PRIOR_MAX_FRACTION` ≈ 18% of key) is deliberately a bit **below** the learned cap (≈ 25%), so it nudges rather than dominates. Bias is suppressed during gestures/swipes - (`PointerTracker.isInGestureOrKeySwipe`) and only flips near-boundary taps. Gestures are - intentionally NOT context-biased (suggestions don't change mid-swipe). + (`PointerTracker.isInGestureOrKeySwipe`) and only flips near-boundary taps. + + The context prior is **tap-only by design** — not because swipe suggestions are + unavailable (they do update live during a swipe), but because the prior is a *per-key* + mechanism (it enlarges the next key's tap target), and a swipe resolves a *whole word* + holistically via the recognizer, which already incorporates previous-word context through + its language model. So there is no single "next key" to enlarge mid-stroke. Making swipe + *word selection* more context-aware is the separate word-level re-ranking lever in + `SUGGESTION_RANKING.md`, not this prior. Note the resulting asymmetry: for a fresh word the + first *tap* is context-biased, but the first point of a *swipe* is not. + + The **learned geometry** (Layer B), by contrast, applies to the entire swipe including its + start — every key the stroke passes is matched against its learned-shifted sweet spot — and + a swipe's endpoints also feed the model (gesture-endpoint learning). 6. ⬜ **Strength/cap tuning** + optional heatmap visualization + interior-key gesture learning (needs corner-cutting handling or native alignment). From e8fdf9aee652af8c34e4badf4e78c464a613d4f5 Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Sun, 7 Jun 2026 07:26:32 -0700 Subject: [PATCH 12/26] Add independent context-prior toggle and group adaptive settings Split the next-key context prior into its own opt-in setting (adaptive_context_prior, default off) alongside the existing learned key-geometry toggle, and group both under a single "Adaptive typing" section in the Gesture typing settings. - Settings/Defaults/SettingsValues: new mAdaptiveContextPrior flag. - KeyDetector: independently gate learned-offset bias vs prior boost; either alone can bias a near-boundary tap, both share the strength slider. adjustedDistance now takes usePrior and gates the boost. - InputLogic.setSuggestedWords: build the prior only when the prior toggle is on; clear AdaptiveKeyContext otherwise. - GestureTypingScreen: new "Adaptive typing" category holding both toggles; strength slider shown if either is on; stats shown if learning is on. Co-Authored-By: Claude Opus 4.8 --- .../keyboard/keyboard/KeyDetector.java | 28 ++++++++++++------- .../keyboard/latin/inputlogic/InputLogic.java | 4 ++- .../keyboard/latin/settings/Defaults.kt | 1 + .../keyboard/latin/settings/Settings.java | 3 ++ .../latin/settings/SettingsValues.java | 4 +++ .../settings/screens/GestureTypingScreen.kt | 18 ++++++++---- app/src/main/res/values/strings.xml | 3 ++ 7 files changed, 45 insertions(+), 16 deletions(-) diff --git a/app/src/main/java/helium314/keyboard/keyboard/KeyDetector.java b/app/src/main/java/helium314/keyboard/keyboard/KeyDetector.java index caf093e25..b7aa54c3d 100644 --- a/app/src/main/java/helium314/keyboard/keyboard/KeyDetector.java +++ b/app/src/main/java/helium314/keyboard/keyboard/KeyDetector.java @@ -130,32 +130,37 @@ public Key detectHitKey(final int x, final int y) { * or {@code null} to keep the plain geometric result. */ private Key applyAdaptiveBias(final int touchX, final int touchY, final Key geo) { final SettingsValues sv = Settings.getValues(); - if (sv == null || !sv.mAdaptiveKeyGeometry || sv.mAdaptiveKeyGeometryStrength <= 0) return null; + if (sv == null || sv.mAdaptiveKeyGeometryStrength <= 0) return null; + // The two halves are independently toggleable: learned per-key offset, and the + // context prior. Either alone is enough to bias a tap; both share the strength slider. + final boolean learn = sv.mAdaptiveKeyGeometry; + final boolean usePrior = sv.mAdaptiveContextPrior; + if (!learn && !usePrior) return null; final Context ctx = Settings.getCurrentContext(); if (ctx == null || mKeyboard == null) return null; - final TouchModelDao dao = TouchModelDao.getInstance(ctx); - final boolean hasPrior = AdaptiveKeyContext.hasPrior(); + final TouchModelDao dao = learn ? TouchModelDao.getInstance(ctx) : null; + final boolean hasPrior = usePrior && AdaptiveKeyContext.hasPrior(); if (dao == null && !hasPrior) return null; // nothing to bias with final String layout = Integer.toString(mKeyboard.mId.mElementId); final int orientation = ctx.getResources().getConfiguration().orientation; final int strength = sv.mAdaptiveKeyGeometryStrength; Key best = geo; - float bestScore = adjustedDistance(geo, touchX, touchY, dao, layout, orientation, strength); + float bestScore = adjustedDistance(geo, touchX, touchY, dao, layout, orientation, strength, usePrior); for (final Key k : mKeyboard.getNearestKeys(touchX, touchY)) { if (k == geo) continue; final int code = k.getCode(); if (code <= 0 || !Character.isLetter(code)) continue; // Only a genuinely-favored neighbor (a confident learned offset and/or a next-key // prior) may steal a near-boundary tap; otherwise leave the geometric result alone. - final float prior = AdaptiveKeyContext.weight(code); + final float prior = usePrior ? AdaptiveKeyContext.weight(code) : 0f; final TouchModelDao.Stat st = (dao == null) ? null : dao.get(code, layout, orientation); final boolean hasLearned = st != null && st.getCount() >= TouchModelDao.MIN_CONFIDENT_SAMPLES; if (prior <= 0f && !hasLearned) continue; // Bound: the touch must be within a margin of the neighbor's hitbox. final float margin = CONSIDER_MARGIN_FRACTION * k.getWidth(); if (k.squaredDistanceToEdge(touchX, touchY) > margin * margin) continue; - final float s = adjustedDistance(k, touchX, touchY, dao, layout, orientation, strength); + final float s = adjustedDistance(k, touchX, touchY, dao, layout, orientation, strength, usePrior); if (s < bestScore) { bestScore = s; best = k; @@ -165,9 +170,11 @@ private Key applyAdaptiveBias(final int touchX, final int touchY, final Key geo) } /** Distance from the touch to the key's effective center (center shifted by the learned - * landing offset) minus the capped next-key prior boost. Smaller wins. */ + * landing offset, when enabled) minus the capped next-key prior boost (when enabled). + * Smaller wins. */ private float adjustedDistance(final Key k, final int touchX, final int touchY, - final TouchModelDao dao, final String layout, final int orientation, final int strength) { + final TouchModelDao dao, final String layout, final int orientation, final int strength, + final boolean usePrior) { final Rect hb = k.getHitBox(); float cx = hb.exactCenterX(); float cy = hb.exactCenterY(); @@ -180,8 +187,9 @@ private float adjustedDistance(final Key k, final int touchX, final int touchY, final float dx = touchX - cx; final float dy = touchY - cy; final float dist = (float) Math.sqrt(dx * dx + dy * dy); - final float boost = AdaptiveKeyContext.weight(k.getCode()) - * PRIOR_MAX_FRACTION * k.getWidth() * (strength / 100f); + final float boost = usePrior + ? AdaptiveKeyContext.weight(k.getCode()) * PRIOR_MAX_FRACTION * k.getWidth() * (strength / 100f) + : 0f; return dist - boost; } } diff --git a/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java b/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java index 449a0a12b..5536ba519 100644 --- a/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java +++ b/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java @@ -1381,10 +1381,12 @@ public void setSuggestedWords(final SuggestedWords suggestedWords) { // (or 0 for a new word, where the predictions' first letters are the likely next keys). // Cheap, and runs on suggestion updates rather than the tap hot path. final SettingsValues svAdaptive = Settings.getValues(); - if (svAdaptive != null && svAdaptive.mAdaptiveKeyGeometry) { + if (svAdaptive != null && svAdaptive.mAdaptiveContextPrior) { final int nextCharIndex = mWordComposer.isComposingWord() ? mWordComposer.getTypedWord().length() : 0; helium314.keyboard.keyboard.AdaptiveKeyContext.update(suggestedWords, nextCharIndex); + } else { + helium314.keyboard.keyboard.AdaptiveKeyContext.clear(); } final boolean newAutoCorrectionIndicator = suggestedWords.mWillAutoCorrect; diff --git a/app/src/main/java/helium314/keyboard/latin/settings/Defaults.kt b/app/src/main/java/helium314/keyboard/latin/settings/Defaults.kt index 1d76bd78a..d899ec83a 100644 --- a/app/src/main/java/helium314/keyboard/latin/settings/Defaults.kt +++ b/app/src/main/java/helium314/keyboard/latin/settings/Defaults.kt @@ -160,6 +160,7 @@ object Defaults { // Adaptive typing (opt-in). Off by default; strength is a 0..100 percentage of the cap. const val PREF_ADAPTIVE_KEY_GEOMETRY = false const val PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH = 50 + const val PREF_ADAPTIVE_CONTEXT_PRIOR = false const val PREF_SHOW_SETUP_WIZARD_ICON = true const val PREF_USE_CONTACTS = false const val PREF_USE_APPS = false diff --git a/app/src/main/java/helium314/keyboard/latin/settings/Settings.java b/app/src/main/java/helium314/keyboard/latin/settings/Settings.java index abff08a16..93f68771a 100644 --- a/app/src/main/java/helium314/keyboard/latin/settings/Settings.java +++ b/app/src/main/java/helium314/keyboard/latin/settings/Settings.java @@ -188,6 +188,9 @@ public final class Settings implements SharedPreferences.OnSharedPreferenceChang // incognito-gated. Strength scales the cap (0 = off). See docs/ADAPTIVE_TYPING.md. public static final String PREF_ADAPTIVE_KEY_GEOMETRY = "adaptive_key_geometry"; public static final String PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH = "adaptive_key_geometry_strength"; + // Context prior: enlarge the touch target of the next key the suggestions predict. Independent + // of the learned-geometry toggle above; both share the strength slider. See ADAPTIVE_TYPING.md. + public static final String PREF_ADAPTIVE_CONTEXT_PRIOR = "adaptive_context_prior"; public static final String PREF_SHOW_SETUP_WIZARD_ICON = "show_setup_wizard_icon"; public static final String PREF_USE_CONTACTS = "use_contacts"; public static final String PREF_USE_APPS = "use_apps"; diff --git a/app/src/main/java/helium314/keyboard/latin/settings/SettingsValues.java b/app/src/main/java/helium314/keyboard/latin/settings/SettingsValues.java index e1add5869..02d3a3962 100644 --- a/app/src/main/java/helium314/keyboard/latin/settings/SettingsValues.java +++ b/app/src/main/java/helium314/keyboard/latin/settings/SettingsValues.java @@ -148,6 +148,7 @@ public class SettingsValues { // Adaptive typing (opt-in): learn where the user lands and bias tap/gesture key geometry. public final boolean mAdaptiveKeyGeometry; public final int mAdaptiveKeyGeometryStrength; + public final boolean mAdaptiveContextPrior; public final boolean mSlidingKeyInputPreviewEnabled; public final int mKeyLongpressTimeout; public final boolean mEnableEmojiAltPhysicalKey; @@ -405,6 +406,9 @@ public SettingsValues(final Context context, final SharedPreferences prefs, fina mAdaptiveKeyGeometryStrength = prefs.getInt( Settings.PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH, Defaults.PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH); + mAdaptiveContextPrior = prefs.getBoolean( + Settings.PREF_ADAPTIVE_CONTEXT_PRIOR, + Defaults.PREF_ADAPTIVE_CONTEXT_PRIOR); mSuggestionStripHiddenPerUserSettings = mToolbarMode == ToolbarMode.HIDDEN || mToolbarMode == ToolbarMode.TOOLBAR_KEYS; mOverrideShowingSuggestions = mInputAttributes.mMayOverrideShowingSuggestions diff --git a/app/src/main/java/helium314/keyboard/settings/screens/GestureTypingScreen.kt b/app/src/main/java/helium314/keyboard/settings/screens/GestureTypingScreen.kt index ab7e3912a..505ac04c4 100644 --- a/app/src/main/java/helium314/keyboard/settings/screens/GestureTypingScreen.kt +++ b/app/src/main/java/helium314/keyboard/settings/screens/GestureTypingScreen.kt @@ -75,16 +75,20 @@ fun GestureTypingScreen( add(Settings.PREF_SPACE_VERTICAL_SWIPE) add(Settings.PREF_TOUCHPAD_SENSITIVITY) add(Settings.PREF_DELETE_SWIPE) - add(Settings.PREF_ADAPTIVE_KEY_GEOMETRY) - if (prefs.getBoolean(Settings.PREF_ADAPTIVE_KEY_GEOMETRY, Defaults.PREF_ADAPTIVE_KEY_GEOMETRY)) { - add(Settings.PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH) - add(SettingsWithoutKey.ADAPTIVE_TYPING_STATS) - } add(Settings.PREF_SHORTCUT_ROWS) if (prefs.getBoolean(Settings.PREF_SHORTCUT_ROWS, Defaults.PREF_SHORTCUT_ROWS)) { add(Settings.PREF_SHORTCUT_TOP_ROW) add(Settings.PREF_SHORTCUT_BOTTOM_ROW) } + + // Adaptive typing — both sub-features grouped in one section. + add(R.string.adaptive_typing_category) + add(Settings.PREF_ADAPTIVE_KEY_GEOMETRY) + add(Settings.PREF_ADAPTIVE_CONTEXT_PRIOR) + val learnOn = prefs.getBoolean(Settings.PREF_ADAPTIVE_KEY_GEOMETRY, Defaults.PREF_ADAPTIVE_KEY_GEOMETRY) + val priorOn = prefs.getBoolean(Settings.PREF_ADAPTIVE_CONTEXT_PRIOR, Defaults.PREF_ADAPTIVE_CONTEXT_PRIOR) + if (learnOn || priorOn) add(Settings.PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH) + if (learnOn) add(SettingsWithoutKey.ADAPTIVE_TYPING_STATS) // the learned model only fills with the geometry half } SearchSettingsScreen( onClickBack = onClickBack, @@ -185,6 +189,10 @@ fun createGestureTypingSettings(context: Context) = listOf( KeyboardSwitcher.getInstance().setThemeNeedsReload() // rebuild keyboard so sweet spots refresh } }, + Setting(context, Settings.PREF_ADAPTIVE_CONTEXT_PRIOR, + R.string.adaptive_context_prior, R.string.adaptive_context_prior_summary) { + SwitchPreference(it, Defaults.PREF_ADAPTIVE_CONTEXT_PRIOR) + }, Setting(context, Settings.PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH, R.string.adaptive_key_geometry_strength) { def -> SliderPreference( name = def.title, diff --git a/app/src/main/res/values/strings.xml b/app/src/main/res/values/strings.xml index 08110c704..2009204a1 100644 --- a/app/src/main/res/values/strings.xml +++ b/app/src/main/res/values/strings.xml @@ -387,6 +387,9 @@ Adaptive key geometry (experimental) Learn where you actually tap each key and nudge gesture and tap recognition toward your style. Stored on-device only, no text is saved. Adaptive geometry strength + Adaptive typing + Anticipate likely keys (experimental) + Make the next key the suggestions predict slightly easier to press. Averages the top suggestions and is capped so it never overrides a clear press. Applies to taps, not swipes. Learned typing model See what the keyboard has learned about your typing Per key: average offset from the key center (where you land), spread (how consistent you are), and number of samples. Higher samples and lower spread mean a stronger, more confident adjustment. Nothing here ever leaves your device. From eb0b118040f5dc3835f6fd04f00b3538e0884242 Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Sun, 7 Jun 2026 07:37:05 -0700 Subject: [PATCH 13/26] Add live adaptive-typing debug overlay A debug toggle ("Show adaptive targets on keyboard") that visualizes the adaptive model directly on the live keyboard, so the feature is visible as you type: - Learned geometry: each letter key shows a faint geometric-center ring, an arrow to its learned landing target, and a dot at that target. - Context prior: keys the suggestions predict get a translucent halo whose radius grows with the prior weight; halos morph between keystrokes. Implemented as AdaptiveTargetsDrawingPreview (an AbstractDrawingPreview, same mechanism as the gesture-debug overlay), drawn on the DrawingPreviewPlacerView above the keys. It is purely visual, reads the same live model / prior / settings the engine uses, and is gated on its pref each frame (zero cost when off). The halo radius is exaggerated vs the engine's sub-key boost for legibility; the visible keys are deliberately not reflowed. - New pref PREF_ADAPTIVE_DEBUG_OVERLAY (5-file), shown under "Adaptive typing" when either adaptive toggle is on. - AdaptiveKeyContext gains a change listener fired on update()/clear(); the overlay repaints on it. MainKeyboardView registers the listener and feeds the overlay the keyboard + padding so markers align with rendered keys. - docs/ADAPTIVE_TYPING.md: document the overlay, the independent context-prior toggle, the grouped settings, the built heatmap, and fix the DB version. Co-Authored-By: Claude Opus 4.8 --- .../keyboard/keyboard/AdaptiveKeyContext.java | 20 ++ .../keyboard/keyboard/MainKeyboardView.java | 14 ++ .../AdaptiveTargetsDrawingPreview.java | 194 ++++++++++++++++++ .../keyboard/latin/settings/Defaults.kt | 1 + .../keyboard/latin/settings/Settings.java | 3 + .../latin/settings/SettingsValues.java | 4 + .../settings/screens/GestureTypingScreen.kt | 7 + app/src/main/res/values/strings.xml | 2 + docs/ADAPTIVE_TYPING.md | 60 +++++- 9 files changed, 297 insertions(+), 8 deletions(-) create mode 100644 app/src/main/java/helium314/keyboard/keyboard/internal/AdaptiveTargetsDrawingPreview.java diff --git a/app/src/main/java/helium314/keyboard/keyboard/AdaptiveKeyContext.java b/app/src/main/java/helium314/keyboard/keyboard/AdaptiveKeyContext.java index e61d14076..26871f2f9 100644 --- a/app/src/main/java/helium314/keyboard/keyboard/AdaptiveKeyContext.java +++ b/app/src/main/java/helium314/keyboard/keyboard/AdaptiveKeyContext.java @@ -30,8 +30,26 @@ public final class AdaptiveKeyContext { private static volatile int[] sCodes; private static volatile float[] sWeights; + /** + * Optional observer notified whenever the prior changes, on the same (UI) thread that mutates + * it. Used only by the debug overlay (see AdaptiveTargetsDrawingPreview) to repaint the live + * keyboard as the prior shifts between keystrokes; null in normal operation. Volatile so the + * keyboard view can register/clear it from its own lifecycle without extra locking. + */ + private static volatile Runnable sChangeListener; + private AdaptiveKeyContext() {} + /** Register (or clear, with {@code null}) the debug repaint observer. */ + public static void setChangeListener(final Runnable listener) { + sChangeListener = listener; + } + + private static void fireChanged() { + final Runnable l = sChangeListener; + if (l != null) l.run(); + } + /** * Rebuild the prior from the top suggestions. * @@ -70,11 +88,13 @@ public static void update(final SuggestedWords words, final int position) { } sCodes = codes; sWeights = weights; + fireChanged(); } public static void clear() { sCodes = null; sWeights = null; + fireChanged(); } /** Prior weight in [0, 1] for the given key code (0 if none / no prior). */ diff --git a/app/src/main/java/helium314/keyboard/keyboard/MainKeyboardView.java b/app/src/main/java/helium314/keyboard/keyboard/MainKeyboardView.java index c642688db..c10eda072 100644 --- a/app/src/main/java/helium314/keyboard/keyboard/MainKeyboardView.java +++ b/app/src/main/java/helium314/keyboard/keyboard/MainKeyboardView.java @@ -37,6 +37,7 @@ import helium314.keyboard.compat.ConfigurationCompatKt; import helium314.keyboard.keyboard.internal.DrawingPreviewPlacerView; import helium314.keyboard.keyboard.internal.DrawingProxy; +import helium314.keyboard.keyboard.internal.AdaptiveTargetsDrawingPreview; import helium314.keyboard.keyboard.internal.GestureDebugPointsDrawingPreview; import helium314.keyboard.keyboard.internal.GestureFloatingTextDrawingPreview; import helium314.keyboard.keyboard.internal.GestureTrailsDrawingPreview; @@ -126,6 +127,9 @@ public final class MainKeyboardView extends KeyboardView implements DrawingProxy private final SlidingKeyInputDrawingPreview mSlidingKeyInputDrawingPreview; // Debug overlay for two-thumb point hinting (#2.1), toggled by PREF_GESTURE_DEBUG_DRAW_POINTS. private final GestureDebugPointsDrawingPreview mGestureDebugPointsDrawingPreview; + // Debug overlay visualizing adaptive typing (learned offsets + next-key prior), toggled by + // PREF_ADAPTIVE_DEBUG_OVERLAY. See docs/ADAPTIVE_TYPING.md. + private final AdaptiveTargetsDrawingPreview mAdaptiveTargetsDrawingPreview; // Key preview private final KeyPreviewDrawParams mKeyPreviewDrawParams; @@ -233,6 +237,13 @@ public MainKeyboardView(final Context context, final AttributeSet attrs, final i // Debug overlay last so it draws ON TOP of the gesture trail / floating preview. mGestureDebugPointsDrawingPreview = new GestureDebugPointsDrawingPreview(); mGestureDebugPointsDrawingPreview.setDrawingView(drawingPreviewPlacerView); + // Adaptive-typing visualization overlay. Always "enabled" at the preview layer; the actual + // drawing is gated on its live pref each frame, so it costs nothing when toggled off. It + // repaints between keystrokes via the AdaptiveKeyContext change listener registered below. + mAdaptiveTargetsDrawingPreview = new AdaptiveTargetsDrawingPreview(); + mAdaptiveTargetsDrawingPreview.setDrawingView(drawingPreviewPlacerView); + mAdaptiveTargetsDrawingPreview.setPreviewEnabled(true); + AdaptiveKeyContext.setChangeListener(mAdaptiveTargetsDrawingPreview::onAdaptiveContextChanged); mainKeyboardViewAttr.recycle(); mDrawingPreviewPlacerView = drawingPreviewPlacerView; @@ -394,6 +405,9 @@ public void setKeyboard(@NonNull final Keyboard keyboard) { keyboard, -getPaddingLeft(), -getPaddingTop() + getVerticalCorrection()); PointerTracker.setKeyDetector(mKeyDetector); mPopupKeysKeyboardCache.clear(); + // Keys render at getX()+paddingLeft / getY()+paddingTop; hand those to the adaptive overlay + // so its markers line up with the drawn keys. + mAdaptiveTargetsDrawingPreview.setKeyboard(keyboard, getPaddingLeft(), getPaddingTop()); mSpaceKey = keyboard.getKey(Constants.CODE_SPACE); final int keyHeight = keyboard.mMostCommonKeyHeight - keyboard.mVerticalGap; diff --git a/app/src/main/java/helium314/keyboard/keyboard/internal/AdaptiveTargetsDrawingPreview.java b/app/src/main/java/helium314/keyboard/keyboard/internal/AdaptiveTargetsDrawingPreview.java new file mode 100644 index 000000000..0f4d5fe05 --- /dev/null +++ b/app/src/main/java/helium314/keyboard/keyboard/internal/AdaptiveTargetsDrawingPreview.java @@ -0,0 +1,194 @@ +/* + * SPDX-License-Identifier: GPL-3.0-only + */ + +package helium314.keyboard.keyboard.internal; + +import android.content.Context; +import android.graphics.Canvas; +import android.graphics.Color; +import android.graphics.Paint; + +import androidx.annotation.NonNull; + +import helium314.keyboard.keyboard.AdaptiveKeyContext; +import helium314.keyboard.keyboard.Key; +import helium314.keyboard.keyboard.Keyboard; +import helium314.keyboard.keyboard.PointerTracker; +import helium314.keyboard.latin.database.TouchModelDao; +import helium314.keyboard.latin.database.TouchModelManager; +import helium314.keyboard.latin.settings.Settings; +import helium314.keyboard.latin.settings.SettingsValues; + +/** + * Debug visualization for adaptive typing (opt-in via {@code PREF_ADAPTIVE_DEBUG_OVERLAY}). Drawn + * on top of the live keyboard, it makes the two halves of the feature visible as you type: + * + *

    + *
  • Learned key geometry — for each letter key with a confident learned landing offset, + * a faint ring marks the key's geometric centre, an arrow points to where the user's taps + * actually land on average, and a filled dot marks that learned target. This is the same + * offset {@link helium314.keyboard.keyboard.KeyDetector} biases taps toward.
    • + *
  • Next-key context prior — keys the current suggestions predict get a translucent + * halo whose radius grows with the prior weight, centred on the (possibly shifted) target. + * The halo appears/grows/shrinks between keystrokes, so the keyboard visibly "leans" toward + * the likely next key.
    • + *
+ * + *

The overlay is purely visual — it never changes detection. It reads the same live model + * ({@link TouchModelDao}), prior ({@link AdaptiveKeyContext}) and {@link SettingsValues} the engine + * uses, so what you see is what the engine does (the halo radius is exaggerated relative to the + * engine's sub-key boost so the effect is legible). Drawing is gated on the live pref each frame, + * so it costs nothing when the toggle is off. + * + *

Threading mirrors the other previews: {@link #setKeyboard} runs on the keyboard-view layout + * path and {@link #drawPreview} on {@code DrawingPreviewPlacerView}'s {@code onDraw}, both on the + * main thread. Repaints between keystrokes are driven by {@link AdaptiveKeyContext}'s change + * listener (also fired on the main thread), wired up by {@code MainKeyboardView}. + */ +public final class AdaptiveTargetsDrawingPreview extends AbstractDrawingPreview { + // Halo radius for a fully-agreed prior (weight 1.0), as a fraction of key width. Deliberately + // larger than KeyDetector's PRIOR_MAX_FRACTION (0.18) — this is a visualization, so the bulge + // is exaggerated to read clearly; smaller weights scale down linearly. + private static final float HALO_MAX_FRACTION = 0.55f; + private static final float GEOM_DOT_RADIUS_PX = 3f; + private static final float EFF_DOT_RADIUS_PX = 6f; + private static final float MIN_ARROW_LENGTH_PX = 1.5f; + + /** Current keyboard, set on the layout path; iterated for keys at draw time. */ + private Keyboard mKeyboard; + /** Keyboard-view padding: keys render at {@code getX()+paddingLeft, getY()+paddingTop}, and the + * placer canvas is already translated to the keyboard-view origin, so we add padding here. */ + private int mPaddingLeft; + private int mPaddingTop; + + private final Paint mGeomPaint = new Paint(); // reference: geometric key centre + private final Paint mArrowPaint = new Paint(); // shift from centre to learned target + private final Paint mEffPaint = new Paint(); // learned landing target + private final Paint mHaloFill = new Paint(); // prior bulge (fill) + private final Paint mHaloStroke = new Paint(); // prior bulge (outline) + + public AdaptiveTargetsDrawingPreview() { + mGeomPaint.setAntiAlias(true); + mGeomPaint.setColor(Color.WHITE); + mGeomPaint.setAlpha(0x66); + mGeomPaint.setStyle(Paint.Style.STROKE); + mGeomPaint.setStrokeWidth(2f); + + mArrowPaint.setAntiAlias(true); + mArrowPaint.setColor(Color.rgb(255, 171, 0)); // amber + mArrowPaint.setAlpha(0xCC); + mArrowPaint.setStyle(Paint.Style.STROKE); + mArrowPaint.setStrokeWidth(3f); + mArrowPaint.setStrokeCap(Paint.Cap.ROUND); + + mEffPaint.setAntiAlias(true); + mEffPaint.setColor(Color.rgb(255, 171, 0)); // amber + mEffPaint.setAlpha(0xEE); + mEffPaint.setStyle(Paint.Style.FILL); + + mHaloFill.setAntiAlias(true); + mHaloFill.setColor(Color.rgb(0, 200, 83)); // green = "predicted, easier to hit" + mHaloFill.setStyle(Paint.Style.FILL); + + mHaloStroke.setAntiAlias(true); + mHaloStroke.setColor(Color.rgb(0, 200, 83)); + mHaloStroke.setStyle(Paint.Style.STROKE); + mHaloStroke.setStrokeWidth(2.5f); + } + + /** Hand the overlay the current keyboard and the view padding at which keys are rendered. */ + public void setKeyboard(final Keyboard keyboard, final int paddingLeft, final int paddingTop) { + mKeyboard = keyboard; + mPaddingLeft = paddingLeft; + mPaddingTop = paddingTop; + invalidateDrawingView(); + } + + /** Repaint hook for {@link AdaptiveKeyContext}'s change listener (fires on each keystroke). */ + public void onAdaptiveContextChanged() { + invalidateDrawingView(); + } + + @Override + public void onDeallocateMemory() { + mKeyboard = null; + } + + @Override + public void setPreviewPosition(@NonNull final PointerTracker tracker) { + // No-op: the overlay is derived from the keyboard + model, not a single pointer. + } + + @Override + public void drawPreview(@NonNull final Canvas canvas) { + if (!isPreviewEnabled()) return; // geometry valid + final Keyboard keyboard = mKeyboard; + if (keyboard == null) return; + final SettingsValues sv = Settings.getValues(); + if (sv == null || !sv.mAdaptiveDebugOverlay) return; + final boolean learn = sv.mAdaptiveKeyGeometry; + final boolean prior = sv.mAdaptiveContextPrior; + if (!learn && !prior) return; + final int strength = sv.mAdaptiveKeyGeometryStrength; + + // Learned-offset lookup is keyed by layout + orientation, like KeyDetector. + TouchModelDao dao = null; + String layout = null; + int orientation = 0; + if (learn) { + final Context ctx = Settings.getCurrentContext(); + if (ctx != null) { + dao = TouchModelDao.getInstance(ctx); + layout = Integer.toString(keyboard.mId.mElementId); + orientation = ctx.getResources().getConfiguration().orientation; + } + } + + for (final Key key : keyboard.getSortedKeys()) { + if (key.isSpacer()) continue; + final int code = key.getCode(); + if (code <= 0 || !Character.isLetter(code)) continue; + final int w = key.getWidth(); + final int h = key.getHeight(); + if (w <= 0 || h <= 0) continue; + final float cx = mPaddingLeft + key.getX() + w / 2f; + final float cy = mPaddingTop + key.getY() + h / 2f; + + float effX = cx; + float effY = cy; + boolean haveLearned = false; + if (dao != null) { + final TouchModelDao.Stat st = dao.get(code, layout, orientation); + if (st != null && st.getCount() >= TouchModelDao.MIN_CONFIDENT_SAMPLES) { + final float[] off = TouchModelManager.adjustedOffset(st, w, h, strength); + effX = cx + off[0]; + effY = cy + off[1]; + haveLearned = true; + } + } + + // Prior halo first, so the learned dot/arrow sit on top of it. + if (prior) { + final float weight = AdaptiveKeyContext.weight(code); + if (weight > 0f) { + final float r = weight * HALO_MAX_FRACTION * w; + mHaloFill.setAlpha(0x33); + canvas.drawCircle(effX, effY, r, mHaloFill); + mHaloStroke.setAlpha(0x99); + canvas.drawCircle(effX, effY, r, mHaloStroke); + } + } + + if (haveLearned) { + canvas.drawCircle(cx, cy, GEOM_DOT_RADIUS_PX, mGeomPaint); + final float dx = effX - cx; + final float dy = effY - cy; + if (dx * dx + dy * dy > MIN_ARROW_LENGTH_PX * MIN_ARROW_LENGTH_PX) { + canvas.drawLine(cx, cy, effX, effY, mArrowPaint); + } + canvas.drawCircle(effX, effY, EFF_DOT_RADIUS_PX, mEffPaint); + } + } + } +} diff --git a/app/src/main/java/helium314/keyboard/latin/settings/Defaults.kt b/app/src/main/java/helium314/keyboard/latin/settings/Defaults.kt index d899ec83a..c02b2079c 100644 --- a/app/src/main/java/helium314/keyboard/latin/settings/Defaults.kt +++ b/app/src/main/java/helium314/keyboard/latin/settings/Defaults.kt @@ -161,6 +161,7 @@ object Defaults { const val PREF_ADAPTIVE_KEY_GEOMETRY = false const val PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH = 50 const val PREF_ADAPTIVE_CONTEXT_PRIOR = false + const val PREF_ADAPTIVE_DEBUG_OVERLAY = false const val PREF_SHOW_SETUP_WIZARD_ICON = true const val PREF_USE_CONTACTS = false const val PREF_USE_APPS = false diff --git a/app/src/main/java/helium314/keyboard/latin/settings/Settings.java b/app/src/main/java/helium314/keyboard/latin/settings/Settings.java index 93f68771a..a0e3c05fd 100644 --- a/app/src/main/java/helium314/keyboard/latin/settings/Settings.java +++ b/app/src/main/java/helium314/keyboard/latin/settings/Settings.java @@ -191,6 +191,9 @@ public final class Settings implements SharedPreferences.OnSharedPreferenceChang // Context prior: enlarge the touch target of the next key the suggestions predict. Independent // of the learned-geometry toggle above; both share the strength slider. See ADAPTIVE_TYPING.md. public static final String PREF_ADAPTIVE_CONTEXT_PRIOR = "adaptive_context_prior"; + // Debug visualization: draw the effective per-key targets (learned-offset shift + prior bulge) + // on the live keyboard so the adaptive behavior is visible as you type. See ADAPTIVE_TYPING.md. + public static final String PREF_ADAPTIVE_DEBUG_OVERLAY = "adaptive_debug_overlay"; public static final String PREF_SHOW_SETUP_WIZARD_ICON = "show_setup_wizard_icon"; public static final String PREF_USE_CONTACTS = "use_contacts"; public static final String PREF_USE_APPS = "use_apps"; diff --git a/app/src/main/java/helium314/keyboard/latin/settings/SettingsValues.java b/app/src/main/java/helium314/keyboard/latin/settings/SettingsValues.java index 02d3a3962..4b86f4618 100644 --- a/app/src/main/java/helium314/keyboard/latin/settings/SettingsValues.java +++ b/app/src/main/java/helium314/keyboard/latin/settings/SettingsValues.java @@ -149,6 +149,7 @@ public class SettingsValues { public final boolean mAdaptiveKeyGeometry; public final int mAdaptiveKeyGeometryStrength; public final boolean mAdaptiveContextPrior; + public final boolean mAdaptiveDebugOverlay; public final boolean mSlidingKeyInputPreviewEnabled; public final int mKeyLongpressTimeout; public final boolean mEnableEmojiAltPhysicalKey; @@ -409,6 +410,9 @@ public SettingsValues(final Context context, final SharedPreferences prefs, fina mAdaptiveContextPrior = prefs.getBoolean( Settings.PREF_ADAPTIVE_CONTEXT_PRIOR, Defaults.PREF_ADAPTIVE_CONTEXT_PRIOR); + mAdaptiveDebugOverlay = prefs.getBoolean( + Settings.PREF_ADAPTIVE_DEBUG_OVERLAY, + Defaults.PREF_ADAPTIVE_DEBUG_OVERLAY); mSuggestionStripHiddenPerUserSettings = mToolbarMode == ToolbarMode.HIDDEN || mToolbarMode == ToolbarMode.TOOLBAR_KEYS; mOverrideShowingSuggestions = mInputAttributes.mMayOverrideShowingSuggestions diff --git a/app/src/main/java/helium314/keyboard/settings/screens/GestureTypingScreen.kt b/app/src/main/java/helium314/keyboard/settings/screens/GestureTypingScreen.kt index 505ac04c4..1ea93a322 100644 --- a/app/src/main/java/helium314/keyboard/settings/screens/GestureTypingScreen.kt +++ b/app/src/main/java/helium314/keyboard/settings/screens/GestureTypingScreen.kt @@ -88,6 +88,7 @@ fun GestureTypingScreen( val learnOn = prefs.getBoolean(Settings.PREF_ADAPTIVE_KEY_GEOMETRY, Defaults.PREF_ADAPTIVE_KEY_GEOMETRY) val priorOn = prefs.getBoolean(Settings.PREF_ADAPTIVE_CONTEXT_PRIOR, Defaults.PREF_ADAPTIVE_CONTEXT_PRIOR) if (learnOn || priorOn) add(Settings.PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH) + if (learnOn || priorOn) add(Settings.PREF_ADAPTIVE_DEBUG_OVERLAY) // visualize the effect on the live keyboard if (learnOn) add(SettingsWithoutKey.ADAPTIVE_TYPING_STATS) // the learned model only fills with the geometry half } SearchSettingsScreen( @@ -193,6 +194,12 @@ fun createGestureTypingSettings(context: Context) = listOf( R.string.adaptive_context_prior, R.string.adaptive_context_prior_summary) { SwitchPreference(it, Defaults.PREF_ADAPTIVE_CONTEXT_PRIOR) }, + Setting(context, Settings.PREF_ADAPTIVE_DEBUG_OVERLAY, + R.string.adaptive_debug_overlay, R.string.adaptive_debug_overlay_summary) { + SwitchPreference(it, Defaults.PREF_ADAPTIVE_DEBUG_OVERLAY) { + KeyboardSwitcher.getInstance().setThemeNeedsReload() // rebuild so the overlay attaches/detaches at once + } + }, Setting(context, Settings.PREF_ADAPTIVE_KEY_GEOMETRY_STRENGTH, R.string.adaptive_key_geometry_strength) { def -> SliderPreference( name = def.title, diff --git a/app/src/main/res/values/strings.xml b/app/src/main/res/values/strings.xml index 2009204a1..40df2caa1 100644 --- a/app/src/main/res/values/strings.xml +++ b/app/src/main/res/values/strings.xml @@ -390,6 +390,8 @@ Adaptive typing Anticipate likely keys (experimental) Make the next key the suggestions predict slightly easier to press. Averages the top suggestions and is capped so it never overrides a clear press. Applies to taps, not swipes. + Show adaptive targets on keyboard (debug) + Draw each key\'s learned landing point and a halo that grows on keys the suggestions predict, so you can watch the adaptive model work as you type. Visual only — does not change typing. Learned typing model See what the keyboard has learned about your typing Per key: average offset from the key center (where you land), spread (how consistent you are), and number of samples. Higher samples and lower spread mean a stronger, more confident adjustment. Nothing here ever leaves your device. diff --git a/docs/ADAPTIVE_TYPING.md b/docs/ADAPTIVE_TYPING.md index 9aeee09a4..ce45003fe 100644 --- a/docs/ADAPTIVE_TYPING.md +++ b/docs/ADAPTIVE_TYPING.md @@ -124,8 +124,8 @@ The literal tap is committed as-is, so the bias is **hard-capped** and confidenc ## Persistence, export/import, backup compatibility Local data lives in a single SQLite DB, `leantype.db` -(`latin/database/Database.kt`, raw `SQLiteOpenHelper`, currently VERSION 2; clipboard -lives there). The settings backup (`settings/preferences/BackupRestorePreference.kt`, +(`latin/database/Database.kt`, raw `SQLiteOpenHelper`, currently VERSION 4 — clipboard + +the `TOUCH_MODEL` table, which also stores per-key width/height for fraction-of-key math). The settings backup (`settings/preferences/BackupRestorePreference.kt`, Advanced tab) **already zips the entire `leantype.db` and restores it**, and restore is lenient (unknown zip entries skipped; missing columns handled by schema checks in `Database.copyFromDb` + `onUpgrade`). @@ -155,12 +155,48 @@ A visualization page (trust + the reset control live here): e.g. "Z corrected ~18% vs E ~2%." - **Reset** button. +The built page (`AdaptiveTypingStatsScreen`) renders the heatmap on a **mock keyboard** +(`MockKeyboardHeatmap`): each key shows its learned offset as a dot displaced from center +and a spread indicator, both expressed as a **fraction of the key**, so "18px on E" reads +as a visible nudge rather than an abstract number. + +## Live debug overlay ("see it in action") + +A debug toggle — **"Show adaptive targets on keyboard"** — draws the adaptive model +*directly on the live keyboard* so you can watch it work as you type. Implemented as +`AdaptiveTargetsDrawingPreview` (an `AbstractDrawingPreview`, the same overlay mechanism +as the gesture-debug points), drawn on the `DrawingPreviewPlacerView` above the keys: + +- **Learned geometry (Layer B):** for each letter key with a confident learned offset, a + faint ring marks the geometric center, an arrow points to the learned landing target, + and a filled dot marks it — the same shift `KeyDetector` biases taps toward. +- **Context prior (Layer A):** keys the current suggestions predict get a translucent + green halo whose radius grows with the prior weight. Because the prior is rebuilt + between keystrokes, the halos appear/grow/shrink **live as you type** — the keyboard + visibly "leans" toward the likely next key. + +It is purely visual (never changes detection), reads the same live model / prior / +settings the engine uses, and is gated on its pref each frame (zero cost when off). +Repaints are driven by an `AdaptiveKeyContext` change listener fired on each keystroke; +`MainKeyboardView` registers it and feeds the overlay the current keyboard + padding so +markers align with the rendered keys. The halo radius is intentionally exaggerated +relative to the engine's sub-key boost so the effect is legible. We deliberately do **not** +reflow/resize the visible keys (jarring, breaks muscle memory) — an overlay communicates +the same thing without destabilizing typing. + ## Configurability -- Master toggle (opt-in, default off). -- Strength slider (off → gentle tie-break → aggressive). -- Reset learned model. -- Honors incognito / no-learning fields. +All adaptive controls live under one **"Adaptive typing"** section in *Gesture typing* +settings. The two halves are **independently toggleable** (either alone, both, or neither): + +- **Adaptive key geometry** (Layer B, learned offsets) — opt-in, default off. +- **Anticipate likely keys** (Layer A, context prior) — opt-in, default off. +- **Strength slider** — shared by both; shown when either toggle is on (off → gentle + tie-break → aggressive). +- **Learned typing model** stats page (heatmap + reset) — shown when learning is on. +- **Show adaptive targets on keyboard (debug)** — live visualization overlay (below); + shown when either toggle is on. +- Honors incognito / no-learning fields; learning records nothing in those contexts. ## Implementation footprint @@ -215,8 +251,16 @@ stats page to keep it honest. Validate the gesture path explicitly (it's the pri The **learned geometry** (Layer B), by contrast, applies to the entire swipe including its start — every key the stroke passes is matched against its learned-shifted sweet spot — and a swipe's endpoints also feed the model (gesture-endpoint learning). -6. ⬜ **Strength/cap tuning** + optional heatmap visualization + interior-key gesture - learning (needs corner-cutting handling or native alignment). +6. ✅ **Independent context-prior toggle + grouped settings:** Layer A (context prior) is now + its own opt-in toggle ("Anticipate likely keys"), separate from Layer B (learned geometry); + both live under one "Adaptive typing" section and share the strength slider. `KeyDetector` + gates the learned-offset bias and the prior boost independently. +7. ✅ **Heatmap stats page:** `AdaptiveTypingStatsScreen` renders the learned model on a mock + keyboard (offsets as a fraction of each key) + reset. +8. ✅ **Live debug overlay:** `AdaptiveTargetsDrawingPreview` draws learned targets + prior + halos on the real keyboard, morphing as you type (toggle: "Show adaptive targets on keyboard"). +9. ⬜ **Strength/cap tuning** + interior-key gesture learning (needs corner-cutting handling or + native alignment). ## Open questions (tracked) From 06dd2e20f0b668882ba873421253f7cadd2b3672 Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Sun, 7 Jun 2026 08:21:44 -0700 Subject: [PATCH 14/26] Fix context-prior overlay lag and capital-letter bias On-device tracing showed the next-key bias is applied correctly in practice (each tap is biased by the prediction for the prefix typed so far), but two issues made it look/behave wrong: 1. Overlay trailed by ~one key. The prior is rebuilt only when the suggestion strip refreshes, which is debounced ~100 ms; the trace showed a consistent ~113 ms gap between a keystroke and its setSuggested, so for most of the inter-key interval the overlay still showed the previous prediction. When the context prior is enabled, shorten that debounce (PROMPT_PRIOR_UPDATE_DELAY_MS = 30 ms in LatinIME.UIHandler) so the prediction lands before the next tap. 2. Bias silently skipped capital letters. The prior stores lowercase next-characters, but a shifted keyboard reports uppercase key codes, so AdaptiveKeyContext.weight() missed them (priorOnGeo=0.0 on the sentence-initial capital). weight() now folds the queried code to lowercase, so the bias and the overlay halos work on the shifted layout too. Also adds debug-overlay-gated tracing (AdaptivePrior tag) in setSuggestedWords and KeyDetector, plus AdaptiveKeyContext.debugString(), to make this diagnosable on-device, and documents both fixes in docs/ADAPTIVE_TYPING.md. Co-Authored-By: Claude Opus 4.8 --- .../keyboard/keyboard/AdaptiveKeyContext.java | 25 ++++++++++++++++--- .../keyboard/keyboard/KeyDetector.java | 8 ++++++ .../helium314/keyboard/latin/LatinIME.java | 15 ++++++++++- .../keyboard/latin/inputlogic/InputLogic.java | 13 ++++++++++ docs/ADAPTIVE_TYPING.md | 7 ++++++ 5 files changed, 63 insertions(+), 5 deletions(-) diff --git a/app/src/main/java/helium314/keyboard/keyboard/AdaptiveKeyContext.java b/app/src/main/java/helium314/keyboard/keyboard/AdaptiveKeyContext.java index 26871f2f9..b98efca1d 100644 --- a/app/src/main/java/helium314/keyboard/keyboard/AdaptiveKeyContext.java +++ b/app/src/main/java/helium314/keyboard/keyboard/AdaptiveKeyContext.java @@ -97,13 +97,16 @@ public static void clear() { fireChanged(); } - /** Prior weight in [0, 1] for the given key code (0 if none / no prior). */ + /** Prior weight in [0, 1] for the given key code (0 if none / no prior). Case-insensitive: + * the prior stores lowercase next-characters, but a shifted keyboard reports uppercase key + * codes, so we fold to lowercase to match (otherwise the bias/overlay miss capital letters). */ public static float weight(final int code) { final int[] c = sCodes; final float[] w = sWeights; - if (c == null) return 0f; - for (int i = 0; i < c.length; i++) { - if (c[i] == code) return w[i]; + if (c == null || w == null) return 0f; + final int lower = Character.toLowerCase(code); + for (int i = 0; i < c.length && i < w.length; i++) { + if (c[i] == lower) return w[i]; } return 0f; } @@ -111,4 +114,18 @@ public static float weight(final int code) { public static boolean hasPrior() { return sCodes != null; } + + /** Human-readable snapshot of the current prior, e.g. {@code [e=0.60,o=0.40]}, for debug logs. */ + public static String debugString() { + final int[] c = sCodes; + final float[] w = sWeights; + if (c == null || w == null) return "(none)"; + final StringBuilder sb = new StringBuilder("["); + for (int i = 0; i < c.length && i < w.length; i++) { + if (i > 0) sb.append(','); + sb.append((char) c[i]).append('=') + .append(String.format(java.util.Locale.US, "%.2f", w[i])); + } + return sb.append(']').toString(); + } } diff --git a/app/src/main/java/helium314/keyboard/keyboard/KeyDetector.java b/app/src/main/java/helium314/keyboard/keyboard/KeyDetector.java index b7aa54c3d..720fed290 100644 --- a/app/src/main/java/helium314/keyboard/keyboard/KeyDetector.java +++ b/app/src/main/java/helium314/keyboard/keyboard/KeyDetector.java @@ -13,6 +13,7 @@ import helium314.keyboard.latin.database.TouchModelManager; import helium314.keyboard.latin.settings.Settings; import helium314.keyboard.latin.settings.SettingsValues; +import helium314.keyboard.latin.utils.Log; /** * This class handles key detection. @@ -166,6 +167,13 @@ private Key applyAdaptiveBias(final int touchX, final int touchY, final Key geo) best = k; } } + if (sv.mAdaptiveDebugOverlay && hasPrior) { + Log.d("AdaptivePrior", "tap geo='" + (char) geo.getCode() + + "' priorOnGeo=" + AdaptiveKeyContext.weight(geo.getCode()) + + " chosen='" + (char) best.getCode() + "'" + + (best == geo ? "" : " (FLIPPED by bias)") + + " prior=" + AdaptiveKeyContext.debugString()); + } return best == geo ? null : best; } diff --git a/app/src/main/java/helium314/keyboard/latin/LatinIME.java b/app/src/main/java/helium314/keyboard/latin/LatinIME.java index cec4c8537..4c4f1210a 100644 --- a/app/src/main/java/helium314/keyboard/latin/LatinIME.java +++ b/app/src/main/java/helium314/keyboard/latin/LatinIME.java @@ -210,6 +210,11 @@ public static final class UIHandler extends LeakGuardHandlerWrapper { private static final int ARG2_UNUSED = 0; private static final int ARG1_TRUE = 1; + // When the adaptive context prior is enabled, suggestions are refreshed with this shorter + // debounce instead of mDelayInMillisecondsToUpdateSuggestions, so the prior (and its debug + // overlay) keep up with the keystroke instead of trailing it by ~one key. See ADAPTIVE_TYPING.md. + private static final long PROMPT_PRIOR_UPDATE_DELAY_MS = 30; + private int mDelayInMillisecondsToUpdateSuggestions; private int mDelayInMillisecondsToUpdateShiftState; @@ -294,8 +299,16 @@ public void handleMessage(@NonNull final Message msg) { } public void postUpdateSuggestionStrip(final int inputStyle) { + long delay = mDelayInMillisecondsToUpdateSuggestions; + final LatinIME latinIme = getOwnerInstance(); + if (latinIme != null) { + final SettingsValues sv = latinIme.mSettings.getCurrent(); + if (sv != null && sv.mAdaptiveContextPrior) { + delay = Math.min(delay, PROMPT_PRIOR_UPDATE_DELAY_MS); + } + } sendMessageDelayed(obtainMessage(MSG_UPDATE_SUGGESTION_STRIP, inputStyle, - 0 /* ignored */), mDelayInMillisecondsToUpdateSuggestions); + 0 /* ignored */), delay); } public void postReopenDictionaries() { diff --git a/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java b/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java index 5536ba519..da89829f2 100644 --- a/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java +++ b/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java @@ -1385,6 +1385,19 @@ public void setSuggestedWords(final SuggestedWords suggestedWords) { final int nextCharIndex = mWordComposer.isComposingWord() ? mWordComposer.getTypedWord().length() : 0; helium314.keyboard.keyboard.AdaptiveKeyContext.update(suggestedWords, nextCharIndex); + if (svAdaptive.mAdaptiveDebugOverlay) { + final StringBuilder words = new StringBuilder(); + for (int i = 0; i < Math.min(5, suggestedWords.size()); i++) { + if (i > 0) words.append('|'); + words.append(suggestedWords.getWord(i)); + } + Log.d("AdaptivePrior", "setSuggested style=" + suggestedWords.mInputStyle + + " composing=" + mWordComposer.isComposingWord() + + " typed='" + mWordComposer.getTypedWord() + "'" + + " pos=" + nextCharIndex + + " words=[" + words + "]" + + " -> prior=" + helium314.keyboard.keyboard.AdaptiveKeyContext.debugString()); + } } else { helium314.keyboard.keyboard.AdaptiveKeyContext.clear(); } diff --git a/docs/ADAPTIVE_TYPING.md b/docs/ADAPTIVE_TYPING.md index ce45003fe..75ccaf2a6 100644 --- a/docs/ADAPTIVE_TYPING.md +++ b/docs/ADAPTIVE_TYPING.md @@ -78,6 +78,13 @@ current position weighted by the candidate's score; sum per letter. Example: typ `H`, candidates `Hello`/`Hey`/`He` → `e` dominates → **E's tap target grows slightly for the next tap.** Recomputed live; never persisted. +The prior is rebuilt from the suggestion strip, which is normally debounced ~100 ms; that +made the prior (and the debug overlay) trail the keystroke by ~one key. When the prior is +enabled we shorten that debounce (`PROMPT_PRIOR_UPDATE_DELAY_MS` in `LatinIME.UIHandler`) +so the prediction is ready before the next tap. Lookups are case-insensitive +(`AdaptiveKeyContext.weight` folds to lowercase) so the bias also applies on the shifted +keyboard, where keys report uppercase codes. + For **gestures** there is no single "next key" to enlarge mid-stroke, so the context prior is tap-only. The contextual-likelihood part for gestures is already handled by the native language model (word-level). The new gesture win is Layer B (learned From f6657db6ed8125fc0857fc4e5f318d01f6b27fca Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Sun, 7 Jun 2026 08:50:38 -0700 Subject: [PATCH 15/26] Scope prior debounce reduction to the debug overlay only MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The suggestion-update debounce (~100 ms) exists because the suggestion compute blocks the UI thread (performUpdateSuggestionStripSync waits on holder.get with a 200 ms timeout); the debounce coalesces that expensive compute so fast typing isn't hit by one block per keystroke. The functional context-prior bias already works at the full debounce (at normal speed the prior is ready before the next tap). Only the debug overlay visibly trails. So instead of shortening the debounce whenever the prior is enabled, shorten it only while the debug overlay is also on — a temporary diagnostic — and raise the value from 30 ms to 50 ms. Ordinary typing (overlay off) keeps the full debounce and is unaffected. Co-Authored-By: Claude Opus 4.8 --- .../java/helium314/keyboard/latin/LatinIME.java | 14 +++++++++----- docs/ADAPTIVE_TYPING.md | 16 ++++++++++------ 2 files changed, 19 insertions(+), 11 deletions(-) diff --git a/app/src/main/java/helium314/keyboard/latin/LatinIME.java b/app/src/main/java/helium314/keyboard/latin/LatinIME.java index 4c4f1210a..e0d81325b 100644 --- a/app/src/main/java/helium314/keyboard/latin/LatinIME.java +++ b/app/src/main/java/helium314/keyboard/latin/LatinIME.java @@ -210,10 +210,12 @@ public static final class UIHandler extends LeakGuardHandlerWrapper { private static final int ARG2_UNUSED = 0; private static final int ARG1_TRUE = 1; - // When the adaptive context prior is enabled, suggestions are refreshed with this shorter - // debounce instead of mDelayInMillisecondsToUpdateSuggestions, so the prior (and its debug - // overlay) keep up with the keystroke instead of trailing it by ~one key. See ADAPTIVE_TYPING.md. - private static final long PROMPT_PRIOR_UPDATE_DELAY_MS = 30; + // While the adaptive debug overlay is on, suggestions are refreshed with this shorter + // debounce instead of mDelayInMillisecondsToUpdateSuggestions, so the visualized prior keeps + // up with the keystroke instead of trailing it by ~one key. Scoped to the overlay (a + // temporary diagnostic) on purpose: the normal ~100 ms debounce coalesces the UI-blocking + // suggestion compute, so we don't shorten it during ordinary typing. See ADAPTIVE_TYPING.md. + private static final long PROMPT_PRIOR_UPDATE_DELAY_MS = 50; private int mDelayInMillisecondsToUpdateSuggestions; private int mDelayInMillisecondsToUpdateShiftState; @@ -303,7 +305,9 @@ public void postUpdateSuggestionStrip(final int inputStyle) { final LatinIME latinIme = getOwnerInstance(); if (latinIme != null) { final SettingsValues sv = latinIme.mSettings.getCurrent(); - if (sv != null && sv.mAdaptiveContextPrior) { + // Only when actively visualizing the prior: overlay on AND prior on. Ordinary + // typing keeps the full debounce so fast typists aren't hit by extra computes. + if (sv != null && sv.mAdaptiveDebugOverlay && sv.mAdaptiveContextPrior) { delay = Math.min(delay, PROMPT_PRIOR_UPDATE_DELAY_MS); } } diff --git a/docs/ADAPTIVE_TYPING.md b/docs/ADAPTIVE_TYPING.md index 75ccaf2a6..fea5bd888 100644 --- a/docs/ADAPTIVE_TYPING.md +++ b/docs/ADAPTIVE_TYPING.md @@ -78,12 +78,16 @@ current position weighted by the candidate's score; sum per letter. Example: typ `H`, candidates `Hello`/`Hey`/`He` → `e` dominates → **E's tap target grows slightly for the next tap.** Recomputed live; never persisted. -The prior is rebuilt from the suggestion strip, which is normally debounced ~100 ms; that -made the prior (and the debug overlay) trail the keystroke by ~one key. When the prior is -enabled we shorten that debounce (`PROMPT_PRIOR_UPDATE_DELAY_MS` in `LatinIME.UIHandler`) -so the prediction is ready before the next tap. Lookups are case-insensitive -(`AdaptiveKeyContext.weight` folds to lowercase) so the bias also applies on the shifted -keyboard, where keys report uppercase codes. +The prior is rebuilt from the suggestion strip, which is normally debounced ~100 ms (that +debounce exists because the suggestion compute *blocks the UI thread* — see +`InputLogic.performUpdateSuggestionStripSync`, which waits on `holder.get(..., 200 ms)` — so +it coalesces the expensive compute during fast typing). At that cadence the prior is still +ready before the next tap at normal speed (the functional bias works), but the *debug +overlay* visibly trails by ~one key. So **only while the debug overlay is on** we shorten +the debounce to 50 ms (`PROMPT_PRIOR_UPDATE_DELAY_MS` in `LatinIME.UIHandler`) so the +visualization keeps up; ordinary typing (overlay off) keeps the full debounce and is +unaffected. Lookups are case-insensitive (`AdaptiveKeyContext.weight` folds to lowercase) +so the bias also applies on the shifted keyboard, where keys report uppercase codes. For **gestures** there is no single "next key" to enlarge mid-stroke, so the context prior is tap-only. The contextual-likelihood part for gestures is already handled by From 6ae3996620995e68a16e8c2fb535c309c83c201d Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Sun, 7 Jun 2026 09:12:02 -0700 Subject: [PATCH 16/26] Compute prior immediately while debug overlay is on MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 50 ms (and even 30 ms) overlay debounce was easy to out-type. A truly non-blocking async refresh isn't safe here: the suggestion compute reads the non-thread-safe WordComposer on a background thread, and the existing design keeps that safe by blocking the UI during the compute (the only existing async compute runs during gestures, when there's no concurrent typing). Doing it async during typing would risk a torn read / crash and would need a composer snapshot — too much for a debug visualization. Instead, drop the suggestion debounce to 0 while the debug overlay + the context prior are both on. The overlay already repaints the instant the prior updates (the AdaptiveKeyContext change listener fires inside setSuggestedWords), so computing immediately is the safe equivalent of "update as soon as the suggestion is made". The only remaining latency is the compute itself (~5-10 ms release, ~20 ms debug build). Ordinary typing (overlay off) keeps the full, smooth debounce. Reverts the earlier PROMPT_PRIOR_UPDATE_DELAY_MS approach. Co-Authored-By: Claude Opus 4.8 --- .../helium314/keyboard/latin/LatinIME.java | 28 +++++++++---------- docs/ADAPTIVE_TYPING.md | 12 ++++++-- 2 files changed, 23 insertions(+), 17 deletions(-) diff --git a/app/src/main/java/helium314/keyboard/latin/LatinIME.java b/app/src/main/java/helium314/keyboard/latin/LatinIME.java index e0d81325b..378ff33cd 100644 --- a/app/src/main/java/helium314/keyboard/latin/LatinIME.java +++ b/app/src/main/java/helium314/keyboard/latin/LatinIME.java @@ -210,13 +210,6 @@ public static final class UIHandler extends LeakGuardHandlerWrapper { private static final int ARG2_UNUSED = 0; private static final int ARG1_TRUE = 1; - // While the adaptive debug overlay is on, suggestions are refreshed with this shorter - // debounce instead of mDelayInMillisecondsToUpdateSuggestions, so the visualized prior keeps - // up with the keystroke instead of trailing it by ~one key. Scoped to the overlay (a - // temporary diagnostic) on purpose: the normal ~100 ms debounce coalesces the UI-blocking - // suggestion compute, so we don't shorten it during ordinary typing. See ADAPTIVE_TYPING.md. - private static final long PROMPT_PRIOR_UPDATE_DELAY_MS = 50; - private int mDelayInMillisecondsToUpdateSuggestions; private int mDelayInMillisecondsToUpdateShiftState; @@ -302,13 +295,20 @@ public void handleMessage(@NonNull final Message msg) { public void postUpdateSuggestionStrip(final int inputStyle) { long delay = mDelayInMillisecondsToUpdateSuggestions; - final LatinIME latinIme = getOwnerInstance(); - if (latinIme != null) { - final SettingsValues sv = latinIme.mSettings.getCurrent(); - // Only when actively visualizing the prior: overlay on AND prior on. Ordinary - // typing keeps the full debounce so fast typists aren't hit by extra computes. - if (sv != null && sv.mAdaptiveDebugOverlay && sv.mAdaptiveContextPrior) { - delay = Math.min(delay, PROMPT_PRIOR_UPDATE_DELAY_MS); + // While the debug overlay + context prior are both on, compute the prediction + // immediately (no debounce) so the visualization keeps up with fast typing. The overlay + // repaints the instant the prior updates, so "no debounce" is the safe equivalent of + // "update as soon as the suggestion is made". The remaining floor is the suggestion + // compute itself, which briefly blocks the UI thread by design (see + // performUpdateSuggestionStripSync) — fast on release, slower on debug builds. Scoped to + // the overlay so ordinary typing keeps the full, smooth debounce. + if (inputStyle == SuggestedWords.INPUT_STYLE_TYPING) { + final LatinIME latinIme = getOwnerInstance(); + if (latinIme != null) { + final SettingsValues sv = latinIme.mSettings.getCurrent(); + if (sv != null && sv.mAdaptiveDebugOverlay && sv.mAdaptiveContextPrior) { + delay = 0; + } } } sendMessageDelayed(obtainMessage(MSG_UPDATE_SUGGESTION_STRIP, inputStyle, diff --git a/docs/ADAPTIVE_TYPING.md b/docs/ADAPTIVE_TYPING.md index fea5bd888..23565e5fa 100644 --- a/docs/ADAPTIVE_TYPING.md +++ b/docs/ADAPTIVE_TYPING.md @@ -83,9 +83,15 @@ debounce exists because the suggestion compute *blocks the UI thread* — see `InputLogic.performUpdateSuggestionStripSync`, which waits on `holder.get(..., 200 ms)` — so it coalesces the expensive compute during fast typing). At that cadence the prior is still ready before the next tap at normal speed (the functional bias works), but the *debug -overlay* visibly trails by ~one key. So **only while the debug overlay is on** we shorten -the debounce to 50 ms (`PROMPT_PRIOR_UPDATE_DELAY_MS` in `LatinIME.UIHandler`) so the -visualization keeps up; ordinary typing (overlay off) keeps the full debounce and is +overlay* visibly trails by ~one key. So **only while the debug overlay is on** we drop the +debounce to 0 in `LatinIME.UIHandler.postUpdateSuggestionStrip` (compute the prediction +immediately); since the overlay repaints the instant the prior updates, that is the safe +equivalent of "update as soon as the suggestion is made". The remaining floor is the +compute itself (~5–10 ms release, ~20 ms debug), which briefly blocks the UI by design. A +fully non-blocking async refresh would be snappier still, but the suggestion compute reads +the non-thread-safe `WordComposer` on the background thread — the blocking design exists to +avoid that race during typing — so doing it safely would need a composer snapshot (not worth +it for a debug visualization). Ordinary typing (overlay off) keeps the full debounce and is unaffected. Lookups are case-insensitive (`AdaptiveKeyContext.weight` folds to lowercase) so the bias also applies on the shifted keyboard, where keys report uppercase codes. From 4be3a3614009402173a0690ed36b8c6222ae6408 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Mon, 8 Jun 2026 02:12:47 +0000 Subject: [PATCH 17/26] chore: update README badges [skip ci] --- docs/badges/downloads.svg | 2 +- docs/badges/stars.svg | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/badges/downloads.svg b/docs/badges/downloads.svg index 044abfbad..eaa6b356a 100644 --- a/docs/badges/downloads.svg +++ b/docs/badges/downloads.svg @@ -1 +1 @@ -DownloadsDownloads2886228862 +DownloadsDownloads2908129081 diff --git a/docs/badges/stars.svg b/docs/badges/stars.svg index a3ec88a5a..9204362ae 100644 --- a/docs/badges/stars.svg +++ b/docs/badges/stars.svg @@ -1 +1 @@ -StarsStars471471 +StarsStars472472 From 9d8e99a33248d3f808cd5f35c31142a20a508ab7 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Tue, 9 Jun 2026 01:50:05 +0000 Subject: [PATCH 18/26] chore: update README badges [skip ci] --- docs/badges/downloads.svg | 2 +- docs/badges/stars.svg | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/badges/downloads.svg b/docs/badges/downloads.svg index eaa6b356a..a81dd9e98 100644 --- a/docs/badges/downloads.svg +++ b/docs/badges/downloads.svg @@ -1 +1 @@ -DownloadsDownloads2908129081 +DownloadsDownloads2925829258 diff --git a/docs/badges/stars.svg b/docs/badges/stars.svg index 9204362ae..bd3adcd64 100644 --- a/docs/badges/stars.svg +++ b/docs/badges/stars.svg @@ -1 +1 @@ -StarsStars472472 +StarsStars475475 From b1729bdda98a01e6194e666eb50007beed0a052e Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Sun, 7 Jun 2026 22:21:21 -0700 Subject: [PATCH 19/26] Fix live-converge word casing (stuck all-caps + dropped auto-cap) Casing bugs specific to two-thumb live-converge / multi-part re-recognition, where every extending tap or swipe REPLACES the whole composing word with the recognizer's fresh (lowercase) output: - Dropped auto-cap (#5): "Hello" -> "hellow" on the first extension. - Stuck all-caps (#4): a short ambiguous swipe whose top pick is an all-caps acronym ("CSA") set WordComposer.isAllUpperCase, which forced every later suggestion upper, so the word stuck in caps ("CSA"->"CAN"->"CAME"). - Swipe-extension downcasing: "Was"+swipe -> "wait" (a second swipe re-entered onStartBatchInput and re-captured the now-cleared shift state). Approach: separate a word's casing INTENT from the recognizer's letters. - WordComposer.mCapitalizedMode is the persistent per-word intent: seeded at word start from auto-cap + shift, it survives the setBatchInputWord rebuild and is cleared only at commitWord. Exposed via getCapitalizedMode(). - New InputLogic.applyComposingCase(lemma, capsMode, locale) treats the recognizer output as a casing-NEUTRAL lemma (lowercased first) and re-applies the intent. Lowercasing first dissolves #4 at the source: the composing word is never all-caps, so isAllUpperCase never arms. - onStartBatchInput captures the intent only for a FRESH word (gated on !extendComposingWord), so an extending gesture preserves the first fragment's intent instead of re-capturing the auto-cleared shift state (#5 / "Wait"). - Fresh-word gesture capitalization is unchanged (still mShiftModeAtGestureStart, captured before any state mutates), so plain glide typing is byte-identical and a standalone acronym swipe ("CSA") still stays as-is. Tests: applyComposingCase covered directly (pure, native-free) plus intent lifecycle tests in InputLogicTest. :app:testOfflineDebugUnitTest green except the 3 documented pre-existing failures. Fixes #4, #5. Co-Authored-By: Claude Opus 4.8 --- .../keyboard/latin/WordComposer.java | 13 +++ .../keyboard/latin/inputlogic/InputLogic.java | 98 ++++++++++++++++--- .../keyboard/latin/InputLogicTest.kt | 85 ++++++++++++++++ 3 files changed, 182 insertions(+), 14 deletions(-) diff --git a/app/src/main/java/helium314/keyboard/latin/WordComposer.java b/app/src/main/java/helium314/keyboard/latin/WordComposer.java index e8d93e519..a7e1b8034 100644 --- a/app/src/main/java/helium314/keyboard/latin/WordComposer.java +++ b/app/src/main/java/helium314/keyboard/latin/WordComposer.java @@ -431,6 +431,19 @@ public void setCapitalizedModeAtStartComposingTime(final int mode) { mCapitalizedMode = mode; } + /** + * The capitalization intent captured when this word started composing (one of the + * {@code CAPS_MODE_*} constants). It is seeded from auto-cap + shift state at word-start, + * survives {@link #reset()} (so it persists across a {@link #setBatchInputWord} rebuild), and + * is cleared only in {@link #commitWord}. This makes it the persistent per-word source of + * truth for casing — used by the live-converge re-recognition path so that re-replacing the + * word never loses (or latches) its case. + * @return the capitalized mode for the current word + */ + public int getCapitalizedMode() { + return mCapitalizedMode; + } + /** * Before fetching suggestions, we don't necessarily know about the capitalized mode yet. *

diff --git a/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java b/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java index 975f571a8..7c322d658 100644 --- a/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java +++ b/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java @@ -115,10 +115,11 @@ public final class InputLogic { // GESTURE-START, not gesture-end (so a long gesture doesn't lose the promotion). private boolean mGestureExtendsByTapPromotion; - // Snapshot of {@code keyboardSwitcher.getKeyboardShiftMode()} captured at the start of - // each gesture. Used by {@link #onUpdateTailBatchInputCompleted} to capitalize the - // recognizer's lowercase output. We can't read getKeyboardShiftMode() at gesture-end - // because the keyboard typically auto-clears the shifted state during the gesture. + // Snapshot of {@code keyboardSwitcher.getKeyboardShiftMode()} captured at the very start of + // each gesture, BEFORE any state mutates (a prior word may auto-commit / the shift indicator + // may auto-clear within onStartBatchInput). Used ONLY by the FRESH-word capitalization in + // {@link #onUpdateTailBatchInputCompleted}. The live-converge / multi-part EXTEND path uses the + // persistent {@link WordComposer#getCapitalizedMode()} instead — see that method's javadoc. private int mShiftModeAtGestureStart = WordComposer.CAPS_MODE_OFF; /** Set to true at the end of {@link #onCombiningGraceExpired} when an autospace was * written, so the next punctuation tap in {@link #handleSeparatorEvent} can strip it @@ -733,9 +734,10 @@ public void onStartBatchInput(final SettingsValues settingsValues, markForceNextSpaceWordStarted(); // Snapshot the keyboard's shift mode BEFORE any state mutates — the shifted indicator // typically auto-clears once the gesture starts moving, so by the time - // onUpdateTailBatchInputCompleted fires the live mode reads as UNSHIFTED. - // We compute the *actual* caps mode (resolves AUTO_SHIFTED into AUTO_SHIFT_LOCKED if - // the input field is in all-caps), so a true all-caps field gives the right answer. + // onUpdateTailBatchInputCompleted fires the live mode reads as UNSHIFTED. This drives the + // FRESH-word capitalization only; the EXTEND path uses WordComposer.mCapitalizedMode (the + // persistent per-word intent). We compute the *actual* caps mode (resolves AUTO_SHIFTED + // into AUTO_SHIFT_LOCKED if the input field is all-caps) so a true all-caps field is right. mShiftModeAtGestureStart = getActualCapsMode(settingsValues, keyboardSwitcher.getKeyboardShiftMode()); mWordBeingCorrectedByCursor = null; mInputLogicHandler.onStartBatchInput(); @@ -855,8 +857,19 @@ public void onStartBatchInput(final SettingsValues settingsValues, } } mConnection.endBatchEdit(); - mWordComposer.setCapitalizedModeAtStartComposingTime( - getActualCapsMode(settingsValues, keyboardSwitcher.getKeyboardShiftMode())); + // Capture the word's casing intent ONLY when this gesture starts a fresh word. When it + // EXTENDS an existing composing word (multi-part swipe+swipe / manual spacing), the intent + // belongs to the word's first fragment and must be preserved: the keyboard auto-clears its + // shifted indicator after the first gesture, so re-capturing here would read UNSHIFTED and + // wrongly downcase a word that started capitalized ("Was"+swipe -> "wait" instead of + // "Wait"). mCapitalizedMode survives the setBatchInputWord rebuild and is cleared at + // commitWord, so leaving it untouched keeps the original intent alive across the extension. + // (Live-converge tap extensions bypass onStartBatchInput entirely, so they were already + // safe; this closes the same gap for swipe extensions.) + if (!extendComposingWord) { + mWordComposer.setCapitalizedModeAtStartComposingTime( + getActualCapsMode(settingsValues, keyboardSwitcher.getKeyboardShiftMode())); + } } /* @@ -3743,6 +3756,44 @@ private static boolean isPlainLetterWord(final String s) { return true; } + /** + * Apply a word's casing INTENT to a casing-neutral lemma. Used by the live-converge + * (merged-trail) re-recognition path, which replaces the whole composing word on every + * extending tap. The recognizer's output is treated as letters only — its own casing (e.g. an + * all-caps dictionary acronym that the engine happened to rank first) is discarded by + * lowercasing — and the case is then re-derived from the persistent per-word intent + * ({@link WordComposer#getCapitalizedMode()}, seeded at word-start from auto-cap + shift and + * alive until commit). This keeps a sentence-start capital across every re-converge (issue #5) + * and, because the lemma is lowercased first, prevents an unsolicited all-caps result from + * latching the whole word in caps (issue #4). + * + *

Pure function of its inputs (no engine / native lib), so the casing behaviour is unit + * testable directly. {@code public static} for that reason. + * + * @param lemma the recognizer's word output (casing not trusted) + * @param capitalizedMode one of the {@link WordComposer} {@code CAPS_MODE_*} constants + * @param locale locale for case mapping + * @return the lemma cased to match the intent + */ + public static String applyComposingCase(final String lemma, final int capitalizedMode, + final Locale locale) { + if (lemma == null || lemma.isEmpty()) return lemma; + final String lower = lemma.toLowerCase(locale); + switch (capitalizedMode) { + case WordComposer.CAPS_MODE_AUTO_SHIFT_LOCKED: + case WordComposer.CAPS_MODE_MANUAL_SHIFT_LOCKED: + // Deliberate caps-lock (or an all-caps input field) — uppercase the whole word. + return lower.toUpperCase(locale); + case WordComposer.CAPS_MODE_AUTO_SHIFTED: + case WordComposer.CAPS_MODE_MANUAL_SHIFTED: + // Sentence-start / shift — first letter only. + return StringUtils.capitalizeFirstCodePoint(lower, locale); + default: + // CAPS_MODE_OFF — no intent, leave the neutral lemma lowercase. + return lower; + } + } + private boolean textBeforeCursorMayBeUrlOrSimilar(final SettingsValues settingsValues, final Boolean forAutoSpace) { // URL / mail field and no space -> may be URL if (InputTypeUtils.isUriOrEmailType(settingsValues.mInputAttributes.mInputType) && @@ -3892,9 +3943,12 @@ public void onUpdateTailBatchInputCompleted(final SettingsValues settingsValues, // those continuation gestures should append in the casing the user already chose for // the start of the word. if (!extendExistingCompose && !batchInputText.isEmpty()) { - // Use the shift mode captured at gesture-start, not the live mode — the - // keyboard auto-clears the shifted indicator during the gesture, so a live - // read here always returns UNSHIFTED. + // Fresh-word capitalization: use the shift mode captured at gesture-start, not the live + // mode — the keyboard auto-clears the shifted indicator during the gesture, so a live + // read here always returns UNSHIFTED. This is the long-standing path for a plain single + // gesture; it ADDS a capital to the recognizer's lowercase output and deliberately does + // NOT neutralize an intrinsic all-caps result, so a standalone acronym swipe ("CSA") + // stays as-is. (The EXTEND path below handles re-cased re-recognition separately.) final int shiftMode = mShiftModeAtGestureStart; if (shiftMode == WordComposer.CAPS_MODE_MANUAL_SHIFTED || shiftMode == WordComposer.CAPS_MODE_AUTO_SHIFTED) { @@ -3904,8 +3958,24 @@ public void onUpdateTailBatchInputCompleted(final SettingsValues settingsValues, batchInputText = batchInputText.toUpperCase(settingsValues.mLocale); } } - // Clear so a stale value from a previous gesture can't leak into a non-gesture - // commit later. + // Live-converge (#1.7) casing — gated to the merged-trail re-recognition path, so a plain + // single gesture (incl. a standalone acronym swipe, handled by the block above) is + // untouched. A merged-trail commit REPLACES the whole word with the recognizer's fresh + // output on every extending tap, which otherwise: + // - dropped the first-letter capital the word started with (issue #5); and + // - could latch an unsolicited all-caps acronym ("CSA"), which then stuck the whole word + // in caps via WordComposer.isAllUpperCase forcing every later suggestion upper (#4). + // The fix treats the recognizer output as a casing-NEUTRAL lemma and re-applies the word's + // persistent intent (WordComposer.mCapitalizedMode). Lowercasing the lemma first is what + // dissolves #4 at the source: the composing word is never all-caps, so isAllUpperCase + // never arms and Suggest stops force-uppercasing — no shift-lock special-case needed. + if (usedMergedTrail) { + batchInputText = applyComposingCase(batchInputText, mWordComposer.getCapitalizedMode(), + settingsValues.mLocale); + } + // Clear the fresh-word snapshot so a stale value from this gesture can't leak into a + // later non-gesture commit. (The persistent extend intent lives in mCapitalizedMode and + // is cleared by WordComposer.commitWord, not here.) mShiftModeAtGestureStart = WordComposer.CAPS_MODE_OFF; final String composedText = prevTypedWord + batchInputText; if (settingsValues.mGestureDebugDrawPoints) { diff --git a/app/src/test/java/helium314/keyboard/latin/InputLogicTest.kt b/app/src/test/java/helium314/keyboard/latin/InputLogicTest.kt index c55350a58..66ade1c57 100644 --- a/app/src/test/java/helium314/keyboard/latin/InputLogicTest.kt +++ b/app/src/test/java/helium314/keyboard/latin/InputLogicTest.kt @@ -692,6 +692,91 @@ class InputLogicTest { assertFalse(composer.isExtendBatchInputBaseSet) } + // --- Live-converge casing (#4 stuck all-caps, #5 dropped auto-cap). The merged-trail + // re-recognition path replaces the whole composing word on every extending tap; its casing is + // derived from the persistent per-word intent (WordComposer.mCapitalizedMode) applied to a + // casing-NEUTRAL lemma, via InputLogic.applyComposingCase. That transform is a pure function + // (no native engine), so the casing behaviour is testable here directly — the on-device + // recognition that produces the lemma is not (no gesture lib / tap coords in the JVM harness). + private val enLocale = "en".constructLocale() + + // #5: a word that started capitalized (sentence-start / shift) keeps its leading capital when + // the recognizer re-resolves it lowercase on an extending tap. "Hello" must not become "hellow". + @Test fun applyComposingCaseKeepsLeadingCapitalAcrossReconverge() { + assertEquals("Hellow", + InputLogic.applyComposingCase("hellow", WordComposer.CAPS_MODE_AUTO_SHIFTED, enLocale)) + assertEquals("Hellow", + InputLogic.applyComposingCase("hellow", WordComposer.CAPS_MODE_MANUAL_SHIFTED, enLocale)) + } + + // #4: an unsolicited all-caps recognizer result ("CSA") is neutralized to the word's intent + // instead of latching. Sentence-start intent -> "Can"; no intent (mid-sentence) -> "can". + // This is the exact case the old prior-word heuristic got wrong (it produced "can" at a + // sentence start because it could only see the previous all-caps fragment). + @Test fun applyComposingCaseNeutralizesUnsolicitedAllCaps() { + assertEquals("Can", + InputLogic.applyComposingCase("CAN", WordComposer.CAPS_MODE_AUTO_SHIFTED, enLocale)) + assertEquals("can", + InputLogic.applyComposingCase("CAN", WordComposer.CAPS_MODE_OFF, enLocale)) + } + + // Deliberate caps-lock (or an all-caps input field) still produces all-caps on the + // merged-trail path — the one case where the whole word legitimately stays upper. + @Test fun applyComposingCaseUppercasesUnderShiftLock() { + assertEquals("CAME", + InputLogic.applyComposingCase("came", WordComposer.CAPS_MODE_MANUAL_SHIFT_LOCKED, enLocale)) + assertEquals("CAME", + InputLogic.applyComposingCase("CAME", WordComposer.CAPS_MODE_AUTO_SHIFT_LOCKED, enLocale)) + } + + // No casing intent: the neutral lemma is left lowercase regardless of the recognizer's own + // casing, so a mid-sentence re-converge never injects stray capitals. + @Test fun applyComposingCaseLeavesLowercaseWhenNoIntent() { + assertEquals("game", + InputLogic.applyComposingCase("Game", WordComposer.CAPS_MODE_OFF, enLocale)) + assertEquals("game", + InputLogic.applyComposingCase("game", WordComposer.CAPS_MODE_OFF, enLocale)) + } + + // Defensive: empty / null lemma passes through untouched (the caller also guards, but the + // helper must be safe on its own). + @Test fun applyComposingCaseHandlesEmptyAndNull() { + assertEquals("", InputLogic.applyComposingCase("", WordComposer.CAPS_MODE_AUTO_SHIFTED, enLocale)) + assertEquals(null, InputLogic.applyComposingCase(null, WordComposer.CAPS_MODE_AUTO_SHIFTED, enLocale)) + } + + // An EXTENDING gesture (swipe+swipe / manual-spacing multi-part) must NOT re-capture the + // word's casing intent at onStartBatchInput. The keyboard auto-clears its shift indicator + // after the first gesture, so re-capturing would overwrite the word-start intent with + // UNSHIFTED and downcase a capitalized word ("Was"+swipe -> "wait" instead of "Wait", the + // on-device regression this guards). Intent must survive to the merged-trail casing step. + // (Live-converge tap extensions bypass onStartBatchInput, so they were never affected.) + @Test fun extendingGestureStartPreservesCasingIntent() { + reset() + latinIME.prefs().edit { putBoolean(Settings.PREF_GESTURE_MANUAL_SPACING, true) } + chainInput("wa") // open a composing word, cursor at end + // Simulate the word having started while shifted (as "Was" did on-device). + composer.setCapitalizedModeAtStartComposingTime(WordComposer.CAPS_MODE_MANUAL_SHIFTED) + // A second gesture starts to EXTEND it (manual spacing -> extendComposingWord = true). + inputLogic.onStartBatchInput(settingsValues, KeyboardSwitcher.getInstance(), latinIME.mHandler) + handleMessages() + // Pre-fix this was clobbered to CAPS_MODE_OFF by the unconditional re-capture. + assertEquals(WordComposer.CAPS_MODE_MANUAL_SHIFTED, composer.capitalizedMode) + } + + // Sanity: a FRESH gesture (no composing word) still captures the current intent — the guard + // must not freeze a stale mode from a previous word. + @Test fun freshGestureStartStillCapturesCasingIntent() { + reset() + latinIME.prefs().edit { putBoolean(Settings.PREF_GESTURE_MANUAL_SPACING, true) } + composer.setCapitalizedModeAtStartComposingTime(WordComposer.CAPS_MODE_MANUAL_SHIFT_LOCKED) + // No composing word -> extendComposingWord = false -> intent is re-captured from the + // keyboard (not shift-locked on a fresh field), not frozen at the stale locked value. + inputLogic.onStartBatchInput(settingsValues, KeyboardSwitcher.getInstance(), latinIME.mHandler) + handleMessages() + assertFalse(composer.capitalizedMode == WordComposer.CAPS_MODE_MANUAL_SHIFT_LOCKED) + } + // Static-seed reachability guard. PointerTracker's tap-seed path (sLastLetterTap*) is gated // on (!isMultipartComposeActive() && mCombiningGraceMs > 0). But grace > 0 forces multi-part // composition active, so that conjunction is unsatisfiable and the seed is currently From 26b789f433db51910f44e874e5c4a990b7445eea Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Tue, 9 Jun 2026 06:26:44 -0700 Subject: [PATCH 20/26] docs: composing-word source-of-truth refactor plan Plan of record for a multi-phase refactor of two-thumb / multi-part composition: make the editor text + composing region the single source of truth for the current word, and derive stroke geometry from text on demand (getCoordinatesForCurrentKeyboard) instead of maintaining drift-prone parallel state (mLiveStroke, mInputPointers survives-reset contract, mGestureFragmentBoundaries). Captures the problem (the recurring stale-stroke / desync bug class), the existing foundation already used for tap recorrection, the target design and state inventory, and a 3-phase incremental plan (fragment-pop fix -> live-converge from text -> generalize to mid-word editing) with on-device validation gates. Co-Authored-By: Claude Opus 4.8 --- docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md | 120 +++++++++++++++++++++++++ 1 file changed, 120 insertions(+) create mode 100644 docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md diff --git a/docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md b/docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md new file mode 100644 index 000000000..a6b63f1df --- /dev/null +++ b/docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md @@ -0,0 +1,120 @@ +# Composing word: editor text as the single source of truth (design) + +> Status: **design / not yet implemented**. Tracking: see the "Composing word source of +> truth" epic on the fork. This note is the plan of record for a multi-phase refactor of +> two-thumb / multi-part composition. Update it as phases land. +> +> Grounding: claims below were verified by read-only scouting of `InputLogic`, +> `WordComposer`, `RichInputConnection`. File/symbol refs are a snapshot — prefer the named +> symbols and re-confirm at implementation. + +## Problem + +Two-thumb / multi-part composition keeps several pieces of **gesture state in memory, +parallel to the editor text**, and they drift out of sync: + +- `WordComposer.mInputPointers` — the raw stroke buffer. Intentionally *not* cleared by + `reset()` / backspace (so a gesture commit can re-feed it via `setBatchInputWord`), and + `ResizableIntArray.addAt` overwrites in place **without shrinking the length**. +- `InputLogic.mLiveStroke` — the live-converge accumulator. +- `WordComposer.mExtendBatchInputBase` — the merged-trail base prepended to each new gesture. +- `InputLogic.mGestureFragmentBoundaries` — char-index boundaries for "delete last fragment". + +Because these are a *second* source of truth, anything that edits the text outside the +tracked flow — a backspace, a partial delete, a cursor move, autocorrect, an abnormal +gesture end — leaves them stale. That is the recurring bug class: + +- **Stale-stroke accumulation:** mis-recognize a multi-part word, backspace, retry → the + next swipe merges with leftover geometry and builds an ever-longer garbage word. (Partly + fixed by resetting `mInputPointers` on the first tap of a fresh word; the **fragment-pop** + path is still uncovered — see Phase 1.) +- **Fragment-backspace fragility:** `mGestureFragmentBoundaries` needs band-aid "filter + stale boundaries past current length" logic, only works in specific spacing configs, and + bails entirely when the cursor is in the middle of the word. +- **No mid-word editing:** you cannot move the cursor into the middle of a composing word, + or delete part of it, and continue building with gestures — the in-memory state assumes + append-at-end. + +## Core principle + +**The editor text + the composing region (and the cursor within it) are the single source +of truth for "the current word." Stroke geometry is derived from that text on demand, not +maintained as separate, drift-prone state.** + +A word's plausible stroke is just the sequence of its keys' centers — recoverable from the +*text* alone. So we never need to remember a stroke across edits; we can always reconstruct +one when re-recognition is needed. + +## The foundation already exists + +HeliBoard already reconstructs a composing word from editor text when the cursor lands in a +word — `restartSuggestionsOnWordTouchedByCursor` → `getWordRangeAtCursor` → +`restartSuggestions`, whose core is: + +```java +mWordComposer.setComposingWord(codePoints, getCoordinatesForCurrentKeyboard(codePoints)); +mWordComposer.setCursorPositionWithinWord(...); +mConnection.setComposingRegion(start, end); +``` + +`getCoordinatesForCurrentKeyboard(text)` **synthesizes per-character key-center +coordinates from text alone** — exactly the "derive a stroke from the word" primitive this +design needs. Today it is used for **tap recorrection only**; it is never wired into the +gesture / live-converge path. The refactor is largely about routing two-thumb composition +through this existing machinery instead of around it. + +## Target design + +- **The word being built = the composing region in the editor.** It can be (re)established + from any cursor position via `getWordRangeAtCursor` + `setComposingRegion`, which the + editor already maintains. +- **Live-converge becomes (near) stateless.** To fold a tap/gesture into the current word: + take the current word's *text* → synthesize its key-center stroke + (`getCoordinatesForCurrentKeyboard`) → append the new gesture's raw points → re-recognize. + No `mLiveStroke` accumulator to leak. +- **Fragment boundaries become derived,** not stored. "Delete last fragment" is computed + from the text/recognition at delete time, so there is nothing to keep in sync. +- **Backspace, partial delete, and cursor moves stop being special cases.** Each just + changes the text; the next gesture rebuilds its base from whatever text is actually there. + +### State inventory +- **Retire:** `mLiveStroke`, `mGestureFragmentBoundaries`, the bespoke + `mInputPointers`-survives-reset contract and its clean-up patches. +- **Keep:** `mExtendBatchInputBase` as the *mechanism* for feeding "prefix + new gesture" to + the recognizer (the re-timing logic stays), but the prefix is sourced from text-derived + geometry rather than a stored stroke. +- **Reuse:** `getWordRangeAtCursor`, `setComposingRegion`, `setComposingWord`, + `getCoordinatesForCurrentKeyboard`, `setCursorPositionWithinWord`. + +## Phases (each independently shippable + on-device validated) + +1. **Stop the bleeding in fragment mode.** On a fragment-pop (`tryFragmentBackspace`), + realign `mInputPointers` to the truncated word (rebuild from its key centers) so a + following swipe-extend no longer merges with the pre-pop stroke. Keeps the current + architecture; closes the fragment case the fresh-word reset doesn't cover. Small. +2. **Prove the model.** Make live-converge derive its merge base from the *current word's + text* via `getCoordinatesForCurrentKeyboard`, instead of `mLiveStroke`. Validate on-device + that re-recognition from synthesized key-center geometry holds up. If so, delete + `mLiveStroke`. Medium. +3. **Generalize.** Route all multi-part composition through "composing region as source of + truth": derive fragment boundaries from text, support mid-word cursor + partial-delete + continuation, retire `mGestureFragmentBoundaries`. Larger; the payoff phase. + +## Risks / validation + +- **Synthesized strokes ≠ real strokes.** `getCoordinatesForCurrentKeyboard` yields key + *centers*, not the user's actual curves. Re-recognition from synthesized geometry may + differ from the captured stroke. **This is the primary thing to validate on-device** (the + recognizer may be robust to clean key-center input, or need tuning). Gate each phase on a + real-device check, conservative defaults, and the trace/replay harness where possible. +- **Performance:** re-synthesize + re-recognize per fragment is more work than appending to + a buffer, but the same order as today's live-converge per-tap recognition. +- **Feel:** mid-word continuation and derived fragment behavior are feel calls — ship as + tunable where it matters, decide by typing on it (per the project's feel-driven principle). + +## Why this is the right direction + +Every current bug here is the same disease — parallel gesture state drifting from the +editor. This refactor **removes** that parallel state rather than adding more patches, and +unlocks mid-word cursor editing and partial deletes for free. Each phase reduces state and +is shippable on its own, so the risk is bounded and the wins are incremental. From cb860c6dd337da582620ffba0c0b986e42452eb4 Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Tue, 9 Jun 2026 06:30:44 -0700 Subject: [PATCH 21/26] docs: add explicit trigger model (taps exact; only swipes compose) Clarify scope per review: knowing the current word from the editor is for CONTEXT, not re-typing. Taps stay exact and bypass live-composition entirely (replaces the re-recognize-taps behavior / PREF_MULTIPART_RERECOGNIZE_TAPS); moving the cursor into a word is context-only and must not re-compose; only a swipe added at the cursor (re)composes by building on that word's text. Updated target design, state inventory, and Phase 2/3 accordingly. Co-Authored-By: Claude Opus 4.8 --- docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md | 54 +++++++++++++++++++------- 1 file changed, 40 insertions(+), 14 deletions(-) diff --git a/docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md b/docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md index a6b63f1df..32ab3ab10 100644 --- a/docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md +++ b/docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md @@ -45,6 +45,27 @@ A word's plausible stroke is just the sequence of its keys' centers — recovera *text* alone. So we never need to remember a stroke across edits; we can always reconstruct one when re-recognition is needed. +## Trigger model (what (re)composes, and what doesn't) + +Knowing "the current word" from the editor is for **context**, not for re-typing it. The +only thing that ever re-composes a word is **a swipe**. Concretely: + +- **Taps are exact and bypass live-composition entirely.** A tapped character is inserted + literally, exactly as it is today — it never triggers re-recognition and never reshapes + the surrounding word. (This replaces the current "re-recognize taps within swiped words" + behavior, `PREF_MULTIPART_RERECOGNIZE_TAPS` — taps no longer fold into the stroke.) +- **Moving the cursor into a word is context-only.** Establishing the composing region over + an existing word (so the engine *knows* the word at the cursor) must **not** live-compose, + re-recognize, or alter it. Nothing changes on screen until the user acts. +- **Only a swipe (re)composes.** A swipe added at the cursor builds onto whatever word is + there — typed, swiped, or one the cursor was just moved into — by deriving that word's + key-center stroke from its text and merging the swipe onto it (the merged-trail path). + This is the single entry point to re-recognition. + +Net: the source-of-truth model makes any word *available* to build on, but the user stays +in control — a tap is always a tap, a cursor move is inert, and a word only re-composes +when they deliberately swipe onto it. Space remains the word-submission point. + ## The foundation already exists HeliBoard already reconstructs a composing word from editor text when the cursor lands in a @@ -68,21 +89,23 @@ through this existing machinery instead of around it. - **The word being built = the composing region in the editor.** It can be (re)established from any cursor position via `getWordRangeAtCursor` + `setComposingRegion`, which the editor already maintains. -- **Live-converge becomes (near) stateless.** To fold a tap/gesture into the current word: - take the current word's *text* → synthesize its key-center stroke - (`getCoordinatesForCurrentKeyboard`) → append the new gesture's raw points → re-recognize. - No `mLiveStroke` accumulator to leak. +- **Swipe-on-word composition becomes (near) stateless.** When a *swipe* extends the word + at the cursor: take that word's *text* → synthesize its key-center stroke + (`getCoordinatesForCurrentKeyboard`) → append the swipe's raw points → re-recognize. No + `mLiveStroke` accumulator to leak. Taps never enter this path (see Trigger model). - **Fragment boundaries become derived,** not stored. "Delete last fragment" is computed from the text/recognition at delete time, so there is nothing to keep in sync. - **Backspace, partial delete, and cursor moves stop being special cases.** Each just - changes the text; the next gesture rebuilds its base from whatever text is actually there. + changes the text (a cursor move is otherwise inert — context-only); the *next swipe* + rebuilds its base from whatever text is actually there. ### State inventory -- **Retire:** `mLiveStroke`, `mGestureFragmentBoundaries`, the bespoke - `mInputPointers`-survives-reset contract and its clean-up patches. +- **Retire:** `mLiveStroke` and the whole tap-re-recognition feature + (`PREF_MULTIPART_RERECOGNIZE_TAPS`) — taps become exact again; `mGestureFragmentBoundaries`; + the bespoke `mInputPointers`-survives-reset contract and its clean-up patches. - **Keep:** `mExtendBatchInputBase` as the *mechanism* for feeding "prefix + new gesture" to the recognizer (the re-timing logic stays), but the prefix is sourced from text-derived - geometry rather than a stored stroke. + geometry rather than a stored stroke. It is armed only on the swipe path. - **Reuse:** `getWordRangeAtCursor`, `setComposingRegion`, `setComposingWord`, `getCoordinatesForCurrentKeyboard`, `setCursorPositionWithinWord`. @@ -92,13 +115,16 @@ through this existing machinery instead of around it. realign `mInputPointers` to the truncated word (rebuild from its key centers) so a following swipe-extend no longer merges with the pre-pop stroke. Keeps the current architecture; closes the fragment case the fresh-word reset doesn't cover. Small. -2. **Prove the model.** Make live-converge derive its merge base from the *current word's - text* via `getCoordinatesForCurrentKeyboard`, instead of `mLiveStroke`. Validate on-device - that re-recognition from synthesized key-center geometry holds up. If so, delete - `mLiveStroke`. Medium. +2. **Prove the model (swipe-on-word).** Make a *swipe* that extends the word at the cursor + derive its merge base from that word's *text* via `getCoordinatesForCurrentKeyboard`, + instead of `mLiveStroke`. Make taps exact (drop tap re-recognition / `mLiveStroke` / + `PREF_MULTIPART_RERECOGNIZE_TAPS`). Validate on-device that re-recognition from synthesized + key-center geometry holds up. Medium. 3. **Generalize.** Route all multi-part composition through "composing region as source of - truth": derive fragment boundaries from text, support mid-word cursor + partial-delete - continuation, retire `mGestureFragmentBoundaries`. Larger; the payoff phase. + truth": derive fragment boundaries from text, and support **swiping onto a word the cursor + was moved into** (and after a partial delete) — with the Trigger model strictly enforced + (cursor move = context-only; taps exact; only a swipe composes). Retire + `mGestureFragmentBoundaries`. Larger; the payoff phase. ## Risks / validation From 122c322a140f694ced7fcfe851123afbcf824b32 Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Tue, 9 Jun 2026 09:14:36 -0700 Subject: [PATCH 22/26] =?UTF-8?q?docs:=20correct=20trigger=20model=20?= =?UTF-8?q?=E2=80=94=20preserve=20re-recognition=20(the=20core=20feature)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Earlier revision wrongly proposed making taps exact and removing tap re-recognition. Re-recognition (combining taps + swipes into one word, re-resolved against context) is the core Nintype behavior and must be preserved. Corrected trigger model: - A swipe always re-recognizes, pulling in the word at the cursor as context (typed / swiped / just-moved-into): "Doc" + swipe "ument" -> "Document". - A tap is exact UNLESS the current word already contains a swipe; then taps before/during/after contribute (the combine feature stays). Tap into a no-swipe word is literal: "Do|c" + tap "g" -> "Dogc". - Cursor move is context-only (inert until a swipe; bare taps stay exact). mLiveStroke is retired as STORED state only — the re-recognition behavior it powered is preserved, re-sourced from the current word's text on demand. Updated state inventory and Phase 2/3 accordingly. Co-Authored-By: Claude Opus 4.8 --- docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md | 76 +++++++++++++++----------- 1 file changed, 44 insertions(+), 32 deletions(-) diff --git a/docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md b/docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md index 32ab3ab10..f6e85376a 100644 --- a/docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md +++ b/docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md @@ -47,24 +47,28 @@ one when re-recognition is needed. ## Trigger model (what (re)composes, and what doesn't) -Knowing "the current word" from the editor is for **context**, not for re-typing it. The -only thing that ever re-composes a word is **a swipe**. Concretely: - -- **Taps are exact and bypass live-composition entirely.** A tapped character is inserted - literally, exactly as it is today — it never triggers re-recognition and never reshapes - the surrounding word. (This replaces the current "re-recognize taps within swiped words" - behavior, `PREF_MULTIPART_RERECOGNIZE_TAPS` — taps no longer fold into the stroke.) -- **Moving the cursor into a word is context-only.** Establishing the composing region over - an existing word (so the engine *knows* the word at the cursor) must **not** live-compose, - re-recognize, or alter it. Nothing changes on screen until the user acts. -- **Only a swipe (re)composes.** A swipe added at the cursor builds onto whatever word is - there — typed, swiped, or one the cursor was just moved into — by deriving that word's - key-center stroke from its text and merging the swipe onto it (the merged-trail path). - This is the single entry point to re-recognition. - -Net: the source-of-truth model makes any word *available* to build on, but the user stays -in control — a tap is always a tap, a cursor move is inert, and a word only re-composes -when they deliberately swipe onto it. Space remains the word-submission point. +**Re-recognition is the core feature (the Nintype behavior) and must be preserved:** +combining taps and swipes — before, during, or after each other — into one word that +re-resolves against the context before the cursor. The rules below govern only *when* it +fires, so a deliberate tap is never silently reshaped: + +- **A swipe always (re)composes, and always carries context.** A swipe at the cursor + re-recognizes, pulling in the word at the cursor as context — whether that word was typed, + swiped, or one the cursor was just moved into. *Example: move to the end of typed "Doc", + swipe "ument" → re-recognizes with the "Doc" context → "Document".* +- **A tap is exact UNLESS the current word already contains a swipe.** A tap into a word with + no swipe in it is literal — it never re-recognizes. *Example: move between "Do" and "c", + tap "g" → "Dogc", exactly.* But once the current word's composition includes a swipe, taps + before/during/after it **do** contribute and re-recognize — this is the combine-taps-and- + swipes feature, kept (today's `PREF_MULTIPART_RERECOGNIZE_TAPS`), just reimplemented on the + source-of-truth model. +- **Moving the cursor is context-only.** It makes the word at the cursor *known* (so a + following swipe can build on it) but never re-composes on its own. The next swipe + re-recognizes with that context; the next bare tap (no swipe in the word) stays exact. + +Net: **swipes drive re-recognition and always carry the surrounding-word context; taps stay +exact until a swipe is in play, then they contribute; cursor moves are inert until you act.** +Space remains the word-submission point. ## The foundation already exists @@ -100,12 +104,18 @@ through this existing machinery instead of around it. rebuilds its base from whatever text is actually there. ### State inventory -- **Retire:** `mLiveStroke` and the whole tap-re-recognition feature - (`PREF_MULTIPART_RERECOGNIZE_TAPS`) — taps become exact again; `mGestureFragmentBoundaries`; - the bespoke `mInputPointers`-survives-reset contract and its clean-up patches. -- **Keep:** `mExtendBatchInputBase` as the *mechanism* for feeding "prefix + new gesture" to - the recognizer (the re-timing logic stays), but the prefix is sourced from text-derived - geometry rather than a stored stroke. It is armed only on the swipe path. +- **Retire as STORED state — behavior preserved:** `mLiveStroke`. The accumulated raw stroke + is replaced by deriving the prefix/context from the *current word's text* on demand. The + re-recognition it powered (taps contributing to a swipe-involving word, and swipe-on-word) + stays — only its source changes from a drift-prone buffer to the editor text. +- **Retire:** `mGestureFragmentBoundaries`; the bespoke `mInputPointers`-survives-reset + contract and its clean-up patches. +- **Keep the feature:** the combine-taps-and-swipes re-recognition (today gated by + `PREF_MULTIPART_RERECOGNIZE_TAPS`) — re-expressed statelessly. Whether it stays a toggle or + becomes always-on core is a follow-up decision; the behavior does not go away. +- **Keep:** `mExtendBatchInputBase` as the *mechanism* for feeding "context prefix + new + input" to the recognizer (the re-timing logic stays), but the prefix is sourced from + text-derived key centers rather than a stored stroke. - **Reuse:** `getWordRangeAtCursor`, `setComposingRegion`, `setComposingWord`, `getCoordinatesForCurrentKeyboard`, `setCursorPositionWithinWord`. @@ -115,16 +125,18 @@ through this existing machinery instead of around it. realign `mInputPointers` to the truncated word (rebuild from its key centers) so a following swipe-extend no longer merges with the pre-pop stroke. Keeps the current architecture; closes the fragment case the fresh-word reset doesn't cover. Small. -2. **Prove the model (swipe-on-word).** Make a *swipe* that extends the word at the cursor - derive its merge base from that word's *text* via `getCoordinatesForCurrentKeyboard`, - instead of `mLiveStroke`. Make taps exact (drop tap re-recognition / `mLiveStroke` / - `PREF_MULTIPART_RERECOGNIZE_TAPS`). Validate on-device that re-recognition from synthesized - key-center geometry holds up. Medium. +2. **Prove the model (re-recognition from text).** Re-express the existing re-recognition so + its prefix/context comes from the *current word's text* via `getCoordinatesForCurrentKeyboard` + instead of the stored `mLiveStroke` — for both a swipe extending the word and a tap + contributing to a swipe-involving word. Behavior is unchanged; the stored stroke is gone. + Validate on-device that re-recognition from synthesized key-center geometry holds up + (vs. today's real-stroke accumulation). Medium. 3. **Generalize.** Route all multi-part composition through "composing region as source of truth": derive fragment boundaries from text, and support **swiping onto a word the cursor - was moved into** (and after a partial delete) — with the Trigger model strictly enforced - (cursor move = context-only; taps exact; only a swipe composes). Retire - `mGestureFragmentBoundaries`. Larger; the payoff phase. + was moved into** (and after a partial delete), pulling in its text as context. Enforce the + Trigger model exactly (swipe always re-recognizes with context; tap exact unless the word + already has a swipe; cursor move = context-only). Retire `mGestureFragmentBoundaries`. + Larger; the payoff phase. ## Risks / validation From 9025bb796bd18a5cb4ab55194d2b4e7ad8c67487 Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Tue, 9 Jun 2026 09:18:19 -0700 Subject: [PATCH 23/26] docs: confirm swipe-onto-any-existing-word (incl. committed) is in scope MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per review: swiping onto a word the cursor was moved into re-recognizes with that word's context even when it is already committed to the text box (re-entered via a cursor move) — the full 'editor as source of truth' payoff, scoped to Phase 3. Co-Authored-By: Claude Opus 4.8 --- docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md b/docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md index f6e85376a..30c6c7db4 100644 --- a/docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md +++ b/docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md @@ -54,8 +54,10 @@ fires, so a deliberate tap is never silently reshaped: - **A swipe always (re)composes, and always carries context.** A swipe at the cursor re-recognizes, pulling in the word at the cursor as context — whether that word was typed, - swiped, or one the cursor was just moved into. *Example: move to the end of typed "Doc", - swipe "ument" → re-recognizes with the "Doc" context → "Document".* + swiped, one the cursor was just moved into, **or one already committed to the text box** + (re-entered via a cursor move). *Example: move to the end of "Doc" — composing OR a + committed word in the field — swipe "ument" → re-recognizes with the "Doc" context → + "Document".* This swipe-onto-any-existing-word capability is the full payoff (Phase 3). - **A tap is exact UNLESS the current word already contains a swipe.** A tap into a word with no swipe in it is literal — it never re-recognizes. *Example: move between "Do" and "c", tap "g" → "Dogc", exactly.* But once the current word's composition includes a swipe, taps @@ -132,8 +134,10 @@ through this existing machinery instead of around it. Validate on-device that re-recognition from synthesized key-center geometry holds up (vs. today's real-stroke accumulation). Medium. 3. **Generalize.** Route all multi-part composition through "composing region as source of - truth": derive fragment boundaries from text, and support **swiping onto a word the cursor - was moved into** (and after a partial delete), pulling in its text as context. Enforce the + truth": derive fragment boundaries from text, and support **swiping onto any existing word + the cursor was moved into — including a word already committed to the text box** (and after + a partial delete), re-establishing the composing region over it and pulling in its text as + context. Enforce the Trigger model exactly (swipe always re-recognizes with context; tap exact unless the word already has a swipe; cursor move = context-only). Retire `mGestureFragmentBoundaries`. Larger; the payoff phase. From 6c96b7f9c2048da2e9438fea22aecbb070f6c447 Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Wed, 10 Jun 2026 06:32:14 -0700 Subject: [PATCH 24/26] docs: hybrid Phase 2 outcome + current cursor-move behavior - Mark Phase 1 & 2 landed/validated; reframe Phase 2 from 'retire mLiveStroke' to 'real stroke primary + text-derived fallback' (the feel-test outcome). - Update state inventory: mLiveStroke is KEPT as primary, backed by the text-derived fallback. - Add 'Current behavior (pre-Phase 3)' section documenting that a cursor move + swipe starts a fresh autospaced word, while a tap edits the existing word, with the code mechanism and how Phase 3 would change the swipe half. Co-Authored-By: Claude Opus 4.8 --- docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md | 63 ++++++++++++++++++++------ 1 file changed, 49 insertions(+), 14 deletions(-) diff --git a/docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md b/docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md index 30c6c7db4..036adadc0 100644 --- a/docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md +++ b/docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md @@ -72,6 +72,29 @@ Net: **swipes drive re-recognition and always carry the surrounding-word context exact until a swipe is in play, then they contribute; cursor moves are inert until you act.** Space remains the word-submission point. +### Current behavior (pre-Phase 3) — how cursor moves actually behave today + +The Trigger model above is the *target*. Until Phase 3 lands, moving the cursor onto an +existing word behaves differently for swipe vs. tap, and this is worth knowing: + +- **Move the cursor into another word, or midway through a word, then SWIPE** → the keyboard + commits/clears the re-entered word, inserts an autospace, and **starts a brand-new word**. + It does *not* re-recognize the word the cursor is in. *Mechanism:* a cursor landing in the + front/middle of the composing word makes `isCursorFrontOrMiddleOfComposingWord()` true, so + `onStartBatchInput` takes the reset branch (`InputLogic.java` ~line 787); with no composing + word left to extend and a letter before the cursor, the autospace (`SpaceState.PHANTOM`) + branch fires (~line 846) and the gesture recognizes fresh. +- **Move the cursor into/onto a word, then TAP** → you **edit that existing word** with a + literal character at the cursor. *Mechanism:* `tryLiveConvergeTap` returns early on the same + front/middle guard (and on `!mCombiningWordHasGestureFragment` for a word with no swipe in + it), so the tap falls through to normal literal insertion. + +This is *not a bug* — it's a usable split: swipe = "start fresh here," tap = "edit what's +here." **Phase 3 is precisely what changes the swipe half**: a swipe onto a re-entered word +would instead re-establish the composing region over that word and re-recognize *with its +text as context* (the "move to end of `Doc`, swipe `ument` → `Document`" payoff). Whether to +take that step — or keep today's simpler split — is an open product decision (see Phases). + ## The foundation already exists HeliBoard already reconstructs a composing word from editor text when the cursor lands in a @@ -106,10 +129,13 @@ through this existing machinery instead of around it. rebuilds its base from whatever text is actually there. ### State inventory -- **Retire as STORED state — behavior preserved:** `mLiveStroke`. The accumulated raw stroke - is replaced by deriving the prefix/context from the *current word's text* on demand. The - re-recognition it powered (taps contributing to a swipe-involving word, and swipe-on-word) - stays — only its source changes from a drift-prone buffer to the editor text. +- **KEEP as PRIMARY (revised after Phase 2 feel testing):** `mLiveStroke`. Originally slated + for retirement, but the real captured stroke re-recognizes better than synthesized + key-centers, so it stays as the high-fidelity primary prefix. It is **backed by** a + text-derived fallback (`buildStrokeFromWordText`) for when it has been invalidated, so the + drift it used to cause is now a graceful degradation rather than a bug: an invalid stroke + just means "fall back to text," not "produce garbage." The re-recognition it powers (taps + contributing to a swipe-involving word) is unchanged. - **Retire:** `mGestureFragmentBoundaries`; the bespoke `mInputPointers`-survives-reset contract and its clean-up patches. - **Keep the feature:** the combine-taps-and-swipes re-recognition (today gated by @@ -123,16 +149,25 @@ through this existing machinery instead of around it. ## Phases (each independently shippable + on-device validated) -1. **Stop the bleeding in fragment mode.** On a fragment-pop (`tryFragmentBackspace`), - realign `mInputPointers` to the truncated word (rebuild from its key centers) so a - following swipe-extend no longer merges with the pre-pop stroke. Keeps the current - architecture; closes the fragment case the fresh-word reset doesn't cover. Small. -2. **Prove the model (re-recognition from text).** Re-express the existing re-recognition so - its prefix/context comes from the *current word's text* via `getCoordinatesForCurrentKeyboard` - instead of the stored `mLiveStroke` — for both a swipe extending the word and a tap - contributing to a swipe-involving word. Behavior is unchanged; the stored stroke is gone. - Validate on-device that re-recognition from synthesized key-center geometry holds up - (vs. today's real-stroke accumulation). Medium. +1. **Stop the bleeding in fragment mode.** ✅ **LANDED & on-device validated.** On a + fragment-pop (`tryFragmentBackspace`), realign `mInputPointers` to the truncated word + (rebuild from its key centers via `seedInputPointersFromKeyCenters`) so a following + swipe-extend no longer merges with the pre-pop stroke. Keeps the current architecture; + closes the fragment case the fresh-word reset doesn't cover. Small. (Lowercases the word + before key lookup and skips `NOT_A_COORDINATE` keys — see the implementation notes.) +2. **Add a text-derived FALLBACK for re-recognition.** ✅ **LANDED & on-device validated.** + *(Plan changed here based on feel testing — see note below.)* The original intent was to + *replace* `mLiveStroke` with text-derived geometry. On-device that felt subpar: the + recognizer reads the user's **real captured swipe curves** noticeably better than + synthesized key-centers. So instead the live-converge tap path now uses a **hybrid prefix**: + the real `mLiveStroke` is the high-fidelity PRIMARY, and `buildStrokeFromWordText` + (key-centers from the current word's text, the Phase 1 primitive) is the FALLBACK, used only + when the real stroke was invalidated — after a backspace, a cursor move into a word, a + partial delete, a commit, or a desync. Those are exactly the "we no longer know the real + stroke" cases; previously they dropped to literal typing, now they still re-recognize (just + from clean key-centers). Net: the original good feel during continuous swipe+tap building is + preserved, and the edit/re-entry cases degrade gracefully instead of failing. `mLiveStroke` + is **kept** (not retired) as the primary source. Medium. 3. **Generalize.** Route all multi-part composition through "composing region as source of truth": derive fragment boundaries from text, and support **swiping onto any existing word the cursor was moved into — including a word already committed to the text box** (and after From fb4cd477d83156f7d76d722e4c1a83c38bdcf0ba Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Wed, 10 Jun 2026 06:34:34 -0700 Subject: [PATCH 25/26] docs: descope Phase 3 swipe-onto-word re-recognition (keep current split) Product decision: swipe at a moved cursor = start fresh word, tap = edit the existing word is the intended behavior, not a stepping stone. Phase 3's swipe-onto-any-existing-word re-recognition is dropped. The fragment-boundary derivation cleanup is noted as an independent optional leftover. Co-Authored-By: Claude Opus 4.8 --- docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md | 28 +++++++++++++++----------- 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md b/docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md index 036adadc0..4b730a69a 100644 --- a/docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md +++ b/docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md @@ -90,10 +90,12 @@ existing word behaves differently for swipe vs. tap, and this is worth knowing: it), so the tap falls through to normal literal insertion. This is *not a bug* — it's a usable split: swipe = "start fresh here," tap = "edit what's -here." **Phase 3 is precisely what changes the swipe half**: a swipe onto a re-entered word -would instead re-establish the composing region over that word and re-recognize *with its -text as context* (the "move to end of `Doc`, swipe `ument` → `Document`" payoff). Whether to -take that step — or keep today's simpler split — is an open product decision (see Phases). +here." Phase 3 would have changed the swipe half (re-recognize the re-entered word with its +text as context — the "move to end of `Doc`, swipe `ument` → `Document`" payoff). + +> **Product decision (2026-06): keep this split.** The swipe = start-fresh / tap = edit +> behavior is the intended design, not a stepping stone. The swipe-onto-an-existing-word +> re-recognition described in the Trigger model is **descoped** — see Phase 3 below. ## The foundation already exists @@ -168,14 +170,16 @@ through this existing machinery instead of around it. from clean key-centers). Net: the original good feel during continuous swipe+tap building is preserved, and the edit/re-entry cases degrade gracefully instead of failing. `mLiveStroke` is **kept** (not retired) as the primary source. Medium. -3. **Generalize.** Route all multi-part composition through "composing region as source of - truth": derive fragment boundaries from text, and support **swiping onto any existing word - the cursor was moved into — including a word already committed to the text box** (and after - a partial delete), re-establishing the composing region over it and pulling in its text as - context. Enforce the - Trigger model exactly (swipe always re-recognizes with context; tap exact unless the word - already has a swipe; cursor move = context-only). Retire `mGestureFragmentBoundaries`. - Larger; the payoff phase. +3. **Generalize.** 🚫 **Descoped (2026-06) — swipe-onto-word re-recognition will NOT be + pursued.** The headline of this phase was supporting *swiping onto any existing word the + cursor was moved into* (incl. a committed word) and re-recognizing it with context. Per the + product decision above, today's split — **swipe at a moved cursor starts a fresh word, tap + edits the existing word** — is the intended behavior, so this is dropped. The Trigger + model's "a swipe always re-recognizes the word at the cursor" rule therefore does *not* + apply to a re-entered word; it governs only a word actively being built at the cursor end. + *Optional leftover:* deriving fragment boundaries from text to retire + `mGestureFragmentBoundaries` is an independent internal cleanup that could still be done on + its own merits, but it is no longer blocked on (or part of) a behavioral phase. ## Risks / validation From 4610c0aad94db966469497a2a61d101149cd3a0f Mon Sep 17 00:00:00 2001 From: SHAWNERZZ Date: Wed, 10 Jun 2026 21:31:33 -0700 Subject: [PATCH 26/26] docs: reconcile to what actually shipped (Phase 2 reverted, consumption-point fix) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The doc described the Phase 2 mLiveStroke hybrid/fallback as landed. In reality that was attempted and REVERTED (the fallback was effectively dead code, and the real captured stroke beats synthesized geometry on feel). The bug was instead fixed surgically at the swipe-extend consumption point (mComposingStrokeStale + realignComposerStrokeToText). - Status header: 'partially shipped — record of what shipped and why the plan changed'. - Phases/outcome: Phase 1 shipped; Phase 2 attempted-then-reverted (with the lesson); Phase 3 descoped; new 'what actually fixed the bug' section; Nintype whole-word backspace noted as planned/next. - State inventory: mLiveStroke kept unchanged; mInputPointers drift fixed not retired. - Marked the 'Target design' bullets as the original aspiration, only partially realized. Co-Authored-By: Claude Opus 4.8 --- docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md | 126 +++++++++++++++---------- 1 file changed, 74 insertions(+), 52 deletions(-) diff --git a/docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md b/docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md index 4b730a69a..697f99351 100644 --- a/docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md +++ b/docs/COMPOSING_WORD_SOURCE_OF_TRUTH.md @@ -1,8 +1,10 @@ # Composing word: editor text as the single source of truth (design) -> Status: **design / not yet implemented**. Tracking: see the "Composing word source of -> truth" epic on the fork. This note is the plan of record for a multi-phase refactor of -> two-thumb / multi-part composition. Update it as phases land. +> Status: **partially shipped — see "Phases / outcome" below.** The drift-prone-stroke bug +> class this note set out to kill is fixed and on-device validated; the broader "retire all +> parallel state" refactor was *not* pursued in full (the targeted fix proved sufficient and +> the real captured stroke beat synthesized geometry on feel). This note is now the record of +> what shipped and why the plan changed, not a forward plan. > > Grounding: claims below were verified by read-only scouting of `InputLogic`, > `WordComposer`, `RichInputConnection`. File/symbol refs are a snapshot — prefer the named @@ -115,7 +117,11 @@ design needs. Today it is used for **tap recorrection only**; it is never wired gesture / live-converge path. The refactor is largely about routing two-thumb composition through this existing machinery instead of around it. -## Target design +## Target design (original aspiration — only partially realized) + +> The bullets below are the *original* "remove all parallel state" vision. In practice only the +> stroke-from-text primitive was adopted, and only at the swipe-extend consumption point — see +> "Phases / outcome" above for what actually shipped. Kept here for context. - **The word being built = the composing region in the editor.** It can be (re)established from any cursor position via `getWordRangeAtCursor` + `setComposingRegion`, which the @@ -130,56 +136,72 @@ through this existing machinery instead of around it. changes the text (a cursor move is otherwise inert — context-only); the *next swipe* rebuilds its base from whatever text is actually there. -### State inventory -- **KEEP as PRIMARY (revised after Phase 2 feel testing):** `mLiveStroke`. Originally slated - for retirement, but the real captured stroke re-recognizes better than synthesized - key-centers, so it stays as the high-fidelity primary prefix. It is **backed by** a - text-derived fallback (`buildStrokeFromWordText`) for when it has been invalidated, so the - drift it used to cause is now a graceful degradation rather than a bug: an invalid stroke - just means "fall back to text," not "produce garbage." The re-recognition it powers (taps - contributing to a swipe-involving word) is unchanged. -- **Retire:** `mGestureFragmentBoundaries`; the bespoke `mInputPointers`-survives-reset - contract and its clean-up patches. -- **Keep the feature:** the combine-taps-and-swipes re-recognition (today gated by - `PREF_MULTIPART_RERECOGNIZE_TAPS`) — re-expressed statelessly. Whether it stays a toggle or - becomes always-on core is a follow-up decision; the behavior does not go away. -- **Keep:** `mExtendBatchInputBase` as the *mechanism* for feeding "context prefix + new - input" to the recognizer (the re-timing logic stays), but the prefix is sourced from - text-derived key centers rather than a stored stroke. -- **Reuse:** `getWordRangeAtCursor`, `setComposingRegion`, `setComposingWord`, - `getCoordinatesForCurrentKeyboard`, `setCursorPositionWithinWord`. - -## Phases (each independently shippable + on-device validated) - -1. **Stop the bleeding in fragment mode.** ✅ **LANDED & on-device validated.** On a +### State inventory — what actually happened +- **`mLiveStroke`: KEPT, UNCHANGED.** The plan was to retire it (derive the tap-re-recognition + prefix from text). An attempt to do so — and a follow-up "real stroke primary, text-derived + fallback" hybrid (`buildStrokeFromWordText`, `mComposingWordHasSwipeContent`) — was **reverted**: + the fallback turned out to be effectively dead code (the guards that admit a re-recognizing tap + are cleared on the same edits that empty the stroke, so it never fired), and the real captured + stroke re-recognizes better than synthesized key-centers anyway. The live-converge tap path + still uses `mLiveStroke` exactly as before. +- **`mInputPointers` drift: FIXED, not retired.** The "buffer survives reset / doesn't shrink on + delete" contract stays, but the drift it caused is now corrected **at the point of consumption** + rather than by removing the buffer (see Phases / outcome). `mGestureFragmentBoundaries` is also + still in use (not retired). +- **`mExtendBatchInputBase`** remains the mechanism for feeding "context prefix + new input" to + the recognizer. The prefix is the *real* stroke when it still matches the text, and is rebuilt + from text-derived key centers only when an edit made it stale. +- **Reused as planned:** `getCoordinatesForCurrentKeyboard` (text → key-center geometry) is the + one primitive from this plan that did get wired into the gesture path, via + `seedInputPointersFromKeyCenters` / `realignComposerStrokeToText`. + +## Phases / outcome (what was planned vs what shipped) + +1. **Stop the bleeding in fragment mode.** ✅ **SHIPPED & on-device validated.** On a fragment-pop (`tryFragmentBackspace`), realign `mInputPointers` to the truncated word (rebuild from its key centers via `seedInputPointersFromKeyCenters`) so a following - swipe-extend no longer merges with the pre-pop stroke. Keeps the current architecture; - closes the fragment case the fresh-word reset doesn't cover. Small. (Lowercases the word - before key lookup and skips `NOT_A_COORDINATE` keys — see the implementation notes.) -2. **Add a text-derived FALLBACK for re-recognition.** ✅ **LANDED & on-device validated.** - *(Plan changed here based on feel testing — see note below.)* The original intent was to - *replace* `mLiveStroke` with text-derived geometry. On-device that felt subpar: the - recognizer reads the user's **real captured swipe curves** noticeably better than - synthesized key-centers. So instead the live-converge tap path now uses a **hybrid prefix**: - the real `mLiveStroke` is the high-fidelity PRIMARY, and `buildStrokeFromWordText` - (key-centers from the current word's text, the Phase 1 primitive) is the FALLBACK, used only - when the real stroke was invalidated — after a backspace, a cursor move into a word, a - partial delete, a commit, or a desync. Those are exactly the "we no longer know the real - stroke" cases; previously they dropped to literal typing, now they still re-recognize (just - from clean key-centers). Net: the original good feel during continuous swipe+tap building is - preserved, and the edit/re-entry cases degrade gracefully instead of failing. `mLiveStroke` - is **kept** (not retired) as the primary source. Medium. -3. **Generalize.** 🚫 **Descoped (2026-06) — swipe-onto-word re-recognition will NOT be - pursued.** The headline of this phase was supporting *swiping onto any existing word the - cursor was moved into* (incl. a committed word) and re-recognizing it with context. Per the - product decision above, today's split — **swipe at a moved cursor starts a fresh word, tap - edits the existing word** — is the intended behavior, so this is dropped. The Trigger - model's "a swipe always re-recognizes the word at the cursor" rule therefore does *not* - apply to a re-entered word; it governs only a word actively being built at the cursor end. - *Optional leftover:* deriving fragment boundaries from text to retire - `mGestureFragmentBoundaries` is an independent internal cleanup that could still be done on - its own merits, but it is no longer blocked on (or part of) a behavioral phase. + swipe-extend no longer merges with the pre-pop stroke. Lowercases the word before the + exact key lookup and skips `NOT_A_COORDINATE` keys. +2. **Re-express re-recognition from text (retire `mLiveStroke`).** ❌ **ATTEMPTED, then + REVERTED.** Two variants were tried: (a) replace `mLiveStroke` with text-derived geometry — + felt subpar, the real captured curves re-recognize better; (b) a "real primary + text-derived + fallback" hybrid (`buildStrokeFromWordText`, `mComposingWordHasSwipeContent`) — the fallback + was effectively unreachable (the guards that admit a re-recognizing tap are cleared by the + same edits that empty the stroke), so it changed nothing. Both were rolled back; `mLiveStroke` + stays as-is. **Lesson:** the parallel stroke state isn't worth removing for the tap path; it's + only a problem for the *swipe-extend base*, which is fixed surgically below. +3. **Generalize / swipe-onto-committed-word.** 🚫 **Descoped (2026-06).** Today's split — + **swipe at a moved cursor starts a fresh word, tap edits the existing word** — is the intended + behavior (product decision), so the Trigger model's "a swipe always re-recognizes the word at + the cursor" does *not* apply to a re-entered committed word; it governs only a word actively + being built at the cursor end. + +### What actually fixed the bug (the shipped general fix) + +Instead of removing the parallel stroke buffer, the drift is corrected **at the single point it +is consumed** — when a swipe arms its merged-trail extend base in `onStartBatchInput`: + +- `mComposingStrokeStale` is set on **any** backspace and cleared when a gesture rebuilds the + buffer (or the word ends). It means "the composing text was edited since the stroke last + matched it." +- At the extend-arm site, if stale, the base is rebuilt from the composing word's text via + `realignComposerStrokeToText` (→ `getCoordinatesForCurrentKeyboard` → + `seedInputPointersFromKeyCenters`); if not stale, the **real captured stroke** is used. +- This covers every edit path at once (single-char, fragment-pop, selection / multi-char delete, + cursor re-compose) without per-path patches, and leaves continuous swipe+swipe and tap+swipe + using their real stroke. `realignComposerStrokeToText` no-ops if the layout can't resolve any + key, so it never disarms the extend. **Shipped & on-device validated** (the recurring + "Thing→delete→Whining" failure is gone). + +### Next: Nintype-style whole-word backspace (planned, not yet built) + +A follow-up, independent of the above: in **"Whole word" backspace mode**, pop the last swipe +fragment while composing, then delete a **whole previous word** (any word, swiped or typed) once +past the composing word — gated by the existing `PREF_COMBINING_BACKSPACE_DELETES_COMPOSING_TEXT` +toggle (to be relabelled to describe this), with key-repeat deleting whole words. Requires +relaxing `tryFragmentBackspace`'s whole-word-mode bail and adding a word-boundary delete for +committed text (`getTextBeforeCursor` + `SpacingAndPunctuations`), gated to two-thumb mode so +plain typists are unaffected. ## Risks / validation