diff --git a/CLAUDE.md b/CLAUDE.md index 5c6a90b3..6e182891 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b9964** +Current llama.cpp pinned version: **b9972** ## Upgrading CUDA Version @@ -421,7 +421,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b9964 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b9972 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -461,7 +461,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b9964`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b9972`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1237,7 +1237,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b9964`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b9972`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index ac373be7..eb03ba7c 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b9964](https://img.shields.io/badge/llama.cpp-%23b9964-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b9964) +[![llama.cpp b9972](https://img.shields.io/badge/llama.cpp-%23b9972-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b9972) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/android-llmservice/README.md b/android-llmservice/README.md index dba224f4..d0338585 100644 --- a/android-llmservice/README.md +++ b/android-llmservice/README.md @@ -13,7 +13,9 @@ what it says: 1. **Pick a GGUF model** from the device's file system (Storage Access Framework — no storage permission, Google-Play compliant). 2. **Chat** with it, fully **on-device**, tokens streaming into a Jetpack Compose UI. -3. **Switch language** via a flag picker, and **Save/Load** the conversation locally. +3. Optionally add a **vision projector (mmproj)** and attach an **image** (📎) to a + message — for multimodal GGUFs such as Gemma 3 4B (vision) or SmolVLM2. +4. **Switch language** via a flag picker, and **Save/Load** the conversation locally. - **App label:** `LLM Service` - **applicationId / package:** `net.ladenthin.android.llmservice` diff --git a/android-llmservice/TODO.md b/android-llmservice/TODO.md index 8ab15a44..f5060935 100644 --- a/android-llmservice/TODO.md +++ b/android-llmservice/TODO.md @@ -83,7 +83,7 @@ Everything else in the brief is feasible with standard AndroidX + the existing b | Markdown rendering (code, lists, tables, headings) | Readable answers | M | ⬜ | add a Compose Markdown renderer (e.g. compose-markdown) | | Copy / regenerate / stop / clear-chat actions | Standard chat UX | M | ✅ | **stop** cancels the streaming coroutine (façade closes the source); **copy**/**regenerate** act on the last reply; **long-press any bubble** copies that message; **clear** in the top bar with a confirm | | Prompt shortcut chips (Summarize / Explain code / Translate / …) | Fast starts | S | ✅ | localized `SuggestionChip`s above the input (fill the draft); 3 chips × 13 languages | -| Attachment button → image input (vision) | Multimodal chat | L | ⬜ | `ContentPart.imageFile(...)` + a **vision GGUF + mmproj**; SAF image picker | +| Attachment button → image input (vision) | Multimodal chat | L | ✅ | SAF pickers for both a vision **mmproj** GGUF (model-picker screen) and a query **image** (📎 in the chat input, shown only once a vision projector is loaded); wired through `ContentPart.imageBytes(...)` + `ChatMessage`. Attached images are session-transient (not saved to `session.json`) | | Audio input (speech) | Multimodal chat | L | ⬜ | `ContentPart.audioFile(...)` + an audio-capable model + mmproj | | Explicit states: loading / generating / stopped / unavailable / "too large for RAM" | Transparency | S | 🔧 | basic states exist; add the rest + messages | @@ -154,7 +154,7 @@ See decision #1. All rows below assume a new in-app Ktor/NanoHTTPD HTTP layer wr | Item | Reality today | Effort to adopt | |---|---|---| | GPU acceleration | **OpenCL/Adreno only** via the separate `net.ladenthin:llama-android-opencl` AAR (Qualcomm Snapdragon/Adreno; device must supply an OpenCL ICD). **No Vulkan** Android artifact exists. | M — swap/offer the OpenCL AAR flavor + runtime detection; non-Adreno devices stay CPU | -| Vision / audio input | Supported by the binding (`ContentPart.imageFile`/`audioFile`), needs a multimodal GGUF + matching mmproj | L (see §3) | +| Audio input | Supported by the binding (`ContentPart.audioFile`), needs an audio-capable GGUF + matching mmproj | L (see §3) | | 16 KB page size / dlopen-clean `.so` | Already guaranteed by the AAR (CI-enforced) | — | ## 8. Branding & store @@ -175,7 +175,7 @@ See decision #1. All rows below assume a new in-app Ktor/NanoHTTPD HTTP layer wr icon, helper tooltips. - **Phase 2 — the model manager:** managed import library → `GgufInspector` model cards → RAM/compat badges → filters → multi-model switch/delete. Then decide on the **download manager** (`INTERNET`). -- **Phase 3 — multimodal:** image (then audio) input with a vision/audio GGUF + mmproj. +- **Phase 3 — multimodal:** image input ✅ shipped; audio input with an audio-capable GGUF + mmproj remains. - **Phase 4 — hardening/perf:** encrypted conversations, biometric lock, battery-saver + thermal throttling, optional Adreno/OpenCL GPU flavor. - **Phase 5 — the local API server (`XL`, foundational):** in-app Ktor/NanoHTTPD server wrapping the diff --git a/android-llmservice/app/src/main/kotlin/net/ladenthin/android/llmservice/ChatViewModel.kt b/android-llmservice/app/src/main/kotlin/net/ladenthin/android/llmservice/ChatViewModel.kt index d79a9afd..c3c82007 100644 --- a/android-llmservice/app/src/main/kotlin/net/ladenthin/android/llmservice/ChatViewModel.kt +++ b/android-llmservice/app/src/main/kotlin/net/ladenthin/android/llmservice/ChatViewModel.kt @@ -25,7 +25,8 @@ import net.ladenthin.llama.LlamaModel import net.ladenthin.llama.kotlin.generateChatFlow import net.ladenthin.llama.parameters.InferenceParameters import net.ladenthin.llama.parameters.ModelParameters -import net.ladenthin.llama.value.Pair +import net.ladenthin.llama.value.ChatMessage +import net.ladenthin.llama.value.ContentPart /** * File name of the SAF-picked model copied into `filesDir`. Transient working data — wiped on cold @@ -33,6 +34,12 @@ import net.ladenthin.llama.value.Pair */ const val MODEL_COPY_NAME: String = "current-model.gguf" +/** + * File name of the SAF-picked vision projector (mmproj) copied into `filesDir`. Same transient + * lifecycle as [MODEL_COPY_NAME] — wiped on cold start and on [ChatViewModel.unloadModel]. + */ +const val MMPROJ_COPY_NAME: String = "current-mmproj.gguf" + /** * Holds the loaded [LlamaModel] and drives streaming chat over the llama-kotlin * [generateChatFlow] facade. All native work runs on [Dispatchers.IO]; the UI observes @@ -44,8 +51,20 @@ const val MODEL_COPY_NAME: String = "current-model.gguf" */ class ChatViewModel(application: Application) : AndroidViewModel(application) { - /** One chat turn. [text] grows token-by-token while an assistant reply streams. */ - data class Message(val role: String, val text: String) + /** + * One chat turn. [text] grows token-by-token while an assistant reply streams. A user turn may + * carry an image ([imageBytes] + [imageMimeType]) attached via the 🖼️ button; both are null for + * every other turn. Images are session-transient — [SessionStore] persists text only. + */ + data class Message( + val role: String, + val text: String, + val imageBytes: ByteArray? = null, + val imageMimeType: String? = null, + ) + + /** An image picked via the attach button, staged in memory until [send] consumes it. */ + data class PendingImage(val bytes: ByteArray, val mimeType: String, val displayName: String) /** Lifecycle of the model load. */ enum class ModelState { NONE, LOADING, READY } @@ -111,6 +130,10 @@ class ChatViewModel(application: Application) : AndroidViewModel(application) { val notice: Notice? = null, val settings: GenerationSettings = GenerationSettings.DEFAULT, val modelConfig: ModelConfig = ModelConfig.DEFAULT, + /** Display name of the selected vision projector (mmproj), or null when text-only. */ + val visionModelName: String? = null, + /** An image staged via the attach button, waiting for the next [send]. */ + val pendingImage: PendingImage? = null, ) private val _uiState = MutableStateFlow(UiState()) @@ -126,6 +149,7 @@ class ChatViewModel(application: Application) : AndroidViewModel(application) { private var model: LlamaModel? = null private var modelPath: String? = null + private var mmprojPath: String? = null private var generation: Job? = null /** @@ -189,7 +213,7 @@ class ChatViewModel(application: Application) : AndroidViewModel(application) { _uiState.update { it.copy(modelState = ModelState.LOADING, error = null) } viewModelScope.launch { try { - val path = withContext(Dispatchers.IO) { copyToInternal(uri) } + val path = withContext(Dispatchers.IO) { copyUriToInternal(uri, MODEL_COPY_NAME) } openModel(path, displayName, template = null) } catch (t: Throwable) { _uiState.update { @@ -199,6 +223,63 @@ class ChatViewModel(application: Application) : AndroidViewModel(application) { } } + /** + * Copies a SAF-picked vision projector (mmproj) GGUF into app-internal storage and records its + * path. Takes effect the next time a model is (re)loaded — pick it before or after the main + * model, in either order. No-op error path mirrors [loadModelFromUri]. + */ + fun loadMmprojFromUri(uri: Uri, displayName: String) { + viewModelScope.launch { + try { + val path = withContext(Dispatchers.IO) { copyUriToInternal(uri, MMPROJ_COPY_NAME) } + mmprojPath = path + _uiState.update { it.copy(visionModelName = displayName, error = null) } + log("Vision model set: $displayName") + } catch (t: Throwable) { + _uiState.update { it.copy(error = ErrorInfo(ErrorType.LOAD, t.message ?: "I/O error")) } + } + } + } + + /** Clears the selected vision projector; the next model load is text-only again. */ + fun clearMmproj() { + mmprojPath = null + _uiState.update { it.copy(visionModelName = null) } + log("Vision model cleared") + viewModelScope.launch(Dispatchers.IO) { + File(getApplication().filesDir, MMPROJ_COPY_NAME).delete() + } + } + + /** + * Stages a SAF-picked image as the pending attachment for the next [send]. Read fully into + * memory rather than copied to [android.content.Context.getFilesDir]: unlike the model/mmproj + * GGUFs (which llama.cpp must mmap by real path), + * [net.ladenthin.llama.value.ContentPart.imageBytes] consumes raw bytes directly. + */ + fun attachImage(uri: Uri, displayName: String) { + viewModelScope.launch { + try { + val resolver = getApplication().contentResolver + val (mimeType, bytes) = withContext(Dispatchers.IO) { + val type = resolver.getType(uri) ?: "image/jpeg" + val data = resolver.openInputStream(uri)?.use { it.readBytes() } + ?: throw java.io.IOException("content resolver returned no stream for $uri") + type to data + } + _uiState.update { it.copy(pendingImage = PendingImage(bytes, mimeType, displayName), error = null) } + log("Image attached: $displayName") + } catch (t: Throwable) { + _uiState.update { it.copy(error = ErrorInfo(ErrorType.LOAD, t.message ?: "I/O error")) } + } + } + } + + /** Removes the pending image attachment before it's sent. */ + fun clearPendingImage() { + _uiState.update { it.copy(pendingImage = null) } + } + /** * Loads a model already present at an absolute on-device path (no copy). Used by the * instrumented test against the adb-pushed model and by session restore. @@ -212,16 +293,21 @@ class ChatViewModel(application: Application) : AndroidViewModel(application) { private suspend fun openModel(path: String, displayName: String, template: String?) { val config = _uiState.value.modelConfig + val mmproj = mmprojPath log("Loading model: $displayName") try { val loaded = withContext(Dispatchers.IO) { - LlamaModel( - ModelParameters() - .setModel(path) - .setCtxSize(config.contextSize) - .setThreads(config.threads) - .setGpuLayers(0), // CPU-only: portable across every device - ) + val parameters = ModelParameters() + .setModel(path) + .setCtxSize(config.contextSize) + .setThreads(config.threads) + .setGpuLayers(0) // CPU-only: portable across every device + if (mmproj != null) { + // Mirrors the CPU-only + mmproj config validated by MultimodalIntegrationTest: + // no GPU device selection, no mmproj offload attempt. + parameters.setMmproj(mmproj).setDevices("none").setMmprojOffload(false) + } + LlamaModel(parameters) } withContext(Dispatchers.IO) { model?.close() } model = loaded @@ -230,7 +316,10 @@ class ChatViewModel(application: Application) : AndroidViewModel(application) { _uiState.update { it.copy(modelState = ModelState.READY, modelName = displayName, error = null) } - log("Model ready: $displayName (ctx=${config.contextSize}, threads=${config.threads}, CPU)") + log( + "Model ready: $displayName (ctx=${config.contextSize}, threads=${config.threads}, " + + "CPU${if (mmproj != null) ", vision" else ""})", + ) } catch (t: Throwable) { log("Load failed: ${t.message ?: "load failed"}") _uiState.update { @@ -248,10 +337,18 @@ class ChatViewModel(application: Application) : AndroidViewModel(application) { fun send(userText: String, systemPrompt: String) { val text = userText.trim() val active = model - if (text.isEmpty() || active == null || _uiState.value.generating) { + val attachment = _uiState.value.pendingImage + if ((text.isEmpty() && attachment == null) || active == null || _uiState.value.generating) { return } - startGeneration(active, _uiState.value.messages + Message("user", text), systemPrompt) + val userMessage = Message( + role = "user", + text = text, + imageBytes = attachment?.bytes, + imageMimeType = attachment?.mimeType, + ) + _uiState.update { it.copy(pendingImage = null) } + startGeneration(active, _uiState.value.messages + userMessage, systemPrompt) } /** @@ -284,11 +381,15 @@ class ChatViewModel(application: Application) : AndroidViewModel(application) { val s = _uiState.value.settings generation = viewModelScope.launch { - val pairs = history.map { Pair(it.role, it.text) } + val chatMessages = mutableListOf() + if (systemPrompt.isNotEmpty()) { + chatMessages.add(ChatMessage("system", systemPrompt)) + } + history.forEach { chatMessages.add(it.toChatMessage()) } // Apply the sampling knobs. repeatPenalty (> 1) + repeatLastN are the ones that break // the degenerate repetition loop small on-device models fall into with a bare decode. var params = InferenceParameters("") - .withMessages(systemPrompt, pairs) + .withMessages(chatMessages) .withNPredict(s.maxTokens) .withTemperature(s.temperature) .withTopK(s.topK) @@ -357,15 +458,25 @@ class ChatViewModel(application: Application) : AndroidViewModel(application) { model = null modelPath = null chatTemplate = null + mmprojPath = null _uiState.update { - it.copy(modelState = ModelState.NONE, modelName = null, generating = false, error = null) + it.copy( + modelState = ModelState.NONE, + modelName = null, + visionModelName = null, + generating = false, + error = null, + pendingImage = null, + ) } log("Model unloaded") viewModelScope.launch(Dispatchers.IO) { toClose?.close() - // Drop the copied working model (frees storage; nothing lingers). A model loaded by an - // absolute path (test / session restore) has no copy here, so this is then a no-op. + // Drop the copied working model + mmproj (frees storage; nothing lingers). A model + // loaded by an absolute path (test / session restore) has no copy here, so this is then + // a no-op; likewise when no vision projector was ever selected. File(getApplication().filesDir, MODEL_COPY_NAME).delete() + File(getApplication().filesDir, MMPROJ_COPY_NAME).delete() } } @@ -407,6 +518,21 @@ class ChatViewModel(application: Application) : AndroidViewModel(application) { _uiState.update { it.copy(notice = null) } } + /** Converts one turn to the wire type: multimodal (text + image parts) when an image is attached. */ + private fun Message.toChatMessage(): ChatMessage { + val bytes = imageBytes + val mime = imageMimeType + if (bytes != null && mime != null) { + val parts = mutableListOf() + if (text.isNotBlank()) { + parts.add(ContentPart.text(text)) + } + parts.add(ContentPart.imageBytes(bytes, mime)) + return ChatMessage(role, parts) + } + return ChatMessage(role, text) + } + private fun UiState.replaceLastAssistant(newText: String): UiState { if (messages.isEmpty()) return this val updated = messages.toMutableList() @@ -414,9 +540,10 @@ class ChatViewModel(application: Application) : AndroidViewModel(application) { return copy(messages = updated) } - private fun copyToInternal(uri: Uri): String { + /** Copies a `content://` URI into `filesDir/targetName` (llama.cpp mmaps a real path, not a URI). */ + private fun copyUriToInternal(uri: Uri, targetName: String): String { val resolver = getApplication().contentResolver - val target = File(getApplication().filesDir, MODEL_COPY_NAME) + val target = File(getApplication().filesDir, targetName) resolver.openInputStream(uri).use { input -> requireNotNull(input) { "content resolver returned no stream for $uri" } target.outputStream().use { output -> input.copyTo(output, bufferSize = 1 shl 20) } diff --git a/android-llmservice/app/src/main/kotlin/net/ladenthin/android/llmservice/LlmServiceApp.kt b/android-llmservice/app/src/main/kotlin/net/ladenthin/android/llmservice/LlmServiceApp.kt index 98c9cace..0150f129 100644 --- a/android-llmservice/app/src/main/kotlin/net/ladenthin/android/llmservice/LlmServiceApp.kt +++ b/android-llmservice/app/src/main/kotlin/net/ladenthin/android/llmservice/LlmServiceApp.kt @@ -13,8 +13,9 @@ import java.io.File * working data on every **cold start**, so the app always begins fresh regardless of how it was last * killed (swipe-away, low-memory kill, crash — none of which reliably call `onDestroy`). * - * "Working data" = the copied model ([MODEL_COPY_NAME], which can be gigabytes) and the cache dir. - * The **only** thing that intentionally survives is the user's explicitly saved session + * "Working data" = the copied model ([MODEL_COPY_NAME], which can be gigabytes), the copied vision + * projector ([MMPROJ_COPY_NAME]), and the cache dir. The **only** thing that intentionally survives + * is the user's explicitly saved session * ([SessionStore]) — it exists precisely so the user can opt in to keeping a chat; everything else is * ephemeral. `MainActivity` additionally calls [clearWorkingData] on finish for prompt cleanup, but * this cold-start wipe is the reliable guarantee. @@ -36,6 +37,7 @@ class LlmServiceApp : Application() { */ fun clearWorkingData(context: Context) { File(context.filesDir, MODEL_COPY_NAME).delete() + File(context.filesDir, MMPROJ_COPY_NAME).delete() context.cacheDir?.listFiles()?.forEach { it.deleteRecursively() } } } diff --git a/android-llmservice/app/src/main/kotlin/net/ladenthin/android/llmservice/MainActivity.kt b/android-llmservice/app/src/main/kotlin/net/ladenthin/android/llmservice/MainActivity.kt index 5bce8231..101bb7f1 100644 --- a/android-llmservice/app/src/main/kotlin/net/ladenthin/android/llmservice/MainActivity.kt +++ b/android-llmservice/app/src/main/kotlin/net/ladenthin/android/llmservice/MainActivity.kt @@ -145,6 +145,23 @@ private fun ChatScreen(viewModel: ChatViewModel) { } val onChoose: () -> Unit = { picker.launch(arrayOf("*/*")) } + // Vision projector (mmproj) picker — optional, sets or replaces the vision model for the next load. + val mmprojPicker = rememberLauncherForActivityResult(ActivityResultContracts.OpenDocument()) { uri: Uri? -> + if (uri != null) { + viewModel.loadMmprojFromUri(uri, queryDisplayName(context, uri)) + } + } + val onChooseMmproj: () -> Unit = { mmprojPicker.launch(arrayOf("*/*")) } + + // Image attachment picker — stages one image for the next Send; only usable once a vision + // model is loaded (the attach button in Conversation is gated on that). + val imagePicker = rememberLauncherForActivityResult(ActivityResultContracts.OpenDocument()) { uri: Uri? -> + if (uri != null) { + viewModel.attachImage(uri, queryDisplayName(context, uri)) + } + } + val onAttachImage: () -> Unit = { imagePicker.launch(arrayOf("image/*")) } + // One-shot notices (saved / loaded / none) -> snackbar. Resolve the localized text in // composable scope, then show it from the effect. val noticeText = state.notice?.let { stringResource(noticeRes(it)) } @@ -218,16 +235,23 @@ private fun ChatScreen(viewModel: ChatViewModel) { Box(modifier = Modifier.fillMaxSize().padding(padding)) { val hasConversation = state.messages.isNotEmpty() || state.modelState == ChatViewModel.ModelState.READY when { - state.modelState == ChatViewModel.ModelState.LOADING && !hasConversation -> - LoadingView() - - !hasConversation -> + // No model loaded (fresh start, or after unloadModel()/a failed load) always goes + // to the picker screen — regardless of leftover chat history — so its ❌ quit + // button and "choose a model" action are reachable. Leftover messages are kept in + // state and simply reappear once a model becomes READY again (R3.8). + state.modelState == ChatViewModel.ModelState.NONE -> ChooseModelView( error = errorText(state.error), + visionModelName = state.visionModelName, onChoose = onChoose, + onChooseMmproj = onChooseMmproj, + onClearMmproj = viewModel::clearMmproj, onRequestQuit = { showQuitConfirm = true }, ) + state.modelState == ChatViewModel.ModelState.LOADING && !hasConversation -> + LoadingView() + else -> Conversation( state = state, @@ -235,6 +259,8 @@ private fun ChatScreen(viewModel: ChatViewModel) { onStop = viewModel::stopGeneration, onRegenerate = viewModel::regenerate, onChoose = onChoose, + onAttachImage = onAttachImage, + onClearAttachment = viewModel::clearPendingImage, ) } } @@ -502,7 +528,14 @@ private fun LoadingView() { } @Composable -private fun ChooseModelView(error: String?, onChoose: () -> Unit, onRequestQuit: () -> Unit) { +private fun ChooseModelView( + error: String?, + visionModelName: String?, + onChoose: () -> Unit, + onChooseMmproj: () -> Unit, + onClearMmproj: () -> Unit, + onRequestQuit: () -> Unit, +) { Box(modifier = Modifier.fillMaxSize()) { Column( modifier = Modifier.fillMaxSize().padding(24.dp), @@ -517,6 +550,24 @@ private fun ChooseModelView(error: String?, onChoose: () -> Unit, onRequestQuit: ) { Text(stringResource(R.string.action_choose_model)) } + // Optional vision projector (mmproj): picked independently of the main model, applied on + // the next load. Shows the selected file's name with a ✕ to clear it once set. + Row(verticalAlignment = Alignment.CenterVertically, modifier = Modifier.padding(top = 8.dp)) { + TextButton(onClick = onChooseMmproj, modifier = Modifier.testTag("chooseMmprojButton")) { + Text( + if (visionModelName != null) { + stringResource(R.string.action_vision_model_set, visionModelName) + } else { + stringResource(R.string.action_choose_vision_model) + }, + ) + } + if (visionModelName != null) { + IconButton(onClick = onClearMmproj, modifier = Modifier.testTag("clearMmprojButton")) { + Text("✕") + } + } + } DeviceCard(modifier = Modifier.padding(top = 24.dp)) if (error != null) { Text(text = error, color = MaterialTheme.colorScheme.error, modifier = Modifier.padding(top = 16.dp)) @@ -581,6 +632,8 @@ private fun Conversation( onStop: () -> Unit, onRegenerate: (String) -> Unit, onChoose: () -> Unit, + onAttachImage: () -> Unit, + onClearAttachment: () -> Unit, ) { val systemPrompt = stringResource(R.string.system_prompt) val context = LocalContext.current @@ -662,10 +715,38 @@ private fun Conversation( } } + // Pending image attachment: a small removable chip shown above the input once one is staged. + state.pendingImage?.let { pending -> + Row( + modifier = Modifier.fillMaxWidth().padding(horizontal = 12.dp), + verticalAlignment = Alignment.CenterVertically, + ) { + SuggestionChip( + onClick = {}, + label = { Text("🖼️ ${pending.displayName}") }, + modifier = Modifier.weight(1f, fill = false).testTag("pendingImageChip"), + ) + IconButton(onClick = onClearAttachment, modifier = Modifier.testTag("removeAttachmentButton")) { + Text("✕") + } + } + } + Row( modifier = Modifier.fillMaxWidth().padding(12.dp), verticalAlignment = Alignment.CenterVertically, ) { + // Attach an image for the loaded vision model; hidden entirely when no vision projector + // is loaded (attaching an image to a text-only model would just be ignored). + if (state.visionModelName != null) { + IconButton( + onClick = onAttachImage, + enabled = ready && !state.generating, + modifier = Modifier.testTag("attachImageButton"), + ) { + Text("📎") + } + } OutlinedTextField( value = draft, onValueChange = { draft = it }, @@ -687,7 +768,7 @@ private fun Conversation( onSend(draft, systemPrompt) draft = "" }, - enabled = ready && draft.isNotBlank(), + enabled = ready && (draft.isNotBlank() || state.pendingImage != null), modifier = Modifier.padding(start = 8.dp).testTag("sendButton"), ) { Text(stringResource(R.string.action_send)) @@ -761,7 +842,8 @@ private fun MessageBubble(message: ChatViewModel.Message) { }, ), ) { - Text(text = message.text.ifEmpty { "…" }, modifier = Modifier.padding(12.dp)) + val prefix = if (message.imageBytes != null) "🖼️ " else "" + Text(text = prefix + message.text.ifEmpty { "…" }, modifier = Modifier.padding(12.dp)) } } } diff --git a/android-llmservice/app/src/main/res/values-ar/strings.xml b/android-llmservice/app/src/main/res/values-ar/strings.xml index 5c0725ed..74d6c82e 100644 --- a/android-llmservice/app/src/main/res/values-ar/strings.xml +++ b/android-llmservice/app/src/main/res/values-ar/strings.xml @@ -75,4 +75,8 @@ SPDX-License-Identifier: MIT خروج الخروج والتنظيف؟ يغلق التطبيق ويحذف بيانات العمل (النموذج المنسوخ + ذاكرة التخزين المؤقت). تُحفظ المحادثات المخزنة. + + + إضافة الرؤية (اختياري) + الرؤية: %1$s diff --git a/android-llmservice/app/src/main/res/values-de/strings.xml b/android-llmservice/app/src/main/res/values-de/strings.xml index ca0d9471..34a79d12 100644 --- a/android-llmservice/app/src/main/res/values-de/strings.xml +++ b/android-llmservice/app/src/main/res/values-de/strings.xml @@ -75,4 +75,8 @@ SPDX-License-Identifier: MIT Beenden Beenden und aufräumen? Schließt die App und löscht Arbeitsdaten (kopiertes Modell + Cache). Gespeicherte Chats bleiben erhalten. + + + Sehen hinzufügen (optional) + Sehen: %1$s diff --git a/android-llmservice/app/src/main/res/values-es/strings.xml b/android-llmservice/app/src/main/res/values-es/strings.xml index f7c91da9..164450e7 100644 --- a/android-llmservice/app/src/main/res/values-es/strings.xml +++ b/android-llmservice/app/src/main/res/values-es/strings.xml @@ -75,4 +75,8 @@ SPDX-License-Identifier: MIT Salir ¿Salir y limpiar? Cierra la app y elimina los datos de trabajo (el modelo copiado + la caché). Los chats guardados se conservan. + + + Añadir visión (opcional) + Visión: %1$s diff --git a/android-llmservice/app/src/main/res/values-fr/strings.xml b/android-llmservice/app/src/main/res/values-fr/strings.xml index 3411cf2b..9b624127 100644 --- a/android-llmservice/app/src/main/res/values-fr/strings.xml +++ b/android-llmservice/app/src/main/res/values-fr/strings.xml @@ -75,4 +75,8 @@ SPDX-License-Identifier: MIT Quitter Quitter et nettoyer ? Ferme l\'application et supprime les données de travail (le modèle copié + le cache). Les conversations enregistrées sont conservées. + + + Ajouter la vision (facultatif) + Vision : %1$s diff --git a/android-llmservice/app/src/main/res/values-hi/strings.xml b/android-llmservice/app/src/main/res/values-hi/strings.xml index eb78d791..7ff8911c 100644 --- a/android-llmservice/app/src/main/res/values-hi/strings.xml +++ b/android-llmservice/app/src/main/res/values-hi/strings.xml @@ -75,4 +75,8 @@ SPDX-License-Identifier: MIT बंद करें बंद करें और साफ़ करें? ऐप बंद करता है और कार्यशील डेटा (कॉपी किया गया मॉडल + कैश) हटाता है। सहेजी गई चैट बनी रहती हैं। + + + विज़न जोड़ें (वैकल्पिक) + विज़न: %1$s diff --git a/android-llmservice/app/src/main/res/values-it/strings.xml b/android-llmservice/app/src/main/res/values-it/strings.xml index 6f2f0377..67f80105 100644 --- a/android-llmservice/app/src/main/res/values-it/strings.xml +++ b/android-llmservice/app/src/main/res/values-it/strings.xml @@ -75,4 +75,8 @@ SPDX-License-Identifier: MIT Esci Uscire e ripulire? Chiude l\'app ed elimina i dati di lavoro (il modello copiato + la cache). Le chat salvate vengono mantenute. + + + Aggiungi visione (opzionale) + Visione: %1$s diff --git a/android-llmservice/app/src/main/res/values-ja/strings.xml b/android-llmservice/app/src/main/res/values-ja/strings.xml index 18d1d40c..ec54633a 100644 --- a/android-llmservice/app/src/main/res/values-ja/strings.xml +++ b/android-llmservice/app/src/main/res/values-ja/strings.xml @@ -75,4 +75,8 @@ SPDX-License-Identifier: MIT 終了 終了してクリーンアップしますか？ アプリを閉じて作業データ（コピーしたモデル + キャッシュ）を削除します。保存済みのチャットは残ります。 + + + ビジョンを追加（任意） + ビジョン: %1$s diff --git a/android-llmservice/app/src/main/res/values-ko/strings.xml b/android-llmservice/app/src/main/res/values-ko/strings.xml index 59fdcbe6..f189086a 100644 --- a/android-llmservice/app/src/main/res/values-ko/strings.xml +++ b/android-llmservice/app/src/main/res/values-ko/strings.xml @@ -75,4 +75,8 @@ SPDX-License-Identifier: MIT 종료 종료하고 정리할까요? 앱을 닫고 작업 데이터(복사된 모델 + 캐시)를 삭제합니다. 저장된 채팅은 유지됩니다. + + + 비전 추가 (선택 사항) + 비전: %1$s diff --git a/android-llmservice/app/src/main/res/values-pt/strings.xml b/android-llmservice/app/src/main/res/values-pt/strings.xml index c645b913..0ac81219 100644 --- a/android-llmservice/app/src/main/res/values-pt/strings.xml +++ b/android-llmservice/app/src/main/res/values-pt/strings.xml @@ -75,4 +75,8 @@ SPDX-License-Identifier: MIT Sair Sair e limpar? Fecha a app e elimina os dados de trabalho (o modelo copiado + a cache). As conversas guardadas são mantidas. + + + Adicionar visão (opcional) + Visão: %1$s diff --git a/android-llmservice/app/src/main/res/values-ru/strings.xml b/android-llmservice/app/src/main/res/values-ru/strings.xml index 4405fc0c..bbbb4a63 100644 --- a/android-llmservice/app/src/main/res/values-ru/strings.xml +++ b/android-llmservice/app/src/main/res/values-ru/strings.xml @@ -75,4 +75,8 @@ SPDX-License-Identifier: MIT Выйти Выйти и очистить? Закрывает приложение и удаляет рабочие данные (скопированную модель + кэш). Сохранённые чаты остаются. + + + Добавить зрение (опционально) + Зрение: %1$s diff --git a/android-llmservice/app/src/main/res/values-tr/strings.xml b/android-llmservice/app/src/main/res/values-tr/strings.xml index 3f17dd15..16cbe9a3 100644 --- a/android-llmservice/app/src/main/res/values-tr/strings.xml +++ b/android-llmservice/app/src/main/res/values-tr/strings.xml @@ -75,4 +75,8 @@ SPDX-License-Identifier: MIT Çık Çıkılsın ve temizlensin mi? Uygulamayı kapatır ve çalışma verilerini siler (kopyalanan model + önbellek). Kaydedilen sohbetler korunur. + + + Görme ekle (isteğe bağlı) + Görme: %1$s diff --git a/android-llmservice/app/src/main/res/values-zh-rCN/strings.xml b/android-llmservice/app/src/main/res/values-zh-rCN/strings.xml index 65ae4796..c87f2401 100644 --- a/android-llmservice/app/src/main/res/values-zh-rCN/strings.xml +++ b/android-llmservice/app/src/main/res/values-zh-rCN/strings.xml @@ -75,4 +75,8 @@ SPDX-License-Identifier: MIT 退出 退出并清理？ 关闭应用并删除工作数据（已复制的模型 + 缓存）。已保存的对话会保留。 + + + 添加视觉功能（可选） + 视觉模型：%1$s diff --git a/android-llmservice/app/src/main/res/values/strings.xml b/android-llmservice/app/src/main/res/values/strings.xml index bb0c5c97..eb8bb0bf 100644 --- a/android-llmservice/app/src/main/res/values/strings.xml +++ b/android-llmservice/app/src/main/res/values/strings.xml @@ -82,4 +82,8 @@ SPDX-License-Identifier: MIT Quit Quit and clean up? Closes the app and deletes working data (the copied model + cache). Saved chats are kept. + + + Add vision (optional) + Vision: %1$s diff --git a/android-llmservice/requirements.md b/android-llmservice/requirements.md index a375bdf6..32638ef6 100644 --- a/android-llmservice/requirements.md +++ b/android-llmservice/requirements.md @@ -47,7 +47,7 @@ by hand only (no automated test); `build` = enforced at build/resource-compile t | R2.2 | `allowBackup=false` — conversation data is not included in system backups. | `AndroidManifest.xml` | build | | R2.3 | All inference runs **on-device** (CPU); no request is ever made to a remote server by the app. | `ChatViewModel` (no network calls) | manual | | R2.4 | A **"🔒 Offline · fully on-device"** badge is shown on the model-picker screen. | `MainActivity.OfflineBadge`; `badge_offline` | manual | -| R2.5 | **Transient working data is ephemeral:** the copied model (`current-model.gguf`) and the cache dir are **wiped on every cold start** (`LlmServiceApp.onCreate`) — guaranteeing a fresh start regardless of how the app was last killed — and best-effort on finish (`MainActivity.onDestroy` when `isFinishing`). The **only** data that persists is the user's **explicitly saved** session (`session.json`, opt-in). | `LlmServiceApp`; `MainActivity.onDestroy` | manual | +| R2.5 | **Transient working data is ephemeral:** the copied model (`current-model.gguf`), the copied vision projector (`current-mmproj.gguf`, see R12), and the cache dir are **wiped on every cold start** (`LlmServiceApp.onCreate`) — guaranteeing a fresh start regardless of how the app was last killed — and best-effort on finish (`MainActivity.onDestroy` when `isFinishing`). The **only** data that persists is the user's **explicitly saved** session (`session.json`, opt-in). | `LlmServiceApp`; `MainActivity.onDestroy` | manual | | R2.6 | **Quit & clean up:** a ❌ button on the model-picker (main) screen opens a confirm dialog that, on confirm, **wipes the working data** (model copy + cache, keeping the saved session) and **closes the app** (`finishAndRemoveTask` — removed from Recents). | `MainActivity` (`quitButton`; `LlmServiceApp.clearWorkingData` + `finishAndRemoveTask`) | manual | ## R3 — Model selection & loading @@ -55,7 +55,7 @@ by hand only (no automated test); `build` = enforced at build/resource-compile t | ID | Requirement | Source | Verified by | |---|---|---|---| | R3.1 | The user picks a GGUF via the **Storage Access Framework** (`ACTION_OPEN_DOCUMENT`, `*/*`) — no storage permission. | `MainActivity` picker | manual | -| R3.2 | A picked `content://` model is **copied into app-private `filesDir`** (`current-model.gguf` = `MODEL_COPY_NAME`) and loaded **by real path** (llama.cpp mmaps a real path, not a `content://` URI). The copy is **ephemeral** (wiped on cold start / close — see R2.5), so a model is re-picked after the app is fully closed. | `ChatViewModel.copyToInternal` | manual | +| R3.2 | A picked `content://` model is **copied into app-private `filesDir`** (`current-model.gguf` = `MODEL_COPY_NAME`) and loaded **by real path** (llama.cpp mmaps a real path, not a `content://` URI). The copy is **ephemeral** (wiped on cold start / close — see R2.5), so a model is re-picked after the app is fully closed. | `ChatViewModel.copyUriToInternal` | manual | | R3.3 | The model loads **CPU-only** (`setGpuLayers(0)`) with context size **2048**, portable across every device. | `ChatViewModel.openModel` (`CONTEXT_SIZE`) | manual | | R3.4 | While loading, a **LOADING** state is shown (spinner + "Loading model…"); on failure a localized load error is shown and state returns to NONE. | `ChatViewModel.ModelState`; `error_load_model` | manual | | R3.5 | Loading a new model **closes** the previously loaded one (native memory is not GC-managed). | `ChatViewModel.openModel` | manual | @@ -142,6 +142,18 @@ by hand only (no automated test); `build` = enforced at build/resource-compile t | R11.2 | `ChatFlowInstrumentedTest` launches the app with a preloaded model, types a prompt, taps Send, and asserts a **non-empty streamed assistant reply**; it self-skips when the adb-pushed model is absent. | `androidTest/.../ChatFlowInstrumentedTest.kt` | instrumented | | R11.3 | **Coverage gap:** there are currently **no unit tests** for `ChatViewModel`, `SessionStore`, `Languages`, or the settings/log/chat-action logic. This file is the spec of record until such tests exist. | — | — | +## R12 — Vision (mmproj) model + image attachment + +| ID | Requirement | Source | Verified by | +|---|---|---|---| +| R12.1 | The model-picker screen has an **optional** "Add vision" button that picks a vision projector (**mmproj**) GGUF via SAF, independent of and in either order relative to the main model. Once set, the button shows the projector's file name and a ✕ to clear it. | `MainActivity.ChooseModelView` (`chooseMmprojButton`/`clearMmprojButton`); `ChatViewModel.loadMmprojFromUri`/`clearMmproj` | manual | +| R12.2 | The picked mmproj is **copied into app-private `filesDir`** (`current-mmproj.gguf` = `MMPROJ_COPY_NAME`, same real-path-not-URI reasoning as R3.2) and passed to `ModelParameters.setMmproj(...)` (+ `setDevices("none")` + `setMmprojOffload(false)`, mirroring the CPU-only config `MultimodalIntegrationTest` validates) the next time a model is loaded. | `ChatViewModel.openModel` | manual | +| R12.3 | Once a vision projector is loaded, a **📎 attach** button appears in the chat input row; tapping it picks an image (`image/*`) via SAF and stages it as a **pending attachment chip** (🖼️ filename + ✕ to remove) above the input. | `MainActivity.Conversation` (`attachImageButton`/`pendingImageChip`/`removeAttachmentButton`); `ChatViewModel.attachImage`/`clearPendingImage` | manual | +| R12.4 | **Send** accepts either non-blank text, a pending image, or both; a message with an image is built as a multimodal `ChatMessage` (`ContentPart.text` + `ContentPart.imageBytes`) instead of plain text. | `ChatViewModel.send`/`Message.toChatMessage` | manual | +| R12.5 | Attached images are **session-transient**: shown in the message bubble (🖼️ prefix) for the lifetime of the app process, but **not** persisted by `SessionStore` (save/load keeps text only) and not copied to disk (read into memory only). | `ChatViewModel.Message`; `SessionStore` | manual | +| R12.6 | **Unloading the model** (R3.8) also clears the selected vision projector (path + copied file) and any pending image attachment — picking a new model starts vision selection fresh. | `ChatViewModel.unloadModel` | manual | +| R12.7 | The vision projector copy is included in the same **privacy wipe** as the model copy: deleted on cold start (`LlmServiceApp.onCreate`) and on `clearMmproj`/`unloadModel`. | `LlmServiceApp.clearWorkingData` | manual | + --- ## Known coverage gaps / follow-ups diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 08c1f1aa..24e66500 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -475,3 +475,9 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b9957–b9959 | upstream verification (compare diff) | All **eight** patches (`0001`–`0008`) apply unchanged against b9959: the single-range diff touches only `ggml/src/ggml-cpu/arch/arm/quants.c` (a NEON intrinsic-wrapper edit) and `scripts/sync-ggml.last` — no patch-target file (`common/arg.*`, `tools/server/*`) and no OuteTTS generator anchor (`tools/tts/tts.cpp` unchanged). All eight priority headers byte-identical across the range. Full configure (fail-loud patch apply + TTS extraction) + build + `ctest` to be confirmed by the CI pipeline. | | b9959–b9964 | `common/arg.cpp` + `tools/mtmd/{clip-graph.h,clip-model.h,clip.cpp,models/deepseekocr.cpp,models/models.h,mtmd-image.{cpp,h},mtmd.cpp}` + `tools/server/server-http.cpp` | **No public-API break** (13 files, ~46 KiB full / ~35 KiB excluding the auto-followed `tools/ui` housekeeping commit). `common/arg.cpp`'s change is confined to `common_models_handler_apply`'s HF-download task-planning order (speculative-draft/vocoder plan handling moved earlier, an `LLAMA_SERVER_WORKER_CMD`-style download-task debug log added) — a different region than patch `0001`'s `common_params_parse`/`common_params_parse_main` block, so `0001` still applies. The `tools/mtmd/*` churn is DeepSeek-OCR model-family image-preprocessing internals (merges `PROJECTOR_TYPE_DEEPSEEKOCR`/`DEEPSEEKOCR2` onto one preprocessor, adds a debug log, reworks `mtmd-image.{cpp,h}` internals) — `tools/mtmd/mtmd.h` (the public header in the priority-8 review list) is untouched and this project does not call `mtmd-image` internals directly. `tools/server/server-http.cpp` reworks the not-ready-state middleware to allow any embedded frontend asset path (not just `/` and `*.html`) to load while the server is starting — internal HTTP-transport logic, not a patch target (patches touch `server.cpp`/`server-context.cpp`/`server-models.cpp`, all byte-identical across the range) and not part of the JNI-called surface. All **eight** priority headers byte-identical; no project source changes required. | | b9959–b9964 | upstream verification (sandbox) | All **eight** patches (`0001`–`0008`) re-verified against b9964: applied in filename order onto a clean b9964 checkout via a fresh `cmake -B build` (fail-loud `PATCH_COMMAND`), all clean (the range touches no patch-target region — `common/arg.cpp`'s change and `tools/mtmd`/`tools/server/server-http.cpp` sit outside every patch's edited lines). No OuteTTS generator anchor touched (`tools/tts/tts.cpp` unchanged — the generator extracted `tts.cpp @ b9964`). Full local verification: fresh configure (fail-loud patch apply + TTS extraction) + full build (`jllama` + `jllama_test`) + `ctest`; per-platform confirmation by the CI pipeline. | +| b9964–b9967 | `src/llama-model.cpp` + `tools/server/server-schema.cpp` + `ggml/src/ggml-hexagon/**` | **No public-API break** (4 files, ~23 KiB). `src/llama-model.cpp`'s change makes `llama_meta_device_get_split_state`'s per-tensor-name `std::regex` patterns `static` (compiled once instead of per call) — a pure perf fix inside an upstream-compiled TU, no signature change. `tools/server/server-schema.cpp` adds a `has_value()` helper so `field_num`/`field_str`/`field_bool`/`field_json`'s `.eval()` treat an explicit JSON `null` the same as an absent key (falls back to the server default instead of attempting to parse `null` as the field's type) — purely more permissive request parsing, no `eval_llama_cmpl_schema` signature change, and not a patch target. `ggml/src/ggml-hexagon/**` is Qualcomm Hexagon DSP backend work (new argsort ops), not built by this project (not in the classifier matrix). All **eight** priority headers byte-identical; no project source changes required. | +| b9964–b9967 | upstream verification (sandbox) | All **eight** patches (`0001`–`0008`) re-verified against b9967: applied in filename order onto a clean b9967 checkout via a fresh `cmake -B build` (fail-loud `PATCH_COMMAND`), all clean (the range touches no patch-target file). No OuteTTS generator anchor touched (`tools/tts/tts.cpp` unchanged — the generator extracted `tts.cpp @ b9967`). Full local configure verified (fail-loud patch apply + TTS extraction succeeded); per-platform build + `ctest` confirmation by the CI pipeline. | +| b9967–b9968 | `ggml/src/ggml-opencl/**` | **No public-API break** (21 files, ~279 KiB — a single-commit step over the 100 KiB chunk threshold with no smaller intermediate tag available). Entirely new OpenCL MoE/GEMM kernels (mxfp4/q4_0/q4_k/q5_0/q5_k/q6_k/q8_0/q8_1 GEMM+GEMV variants, `moe_combine`, `moe_reorder_quant_a_q8_1`, `quant_a_q8_1`) plus supporting `ggml-opencl.cpp` dispatch code and a `CMakeLists.txt` source-list update — Adreno-tuned MoE inference speedups, entirely inside the OpenCL backend this project *does* build (Android OpenCL classifier + Windows/Linux OpenCL classifiers), but the new kernels are picked up automatically by upstream's own `ggml-opencl/CMakeLists.txt` glob/embed step (`GGML_OPENCL_EMBED_KERNELS`), which this project invokes unmodified. No `ggml.h`/`ggml-backend.h` (or any priority-8 header) surface touched; no project source changes required. | +| b9967–b9968 | upstream verification (sandbox) | All **eight** patches (`0001`–`0008`) re-verified against b9968: applied in filename order onto a clean b9968 checkout via a fresh `cmake -B build` (fail-loud `PATCH_COMMAND`), all clean (the range touches no patch-target file). No OuteTTS generator anchor touched (`tools/tts/tts.cpp` unchanged — the generator extracted `tts.cpp @ b9968`). Full local configure verified (fail-loud patch apply + TTS extraction succeeded); per-platform build + `ctest` confirmation by the CI pipeline. | +| b9968–b9972 | `ggml/include/{ggml.h,ggml-rpc.h}` + `ggml/src/{ggml.c,ggml-cpu/**,ggml-vulkan/ggml-vulkan.cpp}` + `src/{llama-context.cpp,llama-cparams.h,llama-graph.{cpp,h},models/deepseek32.cpp,models/deepseek4.cpp}` + `tools/server/{server-context.cpp,server-http.{cpp,h},server-stream.{cpp,h},server-tools.{cpp,h}}` | **Additive-only ggml API, internal-only server refactor, no break** (19 non-test files, ~62 KiB — a 4-commit final chunk). **(1)** New op `GGML_OP_LIGHTNING_INDEXER` + `ggml_lightning_indexer(...)` for DeepSeek Sparse Attention's "lightning indexer" (purely additive; `GGML_OP_COUNT` 97→98, `ggml-rpc.h`'s matching `static_assert`/`RPC_PROTO_PATCH_VERSION` bumped in lockstep — RPC backend not built here); CPU (`ggml-cpu.c`/`ops.{cpp,h}`) and Vulkan implement it, `deepseek32.cpp`/`deepseek4.cpp` wire it into the DeepSeek graph builders; `llama-cparams.h` gains two internal bools (`fused_lid`/`auto_flid`) alongside the existing fused-gated-delta-net flags. **(2)** `tools/server/server-stream.h`'s producer-pipe API is refactored: `stream_pipe_producer::create` now returns a raw pointer (was `shared_ptr`) and the free functions `server_stream_session_attach_pipe`/`server_stream_aware_should_stop` are folded into a new `server_res_spipe` base class (`set_req`/`should_stop`/`on_complete`/`set_next` methods) — `server_res_generator` (`server-context.cpp`) now extends `server_res_spipe` instead of `server_http_res` directly and calls those methods instead of the removed free functions. **(3)** `server-tools.{cpp,h}`'s `server_tool::invoke()` gains a `stream*` parameter (streamed shell-command output) and `tools_io::run()` gains an `on_chunk` callback. None of (2)/(3) touch a patch-target region (patches `0002`/`0005` anchor `server-context.cpp` near `load_model`/`update_slots`, both far from the SSE-response-generator code this chunk touches; `server-stream.h`/`server-tools.{cpp,h}` are not patch targets at all) and no priority-8 header changed. No project source changes required. | +| b9968–b9972 | upstream verification (sandbox) | All **eight** patches (`0001`–`0008`) re-verified against b9972: applied in filename order onto a clean b9972 checkout via a fresh `cmake -B build` (fail-loud `PATCH_COMMAND`), all clean. No OuteTTS generator anchor touched (`tools/tts/tts.cpp` unchanged — the generator extracted `tts.cpp @ b9972`). **Full local verification given the larger surface of this chunk:** fresh configure + full `cmake --build` (`jllama` + `jllama_test` link cleanly, confirming the `server_res_spipe` refactor and the new `GGML_OP_LIGHTNING_INDEXER` op compile against this project's TUs) + `ctest` **485/485 passing**; per-platform confirmation by the CI pipeline. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index a72e1290..f799380f 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b9964 + GIT_TAG b9972 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= @@ -196,7 +196,7 @@ execute_process( COMMAND ${CMAKE_COMMAND} -DTTS_SRC=${llama.cpp_SOURCE_DIR}/tools/tts/tts.cpp -DOUT_CPP=${JLLAMA_TTS_GEN_CPP} - -DLLAMA_TAG=b9964 + -DLLAMA_TAG=b9972 -P ${CMAKE_CURRENT_SOURCE_DIR}/cmake/generate-tts-upstream.cmake RESULT_VARIABLE JLLAMA_TTS_GEN_RESULT ) diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index f1e900f0..2b85c560 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b9964"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b9972"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b9964-0badc06ab"} — call + * plus the resolved upstream commit, e.g. {@code "b9972-0badc06ab"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b9964"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b9972"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b9964"; + public static final String LLAMA_CPP_VERSION = "b9972"; // Constants holder — not instantiable. private LlamaCppVersion() {}