From d7cb501d700f663a24e243ea8b8478099af3c9f8 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 13 Jul 2026 07:10:15 +0000 Subject: [PATCH 1/3] Upgrade llama.cpp from b9975 to b9981 Drop patch 0005 (recurrent near-prompt-end checkpoints): upstream's own server-context.cpp create_checkpoint/do_checkpoint now solve the same checkpoint-starvation problem with a strict superset fix (id_task-tracked eviction + an unconditional near_prompt_end min-step exemption, vs our patch's recurrent/hybrid-only scoping), so 0005 no longer applies and is redundant. Verified via a fresh fail-loud cmake configure against b9981 with the remaining seven patches (0001-0004, 0006-0008) applying cleanly. --- CLAUDE.md | 20 +++++++--- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 + llama/CMakeLists.txt | 4 +- ...ecurrent-near-prompt-end-checkpoints.patch | 39 ------------------- .../llama/value/LlamaCppVersion.java | 8 ++-- 6 files changed, 24 insertions(+), 51 deletions(-) delete mode 100644 llama/patches/0005-server-recurrent-near-prompt-end-checkpoints.patch diff --git a/CLAUDE.md b/CLAUDE.md index e21a06ab..db6b7418 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b9975** +Current llama.cpp pinned version: **b9981** ## Upgrading CUDA Version @@ -421,7 +421,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b9975 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b9981 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -461,7 +461,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b9975`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b9981`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -573,11 +573,21 @@ Current patches: | `0002-server-preserve-caller-load-progress-callback.patch` | Load-progress-callback regression introduced in llama.cpp **b9789**: `server_context::load_model` (`tools/server/server-context.cpp`) now **unconditionally** installs the server's own load-progress reporter on `params_base.load_progress_callback` immediately before `common_init_from_params`, clobbering any callback the embedding caller already set. libjllama's `LoadProgressCallback` feature wires `common_params.load_progress_callback` to a JNI trampoline *before* calling `load_model`, so the bump silently killed it — `LoadProgressCallbackTest` saw zero progress updates and the abort-on-`false` path never threw. The patch guards the assignment with `if (params_base.load_progress_callback == nullptr)`, so the server installs its own reporter **only when the caller hasn't** — a caller-supplied callback survives and fires during load. Standalone `llama-server` (no caller callback, so the field is null) is unaffected. Same JNI-vs-standalone divergence class as `0001`. | | `0003-pr22393-server-add-slot-prompt-similarity-getter-setter.patch` | **Upstream-PR carry** of [ggml-org/llama.cpp#22393](https://github.com/ggml-org/llama.cpp/pull/22393) ("server : add slot_prompt_similarity getter/setter") while it is still open upstream. Purely additive: adds `server_context::get_slot_prompt_similarity()` / `set_slot_prompt_similarity(float)` (`tools/server/server-context.{cpp,h}`) so an embedding/JNI caller can query and tune the slot-selection threshold at runtime without reloading the model. Verbatim copy of the PR — drop it once a pinned `b` includes the change. | | `0004-pr23116-server-per-request-reasoning-budget-tokens.patch` | **Upstream-PR carry** of [ggml-org/llama.cpp#23116](https://github.com/ggml-org/llama.cpp/pull/23116) ("server: honour per-request reasoning_budget_tokens in chat completions"), motivated by java-llama.cpp#140, while it is still open upstream. `oaicompat_chat_params_parse` (`tools/server/server-common.cpp`) only read the Anthropic `thinking_budget_tokens` alias and always wrote the server-level `reasoning_budget_message`, so a per-request `reasoning_budget_tokens` / `reasoning_budget_message` on a chat-completions request was ignored. The patch reads both overrides **before** the generic copy loop (precedence: `reasoning_budget_tokens` > `thinking_budget_tokens` alias > server default) and threads the per-request message through. Carries the upstream `tests/test-chat.cpp` additions verbatim so the patch is submittable as-is; like `0001`'s test/call-site flips they are **applied-but-not-compiled** here (`LLAMA_BUILD_TESTS` is OFF for the FetchContent subproject). Drop it once a pinned `b` includes the change. | -| `0005-server-recurrent-near-prompt-end-checkpoints.patch` | **Multi-turn tool-calling perf fix for recurrent/hybrid models (e.g. Granite-4)**, upstream-submittable. In `server_context::update_slots` (`tools/server/server-context.cpp`) the near-prompt-end context checkpoints are gated by `checkpoint_min_step` (default 8192 tokens). An agentic conversation that appends only assistant/tool messages never produces a new user-message checkpoint (`is_user_start`/`is_last_user_message` match `COMMON_CHAT_ROLE_USER` only), so after turn 1 no new checkpoint is ever created and — because recurrent state can only roll back to a checkpoint — **every turn re-prefills the whole conversation tail** (measured on a synthetic granitehybrid model: prefilled tokens grew 901 → 1544 → 2187 → 2830 → 3473 over turns 2–6). The patch (1) exempts near-prompt-end checkpoints from the min-step spacing when the memory can only roll back via checkpoints (`ctx_tgt_seq_rm_type` is `FULL` or `RS` — SWA-only models are unaffected), and (2) skips creating a checkpoint whose position equals the newest one (the last-user-message checkpoint was re-created identically on every turn, flooding the 32-entry list). After the patch each turn restores the previous turn's near-end checkpoint and prefill is constant (~new-turn-sized; 647 tokens/turn in the same measurement, ≈5.4× less prefill at turn 6 and growing with conversation length). Validated output-identical (`temperature=0`) vs. unpatched. Complements — not duplicates — open upstream PRs #24035/#24899/#24891 (they fix checkpoint *invalidation/retention*; this fixes checkpoint *starvation*). Drop once upstream solves agentic checkpoint placement (e.g. a merged role-boundary checkpointing design, cf. #21885 / #22826 discussion). | | `0007-server-attach-http-frontend.patch` | **Adds `llama_server_attach(argc, argv, server_context&)`** so the `NativeServer` *attach mode* can serve an **already-loaded `LlamaModel`** over the full upstream HTTP frontend — no second model load, no `start_loop()`; the LlamaModel's worker keeps driving the shared `server_context` and the HTTP routes post tasks to its queue (the queue is the synchronization point). Mechanically: (1) extracts the common route table + CORS-proxy/tools blocks out of `llama_server()` into `llama_server_register_common_routes(...)` (shared verbatim, so the entry points cannot drift; returns `false` on tools-setup failure); (2) adds `llama_server_attach`, which parses only the HTTP-side argv via `common_params_parse`, starts `g_stream_sessions` GC + `server_http_context`, registers the common routes plus the non-router resumable-streaming handlers, marks ready immediately (model already loaded), and blocks on the HTTP thread until `llama_server_request_shutdown()` — never calling `common_init()`, backend init, `ctx_server.terminate()` or `llama_backend_free()` (the embedding caller owns those). Applies after `0001`+`0006` (same file); closes the "NativeServer — reuse an already-loaded LlamaModel" TODO. Upstream-submittable ("server: let embedding callers attach the HTTP frontend to an existing server_context"). | | `0008-server-models-worker-cmd-override.patch` | **Makes router mode usable in-JVM.** The router (`server-models.cpp`) spawns each model worker by re-executing its own binary (`get_server_exec_path()` = `/proc/self/exe` & friends) — inside a JVM that binary is `java`, not a llama-server, so embedded router workers could never start. The patch adds env `LLAMA_SERVER_WORKER_CMD` (whitespace-split; read in `server_model_meta::update_args`) which replaces only the leading binary-path token of the rendered worker args, letting an embedding host relaunch workers through its own bootstrap — e.g. `java -cp app.jar net.ladenthin.llama.server.NativeServer` (each worker is then a fresh JVM running the classic single-model `NativeServer`). Exposed in Java as `NativeServer.setWorkerCommand(String...)` (JNI `setenv`); exercised by `RouterModeIntegrationTest` (Linux CI). Upstream-submittable (also useful for containerized/wrapped deployments). | | `0006-server-embed-native-server-jni.patch` | **Makes `server.cpp`'s `llama_server` embeddable in the JVM** so the `NativeServer` JNI bridge can run the full upstream HTTP server (WebUI included) inside `libjllama` — see "Two server modes" below. b9870 already exposes `int llama_server(int, char**)` (non-static; no `main` in the file), so the patch only adds embedded-mode support: (1) a `g_llama_server_embedded` flag + `llama_server_set_embedded()` / `llama_server_request_shutdown()` (declared in the committed `src/main/cpp/native_server_bridge.h`); (2) skips installing the process-wide SIGINT/SIGTERM handlers when embedded (they would hijack the JVM's); (3) in embedded mode parses the **forwarded** argv via `common_params_parse` instead of `common_params_parse_main` (whose `GetCommandLineW` recovery would pick up `java.exe`'s command line — the same Windows class of bug `0001` fixes). `llama_server_request_shutdown()` mirrors the SIGTERM path (invokes the installed `shutdown_handler` → `ctx_server.terminate()` unblocks `start_loop()`), giving JNI an out-of-band stop since `ctx_server` is loop-local. Applies **after `0001`** (which flips this call site to `common_params_parse_main`), so its context is the post-`0001` tree; regenerate against `0001`+source on a bump. Only touches `tools/server/server.cpp`. | +**`0005` was dropped at the b9981 bump.** Upstream's own `server-context.cpp` picked up an +equivalent — and broader — fix for the same checkpoint-starvation problem: `create_checkpoint` +now tracks `id_task` and evicts a stale checkpoint that sits within `checkpoint_min_step` of a +newer one (instead of our pre-creation duplicate-position check), and the `do_checkpoint` gate +now exempts **every** near-prompt-end checkpoint from the min-step spacing (`near_prompt_end`, +unconditionally) rather than only recurrent/hybrid (`ctx_tgt_seq_rm_type` `FULL`/`RS`) models as +our patch scoped it. The upstream version is a strict superset of what `0005` did, so it applies +cleanly with the patch removed and needs no replacement. If a regression in agentic multi-turn +prefill behavior shows up, re-check `tools/server/server-context.cpp`'s `create_checkpoint` / +`do_checkpoint` logic against this description before reintroducing a local patch. + ## OuteTTS build-time extraction (`cmake/generate-tts-upstream.cmake`) The `TextToSpeech` native pipeline reuses llama.cpp's OuteTTS helpers (`tools/tts/tts.cpp`) @@ -1237,7 +1247,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b9975`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b9981`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index 72af61a7..e743dc85 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b9975](https://img.shields.io/badge/llama.cpp-%23b9975-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b9975) +[![llama.cpp b9981](https://img.shields.io/badge/llama.cpp-%23b9981-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b9981) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index d0da4761..9d2080de 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -483,3 +483,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b9968–b9972 | upstream verification (sandbox) | All **eight** patches (`0001`–`0008`) re-verified against b9972: applied in filename order onto a clean b9972 checkout via a fresh `cmake -B build` (fail-loud `PATCH_COMMAND`), all clean. No OuteTTS generator anchor touched (`tools/tts/tts.cpp` unchanged — the generator extracted `tts.cpp @ b9972`). **Full local verification given the larger surface of this chunk:** fresh configure + full `cmake --build` (`jllama` + `jllama_test` link cleanly, confirming the `server_res_spipe` refactor and the new `GGML_OP_LIGHTNING_INDEXER` op compile against this project's TUs) + `ctest` **485/485 passing**; per-platform confirmation by the CI pipeline. | | b9972–b9975 | `ggml/src/ggml-cuda/ggml-cuda.cu` + `ggml/src/gguf.cpp` + `src/llama-kv-cache-dsv4.{cpp,h}` | **Internal-only, no public-API surface** (4 non-test files, ~15 KiB, 3 commits). `ggml-cuda.cu`'s `ggml_backend_cuda_device_get_memory` now handles a failing `cudaMemGetInfo` gracefully (logs + returns `free=total=0`) instead of aborting via `CUDA_CHECK`. `gguf.cpp` rejects an empty GGUF key during parsing (new `HANDCRAFTED_KV_EMPTY_KEY` fuzz case in `tests/test-gguf.cpp`). `llama-kv-cache-dsv4.{cpp,h}` reworks DSV4 compressed-cache clearing to be **per-sequence** instead of whole-cache: `llama_dsv4_comp_state::clear(bool)` → `clear(llama_seq_id, bool)` and `llama_kv_cache_dsv4::clear_compressed(bool)` → `clear_compressed(llama_seq_id, bool)` (a new `-1` sentinel means "all sequences", matching the old behavior); `seq_cp`/`seq_add`/`seq_div` drop their now-unnecessary full-cache clear and `seq_keep` clears every other sequence individually — an isolation fix so removing one sequence's KV state no longer disturbs sibling sequences (new `test_seq_rm_isolated` regression case in `tests/test-save-load-state.cpp`). Both DSV4 functions are `private` members with no `llama.h`/`common.h` surface; not called by project code. No priority-8 header touched, `tools/ui` untouched, no project source changes required. | | b9972–b9975 | upstream verification (sandbox) | All **eight** patches (`0001`–`0008`) re-verified against b9975: applied in filename order onto a clean b9975 checkout via a fresh `cmake -B build` (fail-loud `PATCH_COMMAND`), all clean (the range touches no patch-target file). No OuteTTS generator anchor touched (`tools/tts/tts.cpp` unchanged — the generator extracted `tts.cpp @ b9975`). Full local configure verified (fail-loud patch apply + TTS extraction succeeded); full `cmake --build` + `ctest` verification in progress locally, per-platform confirmation by the CI pipeline. | +| b9975–b9981 | `common/common.h` + `tools/mtmd/{mtmd.h,mtmd.cpp,mtmd-cli.cpp}` + `tools/server/{server-chat.cpp,server-common.cpp,server-context.cpp,server-models.cpp}` + `vendor/cpp-httplib/{httplib.h,httplib.cpp}` + `scripts/sync_vendor.py` | **One patch-target conflict, otherwise additive (13 files, ~20 KiB, 6 commits).** `common/common.h`'s `common_prompt_checkpoint` gains an `id_task` field (additive). `mtmd.h`'s `mtmd_input_text` gains a `text_len` field alongside `text` so the tokenizer can carry embedded NUL bytes (`mtmd.cpp`/`mtmd-cli.cpp` updated to match — internal to upstream's own callers, not constructed by project code). `server-chat.cpp`'s Anthropic→OAI tool-result conversion now emits multimodal `content_parts` for `image` blocks (previously text-only) — internal to the Anthropic Messages translation path, not a patch target. `server-common.cpp`'s `process_mtmd_prompt` follows the `mtmd_input_text` field split. `server-models.cpp`'s router capability probe adds `LLAMA_ARG_MMPROJ_AUTO` to the offline-safe allowlist and checks `params.no_mmproj`. `vendor/cpp-httplib` bumped 0.49.0→0.50.1 (additive `Get(path, Params, DownloadProgress)` overloads). **The one patch-target hit:** `tools/server/server-context.cpp`'s `update_slots`/`create_checkpoint` picked up upstream's own fix for the exact problem `0005-server-recurrent-near-prompt-end-checkpoints.patch` addressed — `create_checkpoint` now tracks `id_task` and evicts a checkpoint that sits within `checkpoint_min_step` of a newer one, and `do_checkpoint`'s min-step exemption now covers **every** near-prompt-end checkpoint (`near_prompt_end`, unconditional) rather than only recurrent/hybrid models as `0005` scoped it — a strict superset of `0005`'s behavior. `0005` no longer applies (`patch does not apply` at `server-context.cpp:3560`) and was **dropped** rather than refreshed; see the CLAUDE.md patches-table note under `0005`'s old slot. No other patch-target file (`common/arg.*`, `server.cpp`, `server-models.cpp`'s worker-cmd region) touched; all eight priority-8 headers otherwise byte-identical (only `mtmd.h`, additive). | +| b9975–b9981 | upstream verification (sandbox) | **Seven** patches (`0001`–`0004`, `0006`–`0008`) re-verified against b9981: applied in filename order onto a clean b9981 checkout via a fresh `cmake -B build` (fail-loud `PATCH_COMMAND`), all clean after removing `0005` (superseded — see the row above). No OuteTTS generator anchor touched (`tools/tts/tts.cpp` unchanged — the generator extracted `tts.cpp @ b9981`). Full local configure verified (fail-loud patch apply + TTS extraction succeeded, `ggml commit 34558825a`); full `cmake --build` + `ctest` verification in progress locally, per-platform confirmation by the CI pipeline. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index 084c211e..0e0cf990 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b9975 + GIT_TAG b9981 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= @@ -196,7 +196,7 @@ execute_process( COMMAND ${CMAKE_COMMAND} -DTTS_SRC=${llama.cpp_SOURCE_DIR}/tools/tts/tts.cpp -DOUT_CPP=${JLLAMA_TTS_GEN_CPP} - -DLLAMA_TAG=b9975 + -DLLAMA_TAG=b9981 -P ${CMAKE_CURRENT_SOURCE_DIR}/cmake/generate-tts-upstream.cmake RESULT_VARIABLE JLLAMA_TTS_GEN_RESULT ) diff --git a/llama/patches/0005-server-recurrent-near-prompt-end-checkpoints.patch b/llama/patches/0005-server-recurrent-near-prompt-end-checkpoints.patch deleted file mode 100644 index 59f729ff..00000000 --- a/llama/patches/0005-server-recurrent-near-prompt-end-checkpoints.patch +++ /dev/null @@ -1,39 +0,0 @@ -diff --git a/tools/server/server-context.cpp b/tools/server/server-context.cpp -index 39aa20b..d3d5978 100644 ---- a/tools/server/server-context.cpp -+++ b/tools/server/server-context.cpp -@@ -3560,8 +3560,32 @@ private: - // do not checkpoint after mtmd chunks - do_checkpoint = do_checkpoint && !has_mtmd; - -- // no need to create checkpoints that are too close together, unless it's the last user message -- do_checkpoint = do_checkpoint && (slot.prompt.checkpoints.empty() || is_last_user_message || n_tokens_start > slot.prompt.checkpoints.back().n_tokens + params_base.checkpoint_min_step); -+ // recurrent (and hybrid) models cannot partially roll back their state, so the only way to -+ // avoid re-processing an entire multi-turn conversation on the next request is a checkpoint -+ // near the end of the current prompt. without this, a conversation that appends only -+ // assistant/tool messages (agentic tool-calling) re-processes the whole tail every turn, -+ // because no new user-message checkpoint is ever created and the min-step spacing blocks -+ // the near-prompt-end ones. exempt those models' near-end checkpoints from the spacing. -+ const bool is_ckpt_only_rollback = -+ ctx_tgt_seq_rm_type == COMMON_CONTEXT_SEQ_RM_TYPE_FULL || -+ ctx_tgt_seq_rm_type == COMMON_CONTEXT_SEQ_RM_TYPE_RS; -+ -+ // don't create checkpoints too close together, unless it's the last user message or a -+ // near-prompt-end checkpoint for a checkpoint-only-rollback model (leading empty() guards -+ // the checkpoints.back() access via short-circuit) -+ const bool checkpoint_well_spaced = -+ slot.prompt.checkpoints.empty() || -+ is_last_user_message || -+ (near_prompt_end && is_ckpt_only_rollback) || -+ n_tokens_start > slot.prompt.checkpoints.back().n_tokens + params_base.checkpoint_min_step; -+ -+ // and never duplicate the newest checkpoint's position (else the last-user-message -+ // checkpoint is re-created every turn, flooding the list until useful entries are evicted) -+ const bool checkpoint_not_duplicate = -+ slot.prompt.checkpoints.empty() || -+ slot.prompt.checkpoints.back().n_tokens != n_tokens_start; -+ -+ do_checkpoint = do_checkpoint && checkpoint_well_spaced && checkpoint_not_duplicate; - SLT_DBG(slot, "main/do_checkpoint = %s, pos_min = %d, pos_max = %d\n", do_checkpoint ? "yes" : "no", pos_min, pos_max); - - // note: we create the checkpoint before calling llama_decode(), so the current batch is not diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index 8edeaed2..f96884e9 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b9975"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b9981"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b9975-0badc06ab"} — call + * plus the resolved upstream commit, e.g. {@code "b9981-0badc06ab"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b9975"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b9981"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b9975"; + public static final String LLAMA_CPP_VERSION = "b9981"; // Constants holder — not instantiable. private LlamaCppVersion() {} From 45f3ec659f810e8bdf487e2f8051ffaacf8562a8 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 13 Jul 2026 07:18:20 +0000 Subject: [PATCH 2/3] Confirm b9981 full build + ctest (485/485 passing) Update the b9975-b9981 verification row with the real cmake --build + ctest results now that the full native build finished cleanly. --- docs/history/llama-cpp-breaking-changes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 9d2080de..b4c897fd 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -484,4 +484,4 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b9972–b9975 | `ggml/src/ggml-cuda/ggml-cuda.cu` + `ggml/src/gguf.cpp` + `src/llama-kv-cache-dsv4.{cpp,h}` | **Internal-only, no public-API surface** (4 non-test files, ~15 KiB, 3 commits). `ggml-cuda.cu`'s `ggml_backend_cuda_device_get_memory` now handles a failing `cudaMemGetInfo` gracefully (logs + returns `free=total=0`) instead of aborting via `CUDA_CHECK`. `gguf.cpp` rejects an empty GGUF key during parsing (new `HANDCRAFTED_KV_EMPTY_KEY` fuzz case in `tests/test-gguf.cpp`). `llama-kv-cache-dsv4.{cpp,h}` reworks DSV4 compressed-cache clearing to be **per-sequence** instead of whole-cache: `llama_dsv4_comp_state::clear(bool)` → `clear(llama_seq_id, bool)` and `llama_kv_cache_dsv4::clear_compressed(bool)` → `clear_compressed(llama_seq_id, bool)` (a new `-1` sentinel means "all sequences", matching the old behavior); `seq_cp`/`seq_add`/`seq_div` drop their now-unnecessary full-cache clear and `seq_keep` clears every other sequence individually — an isolation fix so removing one sequence's KV state no longer disturbs sibling sequences (new `test_seq_rm_isolated` regression case in `tests/test-save-load-state.cpp`). Both DSV4 functions are `private` members with no `llama.h`/`common.h` surface; not called by project code. No priority-8 header touched, `tools/ui` untouched, no project source changes required. | | b9972–b9975 | upstream verification (sandbox) | All **eight** patches (`0001`–`0008`) re-verified against b9975: applied in filename order onto a clean b9975 checkout via a fresh `cmake -B build` (fail-loud `PATCH_COMMAND`), all clean (the range touches no patch-target file). No OuteTTS generator anchor touched (`tools/tts/tts.cpp` unchanged — the generator extracted `tts.cpp @ b9975`). Full local configure verified (fail-loud patch apply + TTS extraction succeeded); full `cmake --build` + `ctest` verification in progress locally, per-platform confirmation by the CI pipeline. | | b9975–b9981 | `common/common.h` + `tools/mtmd/{mtmd.h,mtmd.cpp,mtmd-cli.cpp}` + `tools/server/{server-chat.cpp,server-common.cpp,server-context.cpp,server-models.cpp}` + `vendor/cpp-httplib/{httplib.h,httplib.cpp}` + `scripts/sync_vendor.py` | **One patch-target conflict, otherwise additive (13 files, ~20 KiB, 6 commits).** `common/common.h`'s `common_prompt_checkpoint` gains an `id_task` field (additive). `mtmd.h`'s `mtmd_input_text` gains a `text_len` field alongside `text` so the tokenizer can carry embedded NUL bytes (`mtmd.cpp`/`mtmd-cli.cpp` updated to match — internal to upstream's own callers, not constructed by project code). `server-chat.cpp`'s Anthropic→OAI tool-result conversion now emits multimodal `content_parts` for `image` blocks (previously text-only) — internal to the Anthropic Messages translation path, not a patch target. `server-common.cpp`'s `process_mtmd_prompt` follows the `mtmd_input_text` field split. `server-models.cpp`'s router capability probe adds `LLAMA_ARG_MMPROJ_AUTO` to the offline-safe allowlist and checks `params.no_mmproj`. `vendor/cpp-httplib` bumped 0.49.0→0.50.1 (additive `Get(path, Params, DownloadProgress)` overloads). **The one patch-target hit:** `tools/server/server-context.cpp`'s `update_slots`/`create_checkpoint` picked up upstream's own fix for the exact problem `0005-server-recurrent-near-prompt-end-checkpoints.patch` addressed — `create_checkpoint` now tracks `id_task` and evicts a checkpoint that sits within `checkpoint_min_step` of a newer one, and `do_checkpoint`'s min-step exemption now covers **every** near-prompt-end checkpoint (`near_prompt_end`, unconditional) rather than only recurrent/hybrid models as `0005` scoped it — a strict superset of `0005`'s behavior. `0005` no longer applies (`patch does not apply` at `server-context.cpp:3560`) and was **dropped** rather than refreshed; see the CLAUDE.md patches-table note under `0005`'s old slot. No other patch-target file (`common/arg.*`, `server.cpp`, `server-models.cpp`'s worker-cmd region) touched; all eight priority-8 headers otherwise byte-identical (only `mtmd.h`, additive). | -| b9975–b9981 | upstream verification (sandbox) | **Seven** patches (`0001`–`0004`, `0006`–`0008`) re-verified against b9981: applied in filename order onto a clean b9981 checkout via a fresh `cmake -B build` (fail-loud `PATCH_COMMAND`), all clean after removing `0005` (superseded — see the row above). No OuteTTS generator anchor touched (`tools/tts/tts.cpp` unchanged — the generator extracted `tts.cpp @ b9981`). Full local configure verified (fail-loud patch apply + TTS extraction succeeded, `ggml commit 34558825a`); full `cmake --build` + `ctest` verification in progress locally, per-platform confirmation by the CI pipeline. | +| b9975–b9981 | upstream verification (sandbox) | **Seven** patches (`0001`–`0004`, `0006`–`0008`) re-verified against b9981: applied in filename order onto a clean b9981 checkout via a fresh `cmake -B build` (fail-loud `PATCH_COMMAND`), all clean after removing `0005` (superseded — see the row above). No OuteTTS generator anchor touched (`tools/tts/tts.cpp` unchanged — the generator extracted `tts.cpp @ b9981`). Full local verification: fresh configure (fail-loud patch apply + TTS extraction succeeded, `ggml commit 34558825a`) + full `cmake --build` (`jllama` + `jllama_test` link cleanly) + `ctest` **485/485 passing**; per-platform confirmation by the CI pipeline. | From 286092c6b31116d5f2a79cd6f674ff7c94c0a72a Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 13 Jul 2026 07:20:28 +0000 Subject: [PATCH 3/3] Upgrade llama.cpp from b9981 to b9982 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Drop patch 0004 (PR#23116 carry): upstream merged it verbatim in this release — server-common.cpp's oaicompat_chat_params_parse gets the same reasoning_budget_tokens/reasoning_budget_message per-request precedence change, plus the same two tests/test-chat.cpp regression cases, so the local patch is now redundant. A fresh native build + full ctest run is in progress to confirm the remaining patches still apply and compile cleanly against b9982; results will follow in a subsequent commit. --- CLAUDE.md | 17 ++- README.md | 2 +- llama/CMakeLists.txt | 4 +- ...-per-request-reasoning-budget-tokens.patch | 131 ------------------ .../llama/value/LlamaCppVersion.java | 8 +- 5 files changed, 19 insertions(+), 143 deletions(-) delete mode 100644 llama/patches/0004-pr23116-server-per-request-reasoning-budget-tokens.patch diff --git a/CLAUDE.md b/CLAUDE.md index db6b7418..5a31fe19 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b9981** +Current llama.cpp pinned version: **b9982** ## Upgrading CUDA Version @@ -421,7 +421,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network; embed.cpp is plain C++17 (no npm) -git clone --depth 1 --branch b9981 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b9982 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build \ && ( cd dist && find . -type f -not -path './_gzip/*' \ | while read -r f; do mkdir -p "_gzip/$(dirname "$f")"; gzip -9 -c "$f" > "_gzip/$f"; done ) \ @@ -461,7 +461,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b9981`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b9982`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -572,7 +572,6 @@ Current patches: | `0001-win32-arg-parse-embed-guard.patch` | Windows JNI regression from llama.cpp **#24779** (introduced b9739): on Windows `common_params_parse` re-derived argv from the **process** command line (`GetCommandLineW`) and adopted it, so an embedded/JNI caller (`java.exe`) lost its `--model …` args → "Failed to parse model parameters". b9789 narrowed the unconditional override to a **count-guard** (`if (static_cast(utf8.buf.size()) == argc) { argv = utf8.ptrs.data(); }`), but that is exactly the variant the project already found breaks its Windows server-integration tests (when the embedded argv length coincides with `java.exe`'s). The patch carries the **complete upstream change** (so it can be submitted to llama.cpp verbatim and then dropped here): **(1)** `common_params_parse` parses **exactly the argv it is given** (no `GetCommandLineW` magic) and a new `common_params_parse_main()` wrapper holds the UTF-8 recovery for the standalone tools' `main()` (`common/arg.{cpp,h}`); **(2)** the **~34 standalone `main()` call sites** (every `common_params_parse(argc, argv, …)` across `tools/*`, `examples/*` and the `tests/*` programs) flip to `common_params_parse_main()`; **(3)** a `tests/test-arg-parser.cpp` regression case pins that `common_params_parse` honors a caller-supplied argv. The embedded caller (`jllama.cpp`) keeps calling `common_params_parse` and is never overridden. **Our subproject build compiles only the `arg.{cpp,h}` core** — `LLAMA_BUILD_TOOLS`/`LLAMA_BUILD_TESTS` are OFF for a FetchContent subproject — so the flips + test are applied-but-not-compiled here; they were validated via a one-off `-DLLAMA_BUILD_TOOLS=ON -DLLAMA_BUILD_TESTS=ON` build (the new test compiles and its asserts pass; `test-arg-parser`'s only red there is the live `ggml.ai` download check, which is sandbox-network, not the patch). Because it spans **37 files** it must be refreshed on every llama.cpp bump (the applier fails loud). | | `0002-server-preserve-caller-load-progress-callback.patch` | Load-progress-callback regression introduced in llama.cpp **b9789**: `server_context::load_model` (`tools/server/server-context.cpp`) now **unconditionally** installs the server's own load-progress reporter on `params_base.load_progress_callback` immediately before `common_init_from_params`, clobbering any callback the embedding caller already set. libjllama's `LoadProgressCallback` feature wires `common_params.load_progress_callback` to a JNI trampoline *before* calling `load_model`, so the bump silently killed it — `LoadProgressCallbackTest` saw zero progress updates and the abort-on-`false` path never threw. The patch guards the assignment with `if (params_base.load_progress_callback == nullptr)`, so the server installs its own reporter **only when the caller hasn't** — a caller-supplied callback survives and fires during load. Standalone `llama-server` (no caller callback, so the field is null) is unaffected. Same JNI-vs-standalone divergence class as `0001`. | | `0003-pr22393-server-add-slot-prompt-similarity-getter-setter.patch` | **Upstream-PR carry** of [ggml-org/llama.cpp#22393](https://github.com/ggml-org/llama.cpp/pull/22393) ("server : add slot_prompt_similarity getter/setter") while it is still open upstream. Purely additive: adds `server_context::get_slot_prompt_similarity()` / `set_slot_prompt_similarity(float)` (`tools/server/server-context.{cpp,h}`) so an embedding/JNI caller can query and tune the slot-selection threshold at runtime without reloading the model. Verbatim copy of the PR — drop it once a pinned `b` includes the change. | -| `0004-pr23116-server-per-request-reasoning-budget-tokens.patch` | **Upstream-PR carry** of [ggml-org/llama.cpp#23116](https://github.com/ggml-org/llama.cpp/pull/23116) ("server: honour per-request reasoning_budget_tokens in chat completions"), motivated by java-llama.cpp#140, while it is still open upstream. `oaicompat_chat_params_parse` (`tools/server/server-common.cpp`) only read the Anthropic `thinking_budget_tokens` alias and always wrote the server-level `reasoning_budget_message`, so a per-request `reasoning_budget_tokens` / `reasoning_budget_message` on a chat-completions request was ignored. The patch reads both overrides **before** the generic copy loop (precedence: `reasoning_budget_tokens` > `thinking_budget_tokens` alias > server default) and threads the per-request message through. Carries the upstream `tests/test-chat.cpp` additions verbatim so the patch is submittable as-is; like `0001`'s test/call-site flips they are **applied-but-not-compiled** here (`LLAMA_BUILD_TESTS` is OFF for the FetchContent subproject). Drop it once a pinned `b` includes the change. | | `0007-server-attach-http-frontend.patch` | **Adds `llama_server_attach(argc, argv, server_context&)`** so the `NativeServer` *attach mode* can serve an **already-loaded `LlamaModel`** over the full upstream HTTP frontend — no second model load, no `start_loop()`; the LlamaModel's worker keeps driving the shared `server_context` and the HTTP routes post tasks to its queue (the queue is the synchronization point). Mechanically: (1) extracts the common route table + CORS-proxy/tools blocks out of `llama_server()` into `llama_server_register_common_routes(...)` (shared verbatim, so the entry points cannot drift; returns `false` on tools-setup failure); (2) adds `llama_server_attach`, which parses only the HTTP-side argv via `common_params_parse`, starts `g_stream_sessions` GC + `server_http_context`, registers the common routes plus the non-router resumable-streaming handlers, marks ready immediately (model already loaded), and blocks on the HTTP thread until `llama_server_request_shutdown()` — never calling `common_init()`, backend init, `ctx_server.terminate()` or `llama_backend_free()` (the embedding caller owns those). Applies after `0001`+`0006` (same file); closes the "NativeServer — reuse an already-loaded LlamaModel" TODO. Upstream-submittable ("server: let embedding callers attach the HTTP frontend to an existing server_context"). | | `0008-server-models-worker-cmd-override.patch` | **Makes router mode usable in-JVM.** The router (`server-models.cpp`) spawns each model worker by re-executing its own binary (`get_server_exec_path()` = `/proc/self/exe` & friends) — inside a JVM that binary is `java`, not a llama-server, so embedded router workers could never start. The patch adds env `LLAMA_SERVER_WORKER_CMD` (whitespace-split; read in `server_model_meta::update_args`) which replaces only the leading binary-path token of the rendered worker args, letting an embedding host relaunch workers through its own bootstrap — e.g. `java -cp app.jar net.ladenthin.llama.server.NativeServer` (each worker is then a fresh JVM running the classic single-model `NativeServer`). Exposed in Java as `NativeServer.setWorkerCommand(String...)` (JNI `setenv`); exercised by `RouterModeIntegrationTest` (Linux CI). Upstream-submittable (also useful for containerized/wrapped deployments). | | `0006-server-embed-native-server-jni.patch` | **Makes `server.cpp`'s `llama_server` embeddable in the JVM** so the `NativeServer` JNI bridge can run the full upstream HTTP server (WebUI included) inside `libjllama` — see "Two server modes" below. b9870 already exposes `int llama_server(int, char**)` (non-static; no `main` in the file), so the patch only adds embedded-mode support: (1) a `g_llama_server_embedded` flag + `llama_server_set_embedded()` / `llama_server_request_shutdown()` (declared in the committed `src/main/cpp/native_server_bridge.h`); (2) skips installing the process-wide SIGINT/SIGTERM handlers when embedded (they would hijack the JVM's); (3) in embedded mode parses the **forwarded** argv via `common_params_parse` instead of `common_params_parse_main` (whose `GetCommandLineW` recovery would pick up `java.exe`'s command line — the same Windows class of bug `0001` fixes). `llama_server_request_shutdown()` mirrors the SIGTERM path (invokes the installed `shutdown_handler` → `ctx_server.terminate()` unblocks `start_loop()`), giving JNI an out-of-band stop since `ctx_server` is loop-local. Applies **after `0001`** (which flips this call site to `common_params_parse_main`), so its context is the post-`0001` tree; regenerate against `0001`+source on a bump. Only touches `tools/server/server.cpp`. | @@ -588,6 +587,14 @@ cleanly with the patch removed and needs no replacement. If a regression in agen prefill behavior shows up, re-check `tools/server/server-context.cpp`'s `create_checkpoint` / `do_checkpoint` logic against this description before reintroducing a local patch. +**`0004` was dropped at the b9982 bump.** Upstream merged +[ggml-org/llama.cpp#23116](https://github.com/ggml-org/llama.cpp/pull/23116) verbatim: the +`b9981...b9982` diff carries the exact same `oaicompat_chat_params_parse` precedence change +(`reasoning_budget_tokens` > `thinking_budget_tokens` alias > server default, plus the per-request +`reasoning_budget_message` override) and the same two `tests/test-chat.cpp` regression cases +(`test_reasoning_budget_tokens_per_request` / `test_reasoning_budget_message_per_request`, +byte-identical body). No local patch needed — the tree already matches what `0004` used to add. + ## OuteTTS build-time extraction (`cmake/generate-tts-upstream.cmake`) The `TextToSpeech` native pipeline reuses llama.cpp's OuteTTS helpers (`tools/tts/tts.cpp`) @@ -1247,7 +1254,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b9981`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b9982`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index e743dc85..8fa9ddaf 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b9981](https://img.shields.io/badge/llama.cpp-%23b9981-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b9981) +[![llama.cpp b9982](https://img.shields.io/badge/llama.cpp-%23b9982-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b9982) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index 0e0cf990..a9a142c6 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b9981 + GIT_TAG b9982 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= @@ -196,7 +196,7 @@ execute_process( COMMAND ${CMAKE_COMMAND} -DTTS_SRC=${llama.cpp_SOURCE_DIR}/tools/tts/tts.cpp -DOUT_CPP=${JLLAMA_TTS_GEN_CPP} - -DLLAMA_TAG=b9981 + -DLLAMA_TAG=b9982 -P ${CMAKE_CURRENT_SOURCE_DIR}/cmake/generate-tts-upstream.cmake RESULT_VARIABLE JLLAMA_TTS_GEN_RESULT ) diff --git a/llama/patches/0004-pr23116-server-per-request-reasoning-budget-tokens.patch b/llama/patches/0004-pr23116-server-per-request-reasoning-budget-tokens.patch deleted file mode 100644 index 0ac4f3b1..00000000 --- a/llama/patches/0004-pr23116-server-per-request-reasoning-budget-tokens.patch +++ /dev/null @@ -1,131 +0,0 @@ -Upstream PR: ggml-org/llama.cpp#23116 — "server: honour per-request reasoning_budget_tokens in -chat completions" -https://github.com/ggml-org/llama.cpp/pull/23116 - -Carried locally until the PR is merged upstream. Motivated by java-llama.cpp#140: a per-request -`reasoning_budget_tokens` (and `reasoning_budget_message`) sent on a chat-completions request must -override the server-launch default. Upstream `oaicompat_chat_params_parse` only read the Anthropic -`thinking_budget_tokens` alias and always wrote the server-level `reasoning_budget_message`, so the -canonical per-request keys were ignored. The patch reads both overrides before the generic copy loop -(precedence: reasoning_budget_tokens > thinking_budget_tokens alias > server default) and threads the -per-request message through. Includes the upstream test additions (tests/test-chat.cpp) verbatim so -the patch is submittable as-is; LLAMA_BUILD_TESTS is OFF for the FetchContent subproject, so those are -applied-but-not-compiled here. Refresh against the new source on every llama.cpp version bump (the -applier fails loud if the context shifts). - -diff --git a/tests/test-chat.cpp b/tests/test-chat.cpp -index c38aed8..dfa8006 100644 ---- a/tests/test-chat.cpp -+++ b/tests/test-chat.cpp -@@ -5780,6 +5780,71 @@ static void test_developer_role_to_system_workaround() { - } - } - -+static void test_reasoning_budget_tokens_per_request() { -+ LOG_DBG("%s\n", __func__); -+ // Use Qwen3 template which has ... reasoning markers. -+ // The autoparser detects them and sets thinking_start/end_tag, which enables -+ // the reasoning-budget code path in oaicompat_chat_params_parse. -+ auto tmpls = read_templates("models/templates/Qwen-Qwen3-0.6B.jinja"); -+ -+ server_chat_params opt; -+ opt.tmpls = std::move(tmpls); -+ opt.use_jinja = true; -+ opt.enable_thinking = true; -+ opt.reasoning_budget = -1; -+ opt.reasoning_format = COMMON_REASONING_FORMAT_NONE; -+ -+ // Body with per-request reasoning_budget_tokens=0 (suppress thinking). -+ json body = { -+ {"messages", json::array({json{{"role", "user"}, {"content", "hello"}}})}, -+ {"reasoning_budget_tokens", 0}, -+ }; -+ std::vector out_files; -+ auto llama_params = oaicompat_chat_params_parse(body, opt, out_files); -+ -+ // The per-request value must win over the server default (-1). -+ if (!llama_params.contains("reasoning_budget_tokens")) { -+ throw std::runtime_error("reasoning_budget_tokens missing from llama_params (thinking_end_tag may be empty for this template)"); -+ } -+ int got = llama_params["reasoning_budget_tokens"].get(); -+ if (got != 0) { -+ throw std::runtime_error(std::string("Expected reasoning_budget_tokens=0, got ") + std::to_string(got)); -+ } -+} -+ -+static void test_reasoning_budget_message_per_request() { -+ LOG_DBG("%s\n", __func__); -+ // Same code path as test_reasoning_budget_tokens_per_request: the Qwen3 template's -+ // ... markers enable the reasoning-budget block in oaicompat_chat_params_parse. -+ auto tmpls = read_templates("models/templates/Qwen-Qwen3-0.6B.jinja"); -+ -+ server_chat_params opt; -+ opt.tmpls = std::move(tmpls); -+ opt.use_jinja = true; -+ opt.enable_thinking = true; -+ opt.reasoning_budget = -1; -+ opt.reasoning_format = COMMON_REASONING_FORMAT_NONE; -+ opt.reasoning_budget_message = "server default"; -+ -+ // Body with a per-request reasoning_budget_message override. -+ const std::string per_request_message = "per-request message"; -+ json body = { -+ {"messages", json::array({json{{"role", "user"}, {"content", "hello"}}})}, -+ {"reasoning_budget_message", per_request_message}, -+ }; -+ std::vector out_files; -+ auto llama_params = oaicompat_chat_params_parse(body, opt, out_files); -+ -+ // The per-request value must win over the server default. -+ if (!llama_params.contains("reasoning_budget_message")) { -+ throw std::runtime_error("reasoning_budget_message missing from llama_params (thinking_end_tag may be empty for this template)"); -+ } -+ std::string got = llama_params["reasoning_budget_message"].get(); -+ if (got != per_request_message) { -+ throw std::runtime_error("Expected reasoning_budget_message='" + per_request_message + "', got '" + got + "'"); -+ } -+} -+ - static void test_msg_diffs_compute() { - LOG_DBG("%s\n", __func__); - { -@@ -5937,6 +6002,8 @@ int main(int argc, char ** argv) { - test_convert_responses_to_chatcmpl(); - test_developer_role_to_system_workaround(); - test_template_generation_prompt(); -+ test_reasoning_budget_tokens_per_request(); -+ test_reasoning_budget_message_per_request(); - test_template_output_peg_parsers(detailed_debug); - std::cout << "\n[chat] All tests passed!" << '\n'; - } -diff --git a/tools/server/server-common.cpp b/tools/server/server-common.cpp -index ac291d3..26cdfd2 100644 ---- a/tools/server/server-common.cpp -+++ b/tools/server/server-common.cpp -@@ -1116,16 +1116,24 @@ json oaicompat_chat_params_parse( - - // Reasoning budget: pass parameters through to sampling layer - { -- int reasoning_budget = json_value(body, "thinking_budget_tokens", -1); -+ // Per-request overrides, read before writing to llama_params so the generic copy -+ // loop (which skips keys already present) won't clobber the caller-supplied values. -+ // Precedence: canonical reasoning_budget_tokens > Anthropic thinking_budget_tokens -+ // alias > server-level default. -+ int reasoning_budget = json_value(body, "reasoning_budget_tokens", -1); -+ if (reasoning_budget == -1) { -+ reasoning_budget = json_value(body, "thinking_budget_tokens", -1); -+ } - if (reasoning_budget == -1) { - reasoning_budget = opt.reasoning_budget; - } -+ std::string reasoning_budget_message = json_value(body, "reasoning_budget_message", opt.reasoning_budget_message); - - if (!chat_params.thinking_end_tag.empty()) { - llama_params["reasoning_budget_tokens"] = reasoning_budget; - llama_params["reasoning_budget_start_tag"] = chat_params.thinking_start_tag; - llama_params["reasoning_budget_end_tag"] = chat_params.thinking_end_tag; -- llama_params["reasoning_budget_message"] = opt.reasoning_budget_message; -+ llama_params["reasoning_budget_message"] = reasoning_budget_message; - llama_params["reasoning_control"] = json_value(body, "reasoning_control", false); - } - } diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index f96884e9..729726d0 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b9981"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b9982"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b9981-0badc06ab"} — call + * plus the resolved upstream commit, e.g. {@code "b9982-0badc06ab"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b9981"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b9982"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b9981"; + public static final String LLAMA_CPP_VERSION = "b9982"; // Constants holder — not instantiable. private LlamaCppVersion() {}