diff --git a/llamacppllmbasev3/Chart.yaml b/llamacppllmbasev3/Chart.yaml index a3f6cdcd1..53c28ebb3 100644 --- a/llamacppllmbasev3/Chart.yaml +++ b/llamacppllmbasev3/Chart.yaml @@ -1,6 +1,6 @@ apiVersion: v2 -appVersion: b10588 +appVersion: b10969 description: Generic llama.cpp + llm-init base; the GGUF model is supplied at install time via env name: llamacppllmbasev3 type: application -version: 1.2.37 \ No newline at end of file +version: 1.2.38 \ No newline at end of file diff --git a/llamacppllmbasev3/OlaresManifest.yaml b/llamacppllmbasev3/OlaresManifest.yaml index 01bf01017..2e43bef8e 100644 --- a/llamacppllmbasev3/OlaresManifest.yaml +++ b/llamacppllmbasev3/OlaresManifest.yaml @@ -7,7 +7,7 @@ metadata: description: "Generic llama.cpp engine base. Pick any GGUF model at install via env." appid: llamacppllmbasev3 title: llama.cpp Engine Base - version: '1.2.37' + version: '1.2.38' categories: - AI - models @@ -36,37 +36,30 @@ workloadReplicas: llminit: 1 spec: onlyAdmin: true - versionName: 'b10588' + versionName: 'b10969' upgradeDescription: | - Chart 1.2.37 — bump engine to `server-cuda12-b10588`. + Chart 1.2.38 — Upgrade llama.cpp from `server-cuda12-b10588` to `server-cuda12-b10969`. - **Images** - - llama.cpp: `docker.io/beclab/ggml-org-llama.cpp:server-cuda12-b10454` → `server-cuda12-b10588` (amd64 + arm64). - - llm-init: unchanged at `v1.5.0`. + **Model compatibility** + - Add Qwen3.8-Flash-Next (`qwen4exp`) text/vision architecture support ([#27742](https://github.com/ggml-org/llama.cpp/pull/27742)), with subsequent recurrent-state rollback and inference fixes. + - Add NVIDIA Nemotron-3-Puzzle-75B-A9B ([#25444](https://github.com/ggml-org/llama.cpp/pull/25444)) and Tencent Hy 4 preview architecture support ([#28127](https://github.com/ggml-org/llama.cpp/pull/28127)). These require matching model weights and sufficient resources. - 134 upstream commits; core libraries move to llama.cpp 0.2.0 / ggml 0.21.0. + **Speculative decoding and CUDA** + - Add DFlash2 draft-model support ([#27342](https://github.com/ggml-org/llama.cpp/pull/27342)); requires a compatible draft checkpoint. Upstream notes that DFlash2 GGUF files converted before August 27, 2026 need reconversion for vision correctness. + - Add CUDA sparse Flash Attention for DeepSeek-V4/GLM ([#27970](https://github.com/ggml-org/llama.cpp/pull/27970)), extend MoE fusion to speculative decoding ([#27621](https://github.com/ggml-org/llama.cpp/pull/27621)), and optimize Q4_K/Q5_K decoding with DGX Spark prefetch improvements ([#26705](https://github.com/ggml-org/llama.cpp/pull/26705)). Gains depend on model and hardware. - **New model architectures** — clone this base and point `MODEL_SOURCE` / `MODEL_NAME` at the GGUF: - - BailingMoE3 ([#26608](https://github.com/ggml-org/llama.cpp/pull/26608)), with DSpark speculative decode ([#27508](https://github.com/ggml-org/llama.cpp/pull/27508)) - - Granite SWA / GraniteMoE SWA ([#25505](https://github.com/ggml-org/llama.cpp/pull/25505)) - - dots3-note, including vision + audio through `mmproj` ([#27060](https://github.com/ggml-org/llama.cpp/pull/27060), [#27524](https://github.com/ggml-org/llama.cpp/pull/27524)) - - DSpark drafters for LFM2 ([#27383](https://github.com/ggml-org/llama.cpp/pull/27383)) and speculators-format checkpoints ([#26275](https://github.com/ggml-org/llama.cpp/pull/26275)) - - Nemotron 3 Ultra conversion fix ([#27101](https://github.com/ggml-org/llama.cpp/pull/27101)) + **Server and chat behavior** + - Add `--kv-unified-per-slot` to cap each request slot when using a unified KV cache; `-c` controls the total pool ([#24124](https://github.com/ggml-org/llama.cpp/pull/24124)). This update does not change the configured context size automatically. + - Enable `preserve_reasoning` by default for chat templates that support it; use `--no-reasoning-preserve` to disable ([#28174](https://github.com/ggml-org/llama.cpp/pull/28174)). + - Accept `data:` URLs in `input_audio` and `input_video` requests, fixing media decoding errors ([#27735](https://github.com/ggml-org/llama.cpp/pull/27735)); requires a compatible multimodal model and client. - **GPU and performance** - - DGX Spark (`nvidia-gb10`): faster batch-size-1 decode for dense models via MMVQ `nwarps=8` ([#26843](https://github.com/ggml-org/llama.cpp/pull/26843)) - - CUDA MMVQ→MMQ decode crossover is now tuned per GPU and quant type ([#26079](https://github.com/ggml-org/llama.cpp/pull/26079)); cuBLAS handles use a static workspace ([#26574](https://github.com/ggml-org/llama.cpp/pull/26574)) - - Layer auto-fit accounts for `n_streams` ([#27496](https://github.com/ggml-org/llama.cpp/pull/27496)); V is built as a view of K in iSWA attention ([#27392](https://github.com/ggml-org/llama.cpp/pull/27392)) - - Fixes: backend split scheduler race ([#26040](https://github.com/ggml-org/llama.cpp/pull/26040)) and a null-pointer deref in speculative decode ([#27404](https://github.com/ggml-org/llama.cpp/pull/27404)) + **Stability and memory** + - Reduce peak RAM during model loading when tensor repacking is enabled ([#27483](https://github.com/ggml-org/llama.cpp/pull/27483)). + - Fix the model-router LRU hang with concurrent requests ([#28539](https://github.com/ggml-org/llama.cpp/pull/28539)), speculative decoding after image input ([#28715](https://github.com/ggml-org/llama.cpp/pull/28715)), and CUDA Flash Attention synchronization ([#27870](https://github.com/ggml-org/llama.cpp/pull/27870)). - **Server and API** - - `/metrics` and `/slots` stay reachable while the server is sleeping ([#27376](https://github.com/ggml-org/llama.cpp/pull/27376)) - - Model endpoints become private when `--api-key` authentication is enabled ([#26347](https://github.com/ggml-org/llama.cpp/pull/26347)); this chart runs without an engine API key, so the internal entrance is unaffected - - `--mmproj-device` places the vision projector on a chosen device ([#23255](https://github.com/ggml-org/llama.cpp/pull/23255)); WebP inputs are accepted via ffmpeg ([#27520](https://github.com/ggml-org/llama.cpp/pull/27520)) - - Embedding output combined with draft-MTP is fixed ([#27400](https://github.com/ggml-org/llama.cpp/pull/27400)) — relevant to `MODEL_MODE=embedding` - - mtmd fixes for LFM2 image tiling, DeepSeek-OCR and Granite preprocessing; refreshed built-in WebUI + These are upstream engine changes; model-specific capabilities are not all validated by this chart update. llm-init remains at `v1.5.0`. + Full upstream changes: https://github.com/ggml-org/llama.cpp/compare/b10588...b10969 - See https://github.com/ggml-org/llama.cpp/compare/b10454...b10588 for the full upstream diff. fullDescription: | **IMPORTANT NOTE** This app is a template and cannot be used on its own. To use it, set the environment variables below to connect a specific model. diff --git a/llamacppllmbasev3/i18n/en-US/OlaresManifest.yaml b/llamacppllmbasev3/i18n/en-US/OlaresManifest.yaml index 835e2e842..6dac468a5 100644 --- a/llamacppllmbasev3/i18n/en-US/OlaresManifest.yaml +++ b/llamacppllmbasev3/i18n/en-US/OlaresManifest.yaml @@ -4,25 +4,28 @@ metadata: spec: upgradeDescription: | - Chart 1.2.33 — bump engine to `server-cuda12-b10454` and llm-init to `v1.3.8`. + Chart 1.2.38 — Upgrade llama.cpp from `server-cuda12-b10588` to `server-cuda12-b10969`. - **Images** - - llama.cpp: `docker.io/beclab/ggml-org-llama.cpp:server-cuda12-b10362` → `server-cuda12-b10454` (latest published beclab CUDA 12 server tag; amd64 + arm64). Upstream GitHub is already at b10456; no beclab b10455/b10456 image yet. - - llm-init: `docker.io/beclab/llm-init:v1.3.5` → `v1.3.8`. + **Model compatibility** + - Add Qwen3.8-Flash-Next (`qwen4exp`) text/vision architecture support ([#27742](https://github.com/ggml-org/llama.cpp/pull/27742)), with subsequent recurrent-state rollback and inference fixes. + - Add NVIDIA Nemotron-3-Puzzle-75B-A9B ([#25444](https://github.com/ggml-org/llama.cpp/pull/25444)) and Tencent Hy 4 preview architecture support ([#28127](https://github.com/ggml-org/llama.cpp/pull/28127)). These require matching model weights and sufficient resources. - **Muse Glimmer 30B (Meta)** — requires **b10353+**; **b10454 includes** the `muse-glimmer` architecture ([llama.cpp #26841](https://github.com/ggml-org/llama.cpp/pull/26841)). Older builds (e.g. b10331) fail with `unknown model architecture: 'muse-glimmer'`. + **Speculative decoding and CUDA** + - Add DFlash2 draft-model support ([#27342](https://github.com/ggml-org/llama.cpp/pull/27342)); requires a compatible draft checkpoint. Upstream notes that DFlash2 GGUF files converted before August 27, 2026 need reconversion for vision correctness. + - Add CUDA sparse Flash Attention for DeepSeek-V4/GLM ([#27970](https://github.com/ggml-org/llama.cpp/pull/27970)), extend MoE fusion to speculative decoding ([#27621](https://github.com/ggml-org/llama.cpp/pull/27621)), and optimize Q4_K/Q5_K decoding with DGX Spark prefetch improvements ([#26705](https://github.com/ggml-org/llama.cpp/pull/26705)). Gains depend on model and hardware. - Clone this base and point env at [meta-models/Muse-Glimmer-30B-GGUF](https://huggingface.co/meta-models/Muse-Glimmer-30B-GGUF) or community quants (e.g. unsloth). Typical layout: - - Main GGUF + `mmproj` in `MODEL_SOURCE` (comma-separated) for vision - - DFlash drafter (`dflash-kquant.gguf`) + `--spec-type draft-dflash` or `draft-dspark` in `ENGINE_ARGS` for speculative decode + **Server and chat behavior** + - Add `--kv-unified-per-slot` to cap each request slot when using a unified KV cache; `-c` controls the total pool ([#24124](https://github.com/ggml-org/llama.cpp/pull/24124)). This update does not change the configured context size automatically. + - Enable `preserve_reasoning` by default for chat templates that support it; use `--no-reasoning-preserve` to disable ([#28174](https://github.com/ggml-org/llama.cpp/pull/28174)). + - Accept `data:` URLs in `input_audio` and `input_video` requests, fixing media decoding errors ([#27735](https://github.com/ggml-org/llama.cpp/pull/27735)); requires a compatible multimodal model and client. - **Also in b10362 → b10454:** - - Chat templates receive OpenAI `reasoning_effort` ([llama.cpp #26941](https://github.com/ggml-org/llama.cpp/pull/26941)); Qwen3.8 uses `xhigh` / `medium` / `low` via `--chat-template-kwargs` or the request body - - MTP draft type auto-detect; DFlash / DSpark backend sampling - - MiniMax Text01 / M1 conversion support - - Server `/metrics` and `/slots` stay available during decode; mtmd and jinja fixes + **Stability and memory** + - Reduce peak RAM during model loading when tensor repacking is enabled ([#27483](https://github.com/ggml-org/llama.cpp/pull/27483)). + - Fix the model-router LRU hang with concurrent requests ([#28539](https://github.com/ggml-org/llama.cpp/pull/28539)), speculative decoding after image input ([#28715](https://github.com/ggml-org/llama.cpp/pull/28715)), and CUDA Flash Attention synchronization ([#27870](https://github.com/ggml-org/llama.cpp/pull/27870)). + + These are upstream engine changes; model-specific capabilities are not all validated by this chart update. llm-init remains at `v1.5.0`. + Full upstream changes: https://github.com/ggml-org/llama.cpp/compare/b10588...b10969 - See https://github.com/ggml-org/llama.cpp/releases for full upstream release notes. fullDescription: | **IMPORTANT NOTE** This app is a template and cannot be used on its own. To use it, set the environment variables below to connect a specific model. diff --git a/llamacppllmbasev3/i18n/zh-CN/OlaresManifest.yaml b/llamacppllmbasev3/i18n/zh-CN/OlaresManifest.yaml index 666700a95..b6eaeaab1 100644 --- a/llamacppllmbasev3/i18n/zh-CN/OlaresManifest.yaml +++ b/llamacppllmbasev3/i18n/zh-CN/OlaresManifest.yaml @@ -4,25 +4,28 @@ metadata: spec: upgradeDescription: | - Chart 1.2.33 — 引擎升级到 `server-cuda12-b10454`,llm-init 升级到 `v1.3.8`。 + Chart 1.2.38 — llama.cpp 镜像从 `server-cuda12-b10588` 升级到 `server-cuda12-b10969`。 - **镜像** - - llama.cpp:`docker.io/beclab/ggml-org-llama.cpp:server-cuda12-b10362` → `server-cuda12-b10454`(当前已发布的最新 beclab CUDA 12 server 标签;amd64 + arm64)。上游 GitHub 已到 b10456;尚无 beclab 的 b10455/b10456 镜像。 - - llm-init:`docker.io/beclab/llm-init:v1.3.5` → `v1.3.8`。 + **模型兼容性** + - 新增 Qwen3.8-Flash-Next(`qwen4exp`)文本与视觉架构支持([#27742](https://github.com/ggml-org/llama.cpp/pull/27742)),并包含后续循环状态回滚和推理修复。 + - 新增 NVIDIA Nemotron-3-Puzzle-75B-A9B([#25444](https://github.com/ggml-org/llama.cpp/pull/25444))和 Tencent Hy 4 预览架构支持([#28127](https://github.com/ggml-org/llama.cpp/pull/28127));使用时需准备匹配的模型权重和足够的资源。 - **Muse Glimmer 30B(Meta)** — 需要 **b10353+**;**b10454 已包含** `muse-glimmer` 架构([llama.cpp #26841](https://github.com/ggml-org/llama.cpp/pull/26841))。更旧的构建(如 b10331)会报 `unknown model architecture: 'muse-glimmer'`。 + **推测解码与 CUDA** + - 新增 DFlash2 草稿模型支持([#27342](https://github.com/ggml-org/llama.cpp/pull/27342)),需配置兼容的草稿权重。上游提示:2026 年 8 月 27 日前转换的 DFlash2 GGUF 文件需重新转换,以保证视觉功能正确。 + - 新增 DeepSeek-V4/GLM 的 CUDA 稀疏 Flash Attention([#27970](https://github.com/ggml-org/llama.cpp/pull/27970)),将 MoE 融合优化扩展到推测解码([#27621](https://github.com/ggml-org/llama.cpp/pull/27621)),优化 Q4_K/Q5_K 解码和 DGX Spark 预取([#26705](https://github.com/ggml-org/llama.cpp/pull/26705))。实际收益取决于模型与硬件。 - 克隆此基座后,将环境变量指向 [meta-models/Muse-Glimmer-30B-GGUF](https://huggingface.co/meta-models/Muse-Glimmer-30B-GGUF) 或社区量化(如 unsloth)。常见布局: - - 主 GGUF + `mmproj` 写入 `MODEL_SOURCE`(逗号分隔)以支持视觉 - - DFlash 草稿模型(`dflash-kquant.gguf`)并在 `ENGINE_ARGS` 中加 `--spec-type draft-dflash` 或 `draft-dspark` 以开启投机解码 + **服务接口与对话行为** + - 新增 `--kv-unified-per-slot`,可在统一 KV 缓存模式下限制每个请求槽的上下文,`-c` 控制总缓存池([#24124](https://github.com/ggml-org/llama.cpp/pull/24124))。本次升级不会自动修改实例的上下文配置。 + - 对支持的聊天模板,默认启用 `preserve_reasoning`;可通过 `--no-reasoning-preserve` 关闭([#28174](https://github.com/ggml-org/llama.cpp/pull/28174))。 + - `input_audio`、`input_video` 请求支持 `data:` URL,修复媒体解码错误([#27735](https://github.com/ggml-org/llama.cpp/pull/27735));需使用兼容的多模态模型与客户端。 - **b10362 → b10454 其它变化:** - - 聊天模板可接收 OpenAI `reasoning_effort`([llama.cpp #26941](https://github.com/ggml-org/llama.cpp/pull/26941));Qwen3.8 可用 `--chat-template-kwargs` 或请求体设置 `xhigh` / `medium` / `low` - - MTP 草稿类型自动检测;DFlash / DSpark 后端采样 - - 支持 MiniMax Text01 / M1 转换 - - 解码过程中 `/metrics` 与 `/slots` 仍可用;mtmd 与 jinja 修复 + **稳定性与内存** + - 降低启用张量重排时模型加载阶段的内存峰值([#27483](https://github.com/ggml-org/llama.cpp/pull/27483))。 + - 修复模型路由器并发请求的 LRU 卡住问题([#28539](https://github.com/ggml-org/llama.cpp/pull/28539))、图片输入后的推测解码([#28715](https://github.com/ggml-org/llama.cpp/pull/28715))和 CUDA Flash Attention 同步问题([#27870](https://github.com/ggml-org/llama.cpp/pull/27870))。 + + 以上为上游引擎变化,各模型专属能力尚未全部由本次 Chart 升级验证。llm-init 保持 `v1.5.0`。 + 完整上游变更:https://github.com/ggml-org/llama.cpp/compare/b10588...b10969 - 完整上游发布说明见 https://github.com/ggml-org/llama.cpp/releases fullDescription: | **重要提示** 此应用是一个模板,不能单独使用。请按下方环境变量设置模型后,方可使用本应用提供模型服务。 diff --git a/llamacppllmbasev3/templates/llamacpp.yaml b/llamacppllmbasev3/templates/llamacpp.yaml index 7640a301f..f530357ca 100644 --- a/llamacppllmbasev3/templates/llamacpp.yaml +++ b/llamacppllmbasev3/templates/llamacpp.yaml @@ -73,7 +73,7 @@ spec: defaultMode: 0555 containers: - name: llamacpp - image: docker.io/beclab/ggml-org-llama.cpp:server-cuda12-b10588 + image: docker.io/beclab/ggml-org-llama.cpp:server-cuda12-b10969 imagePullPolicy: IfNotPresent command: ["/bin/sh", "/llm-init/wrappers/llamacpp.sh"] env: