From ab5cd96172a10be2e7bfeccec97ddb3677ad9e0e Mon Sep 17 00:00:00 2001 From: Harvey Date: Tue, 15 Sep 2026 12:06:55 +0800 Subject: [PATCH 1/3] Update llamacppllmbasev3 to tested 1.2.47 with CUDA b10969 --- llamacppllmbasev3/Chart.yaml | 4 +- llamacppllmbasev3/OlaresManifest.yaml | 82 +++++++++------- .../i18n/de-DE/OlaresManifest.yaml | 40 ++++++++ .../i18n/en-US/OlaresManifest.yaml | 24 +---- .../i18n/es-ES/OlaresManifest.yaml | 40 ++++++++ .../i18n/fr-FR/OlaresManifest.yaml | 40 ++++++++ .../i18n/it-IT/OlaresManifest.yaml | 40 ++++++++ .../i18n/ja-JP/OlaresManifest.yaml | 40 ++++++++ .../i18n/zh-CN/OlaresManifest.yaml | 24 +---- llamacppllmbasev3/templates/_helpers.tpl | 50 ++++++++++ llamacppllmbasev3/templates/llamacpp.yaml | 97 ++++++++++++++++--- llamacppllmbasev3/templates/llm-init.yaml | 4 +- llamacppllmbasev3/values.yaml | 10 ++ 13 files changed, 402 insertions(+), 93 deletions(-) create mode 100644 llamacppllmbasev3/i18n/de-DE/OlaresManifest.yaml create mode 100644 llamacppllmbasev3/i18n/es-ES/OlaresManifest.yaml create mode 100644 llamacppllmbasev3/i18n/fr-FR/OlaresManifest.yaml create mode 100644 llamacppllmbasev3/i18n/it-IT/OlaresManifest.yaml create mode 100644 llamacppllmbasev3/i18n/ja-JP/OlaresManifest.yaml diff --git a/llamacppllmbasev3/Chart.yaml b/llamacppllmbasev3/Chart.yaml index a3f6cdcd1..f9d32c363 100644 --- a/llamacppllmbasev3/Chart.yaml +++ b/llamacppllmbasev3/Chart.yaml @@ -1,6 +1,6 @@ apiVersion: v2 -appVersion: b10588 +appVersion: b10969 description: Generic llama.cpp + llm-init base; the GGUF model is supplied at install time via env name: llamacppllmbasev3 type: application -version: 1.2.37 \ No newline at end of file +version: 1.2.47 diff --git a/llamacppllmbasev3/OlaresManifest.yaml b/llamacppllmbasev3/OlaresManifest.yaml index 01bf01017..3caa50aeb 100644 --- a/llamacppllmbasev3/OlaresManifest.yaml +++ b/llamacppllmbasev3/OlaresManifest.yaml @@ -7,7 +7,7 @@ metadata: description: "Generic llama.cpp engine base. Pick any GGUF model at install via env." appid: llamacppllmbasev3 title: llama.cpp Engine Base - version: '1.2.37' + version: '1.2.47' categories: - AI - models @@ -36,37 +36,10 @@ workloadReplicas: llminit: 1 spec: onlyAdmin: true - versionName: 'b10588' + versionName: 'b10969 / b10731-rocm / b10752-intel / b10884-intel-iGPU' upgradeDescription: | - Chart 1.2.37 — bump engine to `server-cuda12-b10588`. + Chart 1.2.47 — Upgrade the NVIDIA / Spark CUDA image from server-cuda12-b10752 to server-cuda12-b10969. - **Images** - - llama.cpp: `docker.io/beclab/ggml-org-llama.cpp:server-cuda12-b10454` → `server-cuda12-b10588` (amd64 + arm64). - - llm-init: unchanged at `v1.5.0`. - - 134 upstream commits; core libraries move to llama.cpp 0.2.0 / ggml 0.21.0. - - **New model architectures** — clone this base and point `MODEL_SOURCE` / `MODEL_NAME` at the GGUF: - - BailingMoE3 ([#26608](https://github.com/ggml-org/llama.cpp/pull/26608)), with DSpark speculative decode ([#27508](https://github.com/ggml-org/llama.cpp/pull/27508)) - - Granite SWA / GraniteMoE SWA ([#25505](https://github.com/ggml-org/llama.cpp/pull/25505)) - - dots3-note, including vision + audio through `mmproj` ([#27060](https://github.com/ggml-org/llama.cpp/pull/27060), [#27524](https://github.com/ggml-org/llama.cpp/pull/27524)) - - DSpark drafters for LFM2 ([#27383](https://github.com/ggml-org/llama.cpp/pull/27383)) and speculators-format checkpoints ([#26275](https://github.com/ggml-org/llama.cpp/pull/26275)) - - Nemotron 3 Ultra conversion fix ([#27101](https://github.com/ggml-org/llama.cpp/pull/27101)) - - **GPU and performance** - - DGX Spark (`nvidia-gb10`): faster batch-size-1 decode for dense models via MMVQ `nwarps=8` ([#26843](https://github.com/ggml-org/llama.cpp/pull/26843)) - - CUDA MMVQ→MMQ decode crossover is now tuned per GPU and quant type ([#26079](https://github.com/ggml-org/llama.cpp/pull/26079)); cuBLAS handles use a static workspace ([#26574](https://github.com/ggml-org/llama.cpp/pull/26574)) - - Layer auto-fit accounts for `n_streams` ([#27496](https://github.com/ggml-org/llama.cpp/pull/27496)); V is built as a view of K in iSWA attention ([#27392](https://github.com/ggml-org/llama.cpp/pull/27392)) - - Fixes: backend split scheduler race ([#26040](https://github.com/ggml-org/llama.cpp/pull/26040)) and a null-pointer deref in speculative decode ([#27404](https://github.com/ggml-org/llama.cpp/pull/27404)) - - **Server and API** - - `/metrics` and `/slots` stay reachable while the server is sleeping ([#27376](https://github.com/ggml-org/llama.cpp/pull/27376)) - - Model endpoints become private when `--api-key` authentication is enabled ([#26347](https://github.com/ggml-org/llama.cpp/pull/26347)); this chart runs without an engine API key, so the internal entrance is unaffected - - `--mmproj-device` places the vision projector on a chosen device ([#23255](https://github.com/ggml-org/llama.cpp/pull/23255)); WebP inputs are accepted via ffmpeg ([#27520](https://github.com/ggml-org/llama.cpp/pull/27520)) - - Embedding output combined with draft-MTP is fixed ([#27400](https://github.com/ggml-org/llama.cpp/pull/27400)) — relevant to `MODEL_MODE=embedding` - - mtmd fixes for LFM2 image tiling, DeepSeek-OCR and Granite preprocessing; refreshed built-in WebUI - - See https://github.com/ggml-org/llama.cpp/compare/b10454...b10588 for the full upstream diff. fullDescription: | **IMPORTANT NOTE** This app is a template and cannot be used on its own. To use it, set the environment variables below to connect a specific model. @@ -78,7 +51,7 @@ spec: - `MODEL_SUPPORTS`: Model capability options (Vision / Tools / Thinking / None). Pick what your model supports. This is required. - `ENGINE_ARGS`: `-c 8192 -ngl all -fa on`. Extra llama.cpp options, like `-c 8192 -ngl all -fa on`. To fully use GPU, add `-ngl all`. You can also use values like `LLAMA_ARG_*=val`. - `LOG_LEVEL`: `debug`, `info`, `warn`, or `error`. Log level: choose from `debug`, `info`, `warn`, or `error`. - - `LLAMACPP_REQUIRED_GPU_MEMORY`: `8Gi`, `8192`, or `8192Mi`. Model memory needed. Accepts values like `8Gi`, `8192`, or `8192Mi`. On NVIDIA, this sets `nvidia.com/gpumem` in MiB; on Spark, sets pod memory. + - `LLAMACPP_REQUIRED_GPU_MEMORY`: `8Gi`, `8192`, or `8192Mi`. Model memory needed. Accepts values like `8Gi`, `8192`, or `8192Mi`. On NVIDIA, this sets `nvidia.com/gpumem` in MiB; on Spark and `intel` integrated GPUs, sets pod memory (include model, KV cache and runtime overhead); on `amd-gpu` and `intel-gpu` declares the discrete VRAM quota through the `gpu.bytetrade.io/required-gpu-memory` pod annotation. **How It Works** First, `llm-init` downloads your model and gets it ready, then `llama.cpp` starts serving the model when it’s ready. These two parts run separately but share files and progress info. @@ -92,6 +65,9 @@ spec: |----------------|-------------------|---------------------------------------------------------------------| | `nvidia` | GPU (CUDA) | Set `LLAMACPP_REQUIRED_GPU_MEMORY`; use `-ngl all` for full GPU use | | `nvidia-gb10` | GPU (Spark) | Like above, but memory is managed by the pod | + | `amd-gpu` | GPU (ROCm) | Discrete AMD (e.g. Radeon AI PRO R9700 / gfx1201). Image `server-rocm-b10731`. Use `-ngl all`. Do not use `amd` (Ryzen iGPU). | + | `intel` | Intel integrated GPU (Panther Lake / SYCL) | `server-intel-b10884`; `-ngl all`; LLAMACPP_REQUIRED_GPU_MEMORY = shared pod memory (model + KV + runtime). | + | `intel-gpu` | GPU (SYCL) | Discrete Intel Arc (e.g. Arc Pro B70). Image `server-intel-b10752`. Use `-ngl all`. | **Model Storage** All model files are stored in a shared folder at `appCommon/huggingface`, mounted as `/cache/hf/hub`. All similar apps use this location for models. @@ -107,13 +83,23 @@ spec: locale: - en-US - zh-CN + - de-DE + - es-ES + - it-IT + - fr-FR + - ja-JP license: - text: MIT url: https://github.com/ggml-org/llama.cpp?tab=MIT-1-ov-file accelerator: - # nvidia — discrete GPU; LLAMACPP_REQUIRED_GPU_MEMORY -> gpumem - # nvidia-gb10 — DGX Spark unified memory; same env -> pod memory + # nvidia - discrete GPU; LLAMACPP_REQUIRED_GPU_MEMORY -> nvidia.com/gpumem + # nvidia-gb10 - DGX Spark unified memory; same env -> pod memory + # amd-gpu - discrete AMD ROCm; LLAMACPP_REQUIRED_GPU_MEMORY -> gpu-memory pod annotation + # intel-gpu - discrete Intel Arc SYCL; LLAMACPP_REQUIRED_GPU_MEMORY -> gpu-memory pod annotation + # The two discrete non-NVIDIA modes have no GPU-memory extended resource to + # carry the quota, so the chart puts it on gpu.bytetrade.io/required-gpu-memory + # and app-service reads that to resolve requiredGPUMemory: "-1". - mode: nvidia limitedCpu: "-1" requiredCpu: "-1" @@ -130,11 +116,37 @@ spec: limitedDisk: 500Gi limitedMemory: "-1" requiredMemory: "-1" + - mode: amd-gpu + limitedCpu: "-1" + requiredCpu: "-1" + requiredDisk: 50Mi + limitedDisk: 500Gi + limitedMemory: "-1" + requiredMemory: "-1" + requiredGPUMemory: "-1" + limitedGPUMemory: "-1" + - mode: intel + limitedCpu: "-1" + requiredCpu: "-1" + requiredDisk: 50Mi + limitedDisk: 500Gi + limitedMemory: "-1" + requiredMemory: "-1" + - mode: intel-gpu + limitedCpu: "-1" + requiredCpu: "-1" + requiredDisk: 50Mi + limitedDisk: 500Gi + limitedMemory: "-1" + requiredMemory: "-1" + requiredGPUMemory: "-1" + limitedGPUMemory: "-1" supportArch: - amd64 - arm64 permission: + appCache: true appData: true appCommon: true options: @@ -197,7 +209,7 @@ envs: type: string editable: true applyOnChange: true - description: "Example: -c 8192 -ngl all -fa on. Extra llama.cpp options, to fully use GPU, add `-ngl all`. You can also use values like `LLAMA_ARG_*=val`." + description: "Example: -c 8192 -ngl all -fa on. Extra llama.cpp options, to fully use GPU, add `-ngl all`. You can also use values like `LLAMA_ARG_*=val`. Note that `-c` sizes the whole KV cache rather than one request, and omitting `-np` selects llama.cpp's default of 4 slots sharing it — set `-np 1` for a single long conversation, or raise `-c` to match the concurrency you want." - envName: LOG_LEVEL # single-select: debug | info | warn | error required: false type: string @@ -252,5 +264,5 @@ envs: type: string editable: false applyOnChange: false - description: "Example: `8Gi`, `8192`, or `8192Mi`. GPU memory required for the model. On NVIDIA this sets `nvidia.com/gpumem` in MiB. On Spark it sets pod memory." + description: "Example: `8Gi`, `8192`, or `8192Mi`. GPU memory required for the model. On NVIDIA this sets `nvidia.com/gpumem` in MiB. On Spark and intel integrated GPUs it sets pod memory; include model, KV cache and runtime overhead. On amd-gpu and intel-gpu it declares the discrete VRAM quota through the gpu.bytetrade.io/required-gpu-memory pod annotation." regex: "^[0-9]+(Gi|Mi)?$" diff --git a/llamacppllmbasev3/i18n/de-DE/OlaresManifest.yaml b/llamacppllmbasev3/i18n/de-DE/OlaresManifest.yaml new file mode 100644 index 000000000..796ca5739 --- /dev/null +++ b/llamacppllmbasev3/i18n/de-DE/OlaresManifest.yaml @@ -0,0 +1,40 @@ +metadata: + description: "Generische llama.cpp-Engine-Basis. Beliebiges GGUF-Modell bei der Installation per Env wählen" + title: llama.cpp Engine Base + +spec: + upgradeDescription: | + Chart 1.2.47 — NVIDIA-/Spark-CUDA-Image von server-cuda12-b10752 auf server-cuda12-b10969 aktualisiert. + + fullDescription: | + **WICHTIGER HINWEIS** + Diese App ist eine Vorlage und kann nicht allein genutzt werden. Setzen Sie die unten stehenden Umgebungsvariablen, um ein konkretes Modell anzubinden. + + **Einrichtungsleitfaden für Umgebungsvariablen** + - `MODEL_SOURCE`: `hf:// --include .gguf`. Quelle zum Herunterladen Ihres Modells. Bei Split-Dateien nur den ersten Teil angeben; bei Vision-Language-Modellen den Projektor per Komma hinzufügen, z. B. `hf:// --include model.gguf,hf:// --include mmproj-F16.gguf`. + - `MODEL_NAME`: `owner/repo:quant`. Name Ihres Modells im Format `owner/repo:quant`, z. B. `unsloth/Qwen3.5-2B-GGUF:UD-Q4_K_XL`. Wird auch als OpenAI-Alias verwendet. + - `MODEL_MODE`: `chat` oder `embedding`. Bei embedding fügt das Chart automatisch `--embedding` hinzu. + - `MODEL_SUPPORTS`: Modellfähigkeiten (Vision / Tools / Thinking / None). Wählen Sie, was Ihr Modell unterstützt. Erforderlich. + - `ENGINE_ARGS`: `-c 8192 -ngl all -fa on`. Zusätzliche llama.cpp-Optionen wie `-c 8192 -ngl all -fa on`. Für volle GPU-Nutzung `-ngl all` hinzufügen. Auch Werte wie `LLAMA_ARG_*=val` möglich. + - `LOG_LEVEL`: `debug`, `info`, `warn` oder `error`. Log-Level: Wählen Sie zwischen `debug`, `info`, `warn` oder `error`. + - `LLAMACPP_REQUIRED_GPU_MEMORY`: `8Gi`, `8192` oder `8192Mi`. Benötigter Modellspeicher. Akzeptiert Werte wie `8Gi`, `8192` oder `8192Mi`. Auf NVIDIA setzt dies `nvidia.com/gpumem` in MiB; auf Spark die Pod-Memory; auf `amd-gpu` und `intel-gpu` deklariert es das diskrete VRAM-Kontingent über die Pod-Annotation `gpu.bytetrade.io/required-gpu-memory`. + + **Funktionsweise** + Zuerst lädt `llm-init` Ihr Modell herunter und bereitet es vor, dann startet `llama.cpp` den Serving, wenn es bereit ist. Beide Teile laufen getrennt, teilen aber Dateien und Fortschrittsinformationen. + + **Stack** + - **llama.cpp**: Führt das Modell aus und bedient API-Anfragen (Port 8081) + - **llm-init**: Lädt das Modell herunter, zeigt Fortschritt und stellt die OpenAI-API auf Port 8090 bereit, mit Proxy zu `llamacpp`. + + **Beschleuniger-Modi** + | Modus | Gerät | Details | + |----------------|-------------------|---------------------------------------------------------------------| + | `nvidia` | GPU (CUDA) | `LLAMACPP_REQUIRED_GPU_MEMORY` setzen; `-ngl all` für volle GPU-Nutzung | + | `nvidia-gb10` | GPU (Spark) | Wie oben, aber Speicher wird vom Pod verwaltet | + + **Modellspeicher** + Alle Modelldateien liegen im gemeinsamen Ordner `appCommon/huggingface`, gemountet als `/cache/hf/hub`. Ähnliche Apps nutzen denselben Ort für Modelle. + + **Ressourcentipps** + - CPU, RAM und GPU konfigurieren sich automatisch anhand des Charts; Disk-Einstellungen sind fest. + - Für die GPU `-ngl all` in `ENGINE_ARGS` setzen. `-ngl all` für volles GPU-Offload. diff --git a/llamacppllmbasev3/i18n/en-US/OlaresManifest.yaml b/llamacppllmbasev3/i18n/en-US/OlaresManifest.yaml index 835e2e842..e958e8d74 100644 --- a/llamacppllmbasev3/i18n/en-US/OlaresManifest.yaml +++ b/llamacppllmbasev3/i18n/en-US/OlaresManifest.yaml @@ -4,25 +4,8 @@ metadata: spec: upgradeDescription: | - Chart 1.2.33 — bump engine to `server-cuda12-b10454` and llm-init to `v1.3.8`. + Chart 1.2.47 — Upgrade the NVIDIA / Spark CUDA image from server-cuda12-b10752 to server-cuda12-b10969. - **Images** - - llama.cpp: `docker.io/beclab/ggml-org-llama.cpp:server-cuda12-b10362` → `server-cuda12-b10454` (latest published beclab CUDA 12 server tag; amd64 + arm64). Upstream GitHub is already at b10456; no beclab b10455/b10456 image yet. - - llm-init: `docker.io/beclab/llm-init:v1.3.5` → `v1.3.8`. - - **Muse Glimmer 30B (Meta)** — requires **b10353+**; **b10454 includes** the `muse-glimmer` architecture ([llama.cpp #26841](https://github.com/ggml-org/llama.cpp/pull/26841)). Older builds (e.g. b10331) fail with `unknown model architecture: 'muse-glimmer'`. - - Clone this base and point env at [meta-models/Muse-Glimmer-30B-GGUF](https://huggingface.co/meta-models/Muse-Glimmer-30B-GGUF) or community quants (e.g. unsloth). Typical layout: - - Main GGUF + `mmproj` in `MODEL_SOURCE` (comma-separated) for vision - - DFlash drafter (`dflash-kquant.gguf`) + `--spec-type draft-dflash` or `draft-dspark` in `ENGINE_ARGS` for speculative decode - - **Also in b10362 → b10454:** - - Chat templates receive OpenAI `reasoning_effort` ([llama.cpp #26941](https://github.com/ggml-org/llama.cpp/pull/26941)); Qwen3.8 uses `xhigh` / `medium` / `low` via `--chat-template-kwargs` or the request body - - MTP draft type auto-detect; DFlash / DSpark backend sampling - - MiniMax Text01 / M1 conversion support - - Server `/metrics` and `/slots` stay available during decode; mtmd and jinja fixes - - See https://github.com/ggml-org/llama.cpp/releases for full upstream release notes. fullDescription: | **IMPORTANT NOTE** This app is a template and cannot be used on its own. To use it, set the environment variables below to connect a specific model. @@ -34,7 +17,7 @@ spec: - `MODEL_SUPPORTS`: Model capability options (Vision / Tools / Thinking / None). Pick what your model supports. This is required. - `ENGINE_ARGS`: `-c 8192 -ngl all -fa on`. Extra llama.cpp options, like `-c 8192 -ngl all -fa on`. To fully use GPU, add `-ngl all`. You can also use values like `LLAMA_ARG_*=val`. - `LOG_LEVEL`: `debug`, `info`, `warn`, or `error`. Log level: choose from `debug`, `info`, `warn`, or `error`. - - `LLAMACPP_REQUIRED_GPU_MEMORY`: `8Gi`, `8192`, or `8192Mi`. Model memory needed. Accepts values like `8Gi`, `8192`, or `8192Mi`. On NVIDIA, this sets `nvidia.com/gpumem` in MiB; on Spark, sets pod memory. + - `LLAMACPP_REQUIRED_GPU_MEMORY`: `8Gi`, `8192`, or `8192Mi`. Model memory needed. Accepts values like `8Gi`, `8192`, or `8192Mi`. On NVIDIA, this sets `nvidia.com/gpumem` in MiB; on Spark, sets pod memory; on `amd-gpu` and `intel-gpu` declares the discrete VRAM quota through the `gpu.bytetrade.io/required-gpu-memory` pod annotation. **How It Works** First, `llm-init` downloads your model and gets it ready, then `llama.cpp` starts serving the model when it’s ready. These two parts run separately but share files and progress info. @@ -48,6 +31,9 @@ spec: |----------------|-------------------|---------------------------------------------------------------------| | `nvidia` | GPU (CUDA) | Set `LLAMACPP_REQUIRED_GPU_MEMORY`; use `-ngl all` for full GPU use | | `nvidia-gb10` | GPU (Spark) | Like above, but memory is managed by the pod | + | `amd-gpu` | GPU (ROCm) | Discrete AMD (R9700 / gfx1201). Image `server-rocm-b10731`. Do not use `amd`. | + | `intel` | Intel integrated GPU (Panther Lake / SYCL) | `server-intel-b10884`; `-ngl all`; LLAMACPP_REQUIRED_GPU_MEMORY = shared pod memory (model + KV + runtime). | + | `intel-gpu` | GPU (SYCL) | Discrete Intel Arc (e.g. Arc Pro B70). Image `server-intel-b10752`. Use `-ngl all`. | **Model Storage** All model files are stored in a shared folder at `appCommon/huggingface`, mounted as `/cache/hf/hub`. All similar apps use this location for models. diff --git a/llamacppllmbasev3/i18n/es-ES/OlaresManifest.yaml b/llamacppllmbasev3/i18n/es-ES/OlaresManifest.yaml new file mode 100644 index 000000000..a3f1b8340 --- /dev/null +++ b/llamacppllmbasev3/i18n/es-ES/OlaresManifest.yaml @@ -0,0 +1,40 @@ +metadata: + description: "Base genérica del motor llama.cpp. Elija cualquier modelo GGUF en la instalación vía env" + title: llama.cpp Engine Base + +spec: + upgradeDescription: | + Chart 1.2.47 — Actualiza la imagen CUDA de NVIDIA / Spark de server-cuda12-b10752 a server-cuda12-b10969. + + fullDescription: | + **NOTA IMPORTANTE** + Esta app es una plantilla y no puede usarse sola. Para usarla, configure las variables de entorno siguientes para conectar un modelo concreto. + + **Guía de configuración de variables de entorno** + - `MODEL_SOURCE`: `hf:// --include .gguf`. Fuente para descargar su modelo. En archivos partidos, incluya solo la primera parte; en modelos visión-lenguaje, añada el proyector con una coma, p. ej. `hf:// --include model.gguf,hf:// --include mmproj-F16.gguf`. + - `MODEL_NAME`: `owner/repo:quant`. Nombre de su modelo en formato `owner/repo:quant`, p. ej. `unsloth/Qwen3.5-2B-GGUF:UD-Q4_K_XL`. También se usa como alias OpenAI. + - `MODEL_MODE`: `chat` o `embedding`. Si elige embedding, el chart añade `--embedding` automáticamente. + - `MODEL_SUPPORTS`: Capacidades del modelo (Vision / Tools / Thinking / None). Elija lo que su modelo admite. Obligatorio. + - `ENGINE_ARGS`: `-c 8192 -ngl all -fa on`. Opciones extra de llama.cpp, como `-c 8192 -ngl all -fa on`. Para usar la GPU al completo, añada `-ngl all`. También valores como `LLAMA_ARG_*=val`. + - `LOG_LEVEL`: `debug`, `info`, `warn` o `error`. Nivel de log: elija entre `debug`, `info`, `warn` o `error`. + - `LLAMACPP_REQUIRED_GPU_MEMORY`: `8Gi`, `8192` o `8192Mi`. Memoria de modelo necesaria. Admite valores como `8Gi`, `8192` o `8192Mi`. En NVIDIA establece `nvidia.com/gpumem` en MiB; en Spark, la memoria del pod; en `amd-gpu` e `intel-gpu` declara la cuota de VRAM discreta mediante la anotación de pod `gpu.bytetrade.io/required-gpu-memory`. + + **Cómo funciona** + Primero, `llm-init` descarga su modelo y lo prepara; luego `llama.cpp` empieza a servir el modelo cuando está listo. Ambas partes se ejecutan por separado pero comparten archivos e información de progreso. + + **Stack** + - **llama.cpp**: Ejecuta el modelo y atiende peticiones API (puerto 8081) + - **llm-init**: Descarga el modelo, muestra progreso y proporciona la API OpenAI en el puerto 8090, haciendo proxy a `llamacpp`. + + **Modos de acelerador** + | Modo | Dispositivo | Detalles | + |----------------|-------------------|---------------------------------------------------------------------| + | `nvidia` | GPU (CUDA) | Establezca `LLAMACPP_REQUIRED_GPU_MEMORY`; use `-ngl all` para GPU completa | + | `nvidia-gb10` | GPU (Spark) | Como arriba, pero la memoria la gestiona el pod | + + **Almacenamiento de modelos** + Todos los archivos de modelo se guardan en la carpeta compartida `appCommon/huggingface`, montada como `/cache/hf/hub`. Apps similares usan esta ubicación para modelos. + + **Consejos de recursos** + - CPU, RAM y GPU se autoconfiguran según el chart; los ajustes de disco son fijos. + - Para usar la GPU, establezca `-ngl all` en `ENGINE_ARGS`. Use `-ngl all` para offload GPU completo. diff --git a/llamacppllmbasev3/i18n/fr-FR/OlaresManifest.yaml b/llamacppllmbasev3/i18n/fr-FR/OlaresManifest.yaml new file mode 100644 index 000000000..00d610e5b --- /dev/null +++ b/llamacppllmbasev3/i18n/fr-FR/OlaresManifest.yaml @@ -0,0 +1,40 @@ +metadata: + description: "Base générique du moteur llama.cpp. Choisissez n'importe quel modèle GGUF à l'installation via env" + title: llama.cpp Engine Base + +spec: + upgradeDescription: | + Chart 1.2.47 — Mise à jour CUDA NVIDIA / Spark de server-cuda12-b10752 vers server-cuda12-b10969. + + fullDescription: | + **NOTE IMPORTANTE** + Cette app est un modèle et ne peut pas être utilisée seule. Pour l'utiliser, définissez les variables d'environnement ci-dessous pour connecter un modèle spécifique. + + **Guide de configuration des variables d'environnement** + - `MODEL_SOURCE` : `hf:// --include .gguf`. Source pour télécharger votre modèle. Pour les fichiers fractionnés, n'incluez que la première partie ; pour les modèles vision-langage, ajoutez le projecteur avec une virgule, ex. `hf:// --include model.gguf,hf:// --include mmproj-F16.gguf`. + - `MODEL_NAME` : `owner/repo:quant`. Nom de votre modèle au format `owner/repo:quant`, ex. `unsloth/Qwen3.5-2B-GGUF:UD-Q4_K_XL`. Aussi utilisé comme alias OpenAI. + - `MODEL_MODE` : `chat` ou `embedding`. Si vous choisissez embedding, le chart ajoute automatiquement `--embedding`. + - `MODEL_SUPPORTS` : Capacités du modèle (Vision / Tools / Thinking / None). Choisissez ce que votre modèle prend en charge. Obligatoire. + - `ENGINE_ARGS` : `-c 8192 -ngl all -fa on`. Options llama.cpp supplémentaires, comme `-c 8192 -ngl all -fa on`. Pour utiliser pleinement le GPU, ajoutez `-ngl all`. Aussi des valeurs comme `LLAMA_ARG_*=val`. + - `LOG_LEVEL` : `debug`, `info`, `warn` ou `error`. Niveau de log : choisissez parmi `debug`, `info`, `warn` ou `error`. + - `LLAMACPP_REQUIRED_GPU_MEMORY` : `8Gi`, `8192` ou `8192Mi`. Mémoire modèle nécessaire. Accepte des valeurs comme `8Gi`, `8192` ou `8192Mi`. Sur NVIDIA, définit `nvidia.com/gpumem` en MiB ; sur Spark, la mémoire du pod ; sur `amd-gpu` et `intel-gpu`, déclare le quota de VRAM discrète via l'annotation de pod `gpu.bytetrade.io/required-gpu-memory`. + + **Fonctionnement** + D'abord, `llm-init` télécharge votre modèle et le prépare, puis `llama.cpp` commence à servir le modèle quand il est prêt. Les deux parties tournent séparément mais partagent fichiers et infos de progression. + + **Stack** + - **llama.cpp** : Exécute le modèle et sert les requêtes API (port 8081) + - **llm-init** : Télécharge le modèle, affiche la progression et fournit l'API OpenAI sur le port 8090, en proxy vers `llamacpp`. + + **Modes accélérateur** + | Mode | Périphérique | Détails | + |----------------|-------------------|---------------------------------------------------------------------| + | `nvidia` | GPU (CUDA) | Définir `LLAMACPP_REQUIRED_GPU_MEMORY` ; utiliser `-ngl all` pour GPU complet | + | `nvidia-gb10` | GPU (Spark) | Comme ci-dessus, mais la mémoire est gérée par le pod | + + **Stockage des modèles** + Tous les fichiers modèle sont stockés dans le dossier partagé `appCommon/huggingface`, monté comme `/cache/hf/hub`. Des apps similaires utilisent cet emplacement pour les modèles. + + **Conseils ressources** + - CPU, RAM et GPU s'autoconfigurent selon le chart ; les réglages disque sont fixes. + - Pour utiliser le GPU, définissez `-ngl all` dans `ENGINE_ARGS`. Utilisez `-ngl all` pour un offload GPU complet. diff --git a/llamacppllmbasev3/i18n/it-IT/OlaresManifest.yaml b/llamacppllmbasev3/i18n/it-IT/OlaresManifest.yaml new file mode 100644 index 000000000..c65b2eb9c --- /dev/null +++ b/llamacppllmbasev3/i18n/it-IT/OlaresManifest.yaml @@ -0,0 +1,40 @@ +metadata: + description: "Base generica del motore llama.cpp. Scegli qualsiasi modello GGUF in installazione via env" + title: llama.cpp Engine Base + +spec: + upgradeDescription: | + Chart 1.2.47 — Aggiornamento CUDA NVIDIA / Spark da server-cuda12-b10752 a server-cuda12-b10969. + + fullDescription: | + **NOTA IMPORTANTE** + Questa app è un template e non può essere usata da sola. Per usarla, imposta le variabili d'ambiente sotto per collegare un modello specifico. + + **Guida alla configurazione delle variabili d'ambiente** + - `MODEL_SOURCE`: `hf:// --include .gguf`. Sorgente da cui scaricare il modello. Per file suddivisi includi solo la prima parte; per modelli vision-language aggiungi il proiettore con una virgola, es. `hf:// --include model.gguf,hf:// --include mmproj-F16.gguf`. + - `MODEL_NAME`: `owner/repo:quant`. Nome del modello in formato `owner/repo:quant`, es. `unsloth/Qwen3.5-2B-GGUF:UD-Q4_K_XL`. Usato anche come alias OpenAI. + - `MODEL_MODE`: `chat` o `embedding`. Se scegli embedding, il chart aggiunge automaticamente `--embedding`. + - `MODEL_SUPPORTS`: Capacità del modello (Vision / Tools / Thinking / None). Scegli ciò che il modello supporta. Obbligatorio. + - `ENGINE_ARGS`: `-c 8192 -ngl all -fa on`. Opzioni extra llama.cpp, come `-c 8192 -ngl all -fa on`. Per usare appieno la GPU, aggiungi `-ngl all`. Anche valori come `LLAMA_ARG_*=val`. + - `LOG_LEVEL`: `debug`, `info`, `warn` o `error`. Livello di log: scegli tra `debug`, `info`, `warn` o `error`. + - `LLAMACPP_REQUIRED_GPU_MEMORY`: `8Gi`, `8192` o `8192Mi`. Memoria modello necessaria. Accetta valori come `8Gi`, `8192` o `8192Mi`. Su NVIDIA imposta `nvidia.com/gpumem` in MiB; su Spark, la memoria del pod; su `amd-gpu` e `intel-gpu` dichiara la quota di VRAM discreta tramite l'annotazione del pod `gpu.bytetrade.io/required-gpu-memory`. + + **Come funziona** + Prima `llm-init` scarica il modello e lo prepara, poi `llama.cpp` inizia a servirlo quando è pronto. Le due parti girano separatemente ma condividono file e info di progresso. + + **Stack** + - **llama.cpp**: Esegue il modello e serve le richieste API (porta 8081) + - **llm-init**: Scarica il modello, mostra il progresso e fornisce l'API OpenAI sulla porta 8090, facendo proxy verso `llamacpp`. + + **Modalità acceleratore** + | Modalità | Dispositivo | Dettagli | + |----------------|-------------------|---------------------------------------------------------------------| + | `nvidia` | GPU (CUDA) | Imposta `LLAMACPP_REQUIRED_GPU_MEMORY`; usa `-ngl all` per GPU piena | + | `nvidia-gb10` | GPU (Spark) | Come sopra, ma la memoria è gestita dal pod | + + **Archiviazione modelli** + Tutti i file modello sono nella cartella condivisa `appCommon/huggingface`, montata come `/cache/hf/hub`. App simili usano questa posizione per i modelli. + + **Suggerimenti sulle risorse** + - CPU, RAM e GPU si autoconfigurano in base al chart; le impostazioni disco sono fisse. + - Per usare la GPU, imposta `-ngl all` in `ENGINE_ARGS`. Usa `-ngl all` per offload GPU completo. diff --git a/llamacppllmbasev3/i18n/ja-JP/OlaresManifest.yaml b/llamacppllmbasev3/i18n/ja-JP/OlaresManifest.yaml new file mode 100644 index 000000000..c70f4d8a4 --- /dev/null +++ b/llamacppllmbasev3/i18n/ja-JP/OlaresManifest.yaml @@ -0,0 +1,40 @@ +metadata: + description: "汎用 llama.cpp エンジンベース。インストール時に env で任意の GGUF モデルを選択" + title: llama.cpp Engine Base + +spec: + upgradeDescription: | + Chart 1.2.47 — NVIDIA / Spark CUDA イメージを server-cuda12-b10752 から server-cuda12-b10969 に更新。 + + fullDescription: | + **重要なお知らせ** + このアプリはテンプレートであり、単体では使えません。下の環境変数を設定して特定モデルを接続してください。 + + **環境変数セットアップガイド** + - `MODEL_SOURCE`: `hf:// --include .gguf`。モデルのダウンロード元。分割ファイルは先頭パートのみ; ビジョン言語モデルはカンマでプロジェクタを追加。例: `hf:// --include model.gguf,hf:// --include mmproj-F16.gguf`。 + - `MODEL_NAME`: `owner/repo:quant`。`owner/repo:quant` 形式のモデル名。例: `unsloth/Qwen3.5-2B-GGUF:UD-Q4_K_XL`。OpenAI エイリアスとしても使用。 + - `MODEL_MODE`: `chat` または `embedding`。embedding を選ぶとチャートが自動で `--embedding` を追加。 + - `MODEL_SUPPORTS`: モデル能力(Vision / Tools / Thinking / None)。モデルが対応するものを選択。必須。 + - `ENGINE_ARGS`: `-c 8192 -ngl all -fa on`。`-c 8192 -ngl all -fa on` などの追加 llama.cpp オプション。GPU をフルに使うには `-ngl all`。`LLAMA_ARG_*=val` 形式も可。 + - `LOG_LEVEL`: `debug`、`info`、`warn`、または `error`。ログレベルは `debug`、`info`、`warn`、`error` から選択。 + - `LLAMACPP_REQUIRED_GPU_MEMORY`: `8Gi`、`8192`、または `8192Mi`。必要なモデルメモリ。`8Gi`、`8192`、`8192Mi` などの値を使用可能。NVIDIA では MiB 単位の `nvidia.com/gpumem` を設定; Spark ではポッドメモリを設定; `amd-gpu` と `intel-gpu` では Pod アノテーション `gpu.bytetrade.io/required-gpu-memory` でディスクリート VRAM クォータを宣言。 + + **動作の仕組み** + まず `llm-init` がモデルをダウンロードして準備し、準備ができたら `llama.cpp` がサービングを開始します。両者は別々に動作しますが、ファイルと進捗情報を共有します。 + + **スタック** + - **llama.cpp**: モデルを実行し API リクエストを処理(ポート 8081) + - **llm-init**: モデルをダウンロードし、進捗を表示し、ポート 8090 で OpenAI API を提供して `llamacpp` へプロキシ。 + + **アクセラレータモード** + | モード | デバイス | 詳細 | + |----------------|-------------------|---------------------------------------------------------------------| + | `nvidia` | GPU (CUDA) | `LLAMACPP_REQUIRED_GPU_MEMORY` を設定; フル GPU には `-ngl all` | + | `nvidia-gb10` | GPU (Spark) | 上記と同様だがメモリはポッドが管理 | + + **モデルストレージ** + すべてのモデルファイルは共有フォルダ `appCommon/huggingface` に保存され、`/cache/hf/hub` にマウントされます。類似アプリもモデルにこの場所を使います。 + + **リソースのヒント** + - CPU、RAM、GPU はチャートに基づき自動設定; ディスク設定は固定。 + - GPU を使うには `ENGINE_ARGS` に `-ngl all` を設定。フル GPU オフロードには `-ngl all`。 diff --git a/llamacppllmbasev3/i18n/zh-CN/OlaresManifest.yaml b/llamacppllmbasev3/i18n/zh-CN/OlaresManifest.yaml index 666700a95..85b5dba5e 100644 --- a/llamacppllmbasev3/i18n/zh-CN/OlaresManifest.yaml +++ b/llamacppllmbasev3/i18n/zh-CN/OlaresManifest.yaml @@ -4,25 +4,8 @@ metadata: spec: upgradeDescription: | - Chart 1.2.33 — 引擎升级到 `server-cuda12-b10454`,llm-init 升级到 `v1.3.8`。 + Chart 1.2.47 — NVIDIA / Spark CUDA 镜像从 server-cuda12-b10752 升级到 server-cuda12-b10969。 - **镜像** - - llama.cpp:`docker.io/beclab/ggml-org-llama.cpp:server-cuda12-b10362` → `server-cuda12-b10454`(当前已发布的最新 beclab CUDA 12 server 标签;amd64 + arm64)。上游 GitHub 已到 b10456;尚无 beclab 的 b10455/b10456 镜像。 - - llm-init:`docker.io/beclab/llm-init:v1.3.5` → `v1.3.8`。 - - **Muse Glimmer 30B(Meta)** — 需要 **b10353+**;**b10454 已包含** `muse-glimmer` 架构([llama.cpp #26841](https://github.com/ggml-org/llama.cpp/pull/26841))。更旧的构建(如 b10331)会报 `unknown model architecture: 'muse-glimmer'`。 - - 克隆此基座后,将环境变量指向 [meta-models/Muse-Glimmer-30B-GGUF](https://huggingface.co/meta-models/Muse-Glimmer-30B-GGUF) 或社区量化(如 unsloth)。常见布局: - - 主 GGUF + `mmproj` 写入 `MODEL_SOURCE`(逗号分隔)以支持视觉 - - DFlash 草稿模型(`dflash-kquant.gguf`)并在 `ENGINE_ARGS` 中加 `--spec-type draft-dflash` 或 `draft-dspark` 以开启投机解码 - - **b10362 → b10454 其它变化:** - - 聊天模板可接收 OpenAI `reasoning_effort`([llama.cpp #26941](https://github.com/ggml-org/llama.cpp/pull/26941));Qwen3.8 可用 `--chat-template-kwargs` 或请求体设置 `xhigh` / `medium` / `low` - - MTP 草稿类型自动检测;DFlash / DSpark 后端采样 - - 支持 MiniMax Text01 / M1 转换 - - 解码过程中 `/metrics` 与 `/slots` 仍可用;mtmd 与 jinja 修复 - - 完整上游发布说明见 https://github.com/ggml-org/llama.cpp/releases fullDescription: | **重要提示** 此应用是一个模板,不能单独使用。请按下方环境变量设置模型后,方可使用本应用提供模型服务。 @@ -34,7 +17,7 @@ spec: - `MODEL_SUPPORTS`:模型能力选项(Vision / Tools / Thinking / None)。请选择您的模型支持的功能,必填。 - `ENGINE_ARGS`:`-c 8192 -ngl all -fa on`。如需完全利用 GPU,请加 `-ngl all`。也可用如 `LLAMA_ARG_*=val` 的参数。 - `LOG_LEVEL`:`debug`、`info`、`warn` 或 `error`。日志等级,请从上述选项中选择。 - - `LLAMACPP_REQUIRED_GPU_MEMORY`:`8Gi`、`8192` 或 `8192Mi`。模型(仅 GPU 模式)所需显存。支持 `8Gi`、`8192`、`8192Mi` 等格式;NVIDIA 下设置 `nvidia.com/gpumem`(MiB),Spark 下设置 pod 显存。 + - `LLAMACPP_REQUIRED_GPU_MEMORY`:`8Gi`、`8192` 或 `8192Mi`。模型(仅 GPU 模式)所需显存。支持 `8Gi`、`8192`、`8192Mi` 等格式;NVIDIA 下设置 `nvidia.com/gpumem`(MiB),Spark 下设置 pod 显存,`amd-gpu` 与 `intel-gpu` 下通过 pod annotation `gpu.bytetrade.io/required-gpu-memory` 声明独立显存配额。 **工作流程说明** 首先,`llm-init` 下载并准备模型,然后 `llama.cpp` 在模型准备好后启动服务。这两个部分独立运行但共享文件与进度信息。 @@ -48,6 +31,9 @@ spec: |----------------|------------------|--------------------------------------------------------------| | `nvidia` | GPU (CUDA) | 需设置 `LLAMACPP_REQUIRED_GPU_MEMORY`,用 `-ngl all` 完全利用 GPU | | `nvidia-gb10` | GPU (Spark) | 类似上方,但显存由 pod 管理 | + | `amd-gpu` | GPU (ROCm) | AMD 独显(R9700 / gfx1201)。镜像 `server-rocm-b10731`。不要用 `amd`(核显)。 | + | `intel` | Intel integrated GPU (Panther Lake / SYCL) | `server-intel-b10884`; `-ngl all`; LLAMACPP_REQUIRED_GPU_MEMORY = shared pod memory (model + KV + runtime). | + | `intel-gpu` | GPU (SYCL) | Intel Arc 独显(如 Arc Pro B70)。镜像 `server-intel-b10752`。请用 `-ngl all`。 | **模型存储** 所有模型文件存储在共享目录 `appCommon/huggingface`(挂载为 `/cache/hf/hub`),同类应用均使用此目录。 diff --git a/llamacppllmbasev3/templates/_helpers.tpl b/llamacppllmbasev3/templates/_helpers.tpl index 3dcee99c8..71f04826f 100644 --- a/llamacppllmbasev3/templates/_helpers.tpl +++ b/llamacppllmbasev3/templates/_helpers.tpl @@ -23,3 +23,53 @@ {{- $in := . -}} {{- trim ($in.Args | default "") -}} {{- end -}} +{{- /* Olares GPU mode at install: nvidia | nvidia-gb10 | amd-gpu | intel | intel-gpu. */ -}} +{{- define "llmbase.gpuType" -}} +{{- $gpuObj := .Values.GPU | default dict -}} +{{- $gpuType := .Values.gpu | default "" -}} +{{- if not $gpuType -}} +{{- $gpuType = $gpuObj.Type | default "nvidia" -}} +{{- end -}} +{{- $gpuType -}} +{{- end -}} +{{- /* llama.cpp engine image by accelerator. NVIDIA/Spark CUDA b10752; amd-gpu ROCm b10731; intel-gpu SYCL b10752. */ -}} +{{- define "llamacppllmbasev3.engineImage" -}} +{{- $gpuType := include "llmbase.gpuType" . -}} +{{- $isGb10 := or (eq $gpuType "nvidia-gb10") (eq (include "llmbase.isGb10" .) "true") -}} +{{- $img := .Values.engine.images | default dict -}} +{{- if eq $gpuType "amd-gpu" -}} +{{- $img.amdGpu | default "docker.io/beclab/ggml-org-llama.cpp:server-rocm-b10731" -}} +{{- else if eq $gpuType "intel-gpu" -}} +{{- $img.intelGpu | default "docker.io/beclab/ggml-org-llama.cpp:server-intel-b10752" -}} +{{- else if eq $gpuType "intel" -}} +{{- $img.intel | default "docker.io/beclab/ggml-org-llama.cpp:server-intel-b10884" -}} +{{- else if $isGb10 -}} +{{- $img.nvidiaGb10 | default "docker.io/beclab/ggml-org-llama.cpp:server-cuda12-b10969" -}} +{{- else -}} +{{- $img.nvidia | default "docker.io/beclab/ggml-org-llama.cpp:server-cuda12-b10969" -}} +{{- end -}} +{{- end -}} +{{- /* Spark/GB10: detect from GPU.Type or node hardware (install-time .Values.nodes). */ -}} +{{- define "llmbase.isGb10" -}} +{{- $isGb10 := "false" -}} +{{- if .Values.nodes -}} + {{- range $nodeIndex, $node := .Values.nodes -}} + {{- if eq $nodeIndex 0 -}} + {{- with $node -}} + {{- if .GPUS -}} + {{- range $gpuIndex, $gpu := .GPUS -}} + {{- if eq $gpuIndex 0 -}} + {{- with $gpu -}} + {{- if eq (upper .Model) "GB10" -}} + {{- $isGb10 = "true" -}} + {{- end -}} + {{- end -}} + {{- end -}} + {{- end -}} + {{- end -}} + {{- end -}} + {{- end -}} + {{- end -}} +{{- end -}} +{{- $isGb10 -}} +{{- end -}} diff --git a/llamacppllmbasev3/templates/llamacpp.yaml b/llamacppllmbasev3/templates/llamacpp.yaml index 7640a301f..141bf3f0a 100644 --- a/llamacppllmbasev3/templates/llamacpp.yaml +++ b/llamacppllmbasev3/templates/llamacpp.yaml @@ -13,26 +13,46 @@ {{- $memRequest := trim ($oe.LLAMACPP_MEMORY_REQUEST | default "2Gi") -}} {{- $cpuLimit := trim ($oe.LLAMACPP_CPU_LIMIT | default "6") -}} {{- $memLimit := trim ($oe.LLAMACPP_MEMORY_LIMIT | default "35Gi") -}} -{{- $gpuType := .Values.gpu | default "" -}} -{{- if not $gpuType -}} -{{- $gpuType = .Values.GPU.Type | default "nvidia" -}} +{{- $gpuType := include "llmbase.gpuType" . -}} +{{- $engineImage := include "llamacppllmbasev3.engineImage" . -}} +{{- $gb10FromNodes := eq (include "llmbase.isGb10" .) "true" -}} +{{- $isAmdGpu := eq $gpuType "amd-gpu" -}} +{{- $isIntelGpu := eq $gpuType "intel-gpu" -}} +{{- $isIntel := eq $gpuType "intel" -}} +{{- $needDri := or $isAmdGpu $isIntelGpu $isIntel -}} +{{- $unifiedMem := or $gb10FromNodes (eq $gpuType "nvidia-gb10") $isIntel -}} +{{- /* nvidia.com/gpumem is NVIDIA-HAMi only: HAMi's extender exempts it from + the node capacity check, so it can carry a quota the kubelet never + advertises. The discrete AMD/Intel plugins advertise a card count and + nothing else, so their quota cannot ride on a container resource at all + and travels on a pod annotation instead, which is inert to the + scheduler. app-service reads it to resolve the requiredGPUMemory / + limitedGPUMemory "-1" sentinel this manifest declares for those modes; + without it the sentinel resolves to 0 and the card is dropped from its + own compute binding on resume. */ -}} +{{- $needNvidiaGpumem := and (not $unifiedMem) (not $isAmdGpu) (not $isIntelGpu) -}} +{{- $needGpuMemAnnotation := or $isAmdGpu $isIntelGpu -}} +{{- $shmLimit := "16Gi" -}} +{{- if or $isIntelGpu $isIntel -}} +{{- $shmLimit = "10Gi" -}} {{- end -}} -{{- $unifiedMem := eq $gpuType "nvidia-gb10" -}} {{- $engineArgs = include "llamacppllmbasev3.engineArgs" (dict "Args" $engineArgs) -}} -{{- /* nvidia.com/gpumem needs a BARE MiB int; Spark unified memory reuses the +{{- /* nvidia.com/gpumem needs a BARE MiB int; Spark / Intel integrated unified memory reuses the same value as pod memory ("Mi"). */ -}} {{- $gpuMiB := include "llmbase.gpuMiB" ($oe.LLAMACPP_REQUIRED_GPU_MEMORY | default "4096") -}} {{- if $unifiedMem -}} {{- $memRequest = printf "%sMi" $gpuMiB -}} {{- $memLimit = printf "%sMi" $gpuMiB -}} {{- end -}} -{{- $runStateHostPath := printf "%s/huggingface/llm-init-run/%s" .Values.userspace.appCommon .Release.Name -}} +{{- $runStateHostPath := printf "%s/llm-init-run/%s" .Values.userspace.appCache .Release.Name -}} --- # llama.cpp engine Deployment. wrappers/llamacpp.sh blocks on the sentinel # (download ready), then execs llama-server -hf "$MODEL_NAME". # llm-init reverse-proxies at http://llamacpp:8081 — Service name MUST be -# "llamacpp". GPU base: CUDA server build + gpu-inject; put `-ngl all` in -# ENGINE_ARGS to offload all layers to the GPU. +# "llamacpp". GPU base: CUDA (nvidia / nvidia-gb10), ROCm (amd-gpu), or +# Intel SYCL (intel / intel-gpu); put `-ngl all` in ENGINE_ARGS. +# amd-gpu / intel-gpu VRAM quota is the gpu.bytetrade.io/*-gpu-memory pod +# annotation (not nvidia.com/gpumem). apiVersion: apps/v1 kind: Deployment metadata: @@ -44,6 +64,7 @@ metadata: io.kompose.service: llamacpp annotations: applications.app.bytetrade.io/gpu-inject: "llamacpp" + llamacppllmbasev3.io/engine-image: {{ $engineImage | quote }} spec: replicas: {{ .Values.workloads.llamacppllmbasev3.replicaCount }} selector: @@ -56,6 +77,13 @@ spec: labels: io.kompose.network/chrome-default: "true" io.kompose.service: llamacpp +{{- if $needGpuMemAnnotation }} + # VRAM quota for the modes that cannot express it as a resource. Plain + # Kubernetes quantity (unlike nvidia.com/gpumem, which is a bare MiB int). + annotations: + gpu.bytetrade.io/required-gpu-memory: "{{ $gpuMiB }}Mi" + gpu.bytetrade.io/limited-gpu-memory: "{{ $gpuMiB }}Mi" +{{- end }} spec: restartPolicy: Always volumes: @@ -71,9 +99,35 @@ spec: configMap: name: llm-init-wrappers defaultMode: 0555 +{{- if $isAmdGpu }} + - name: kfd + hostPath: + path: /dev/kfd + type: CharDevice +{{- end }} +{{- if $needDri }} + - name: dri + hostPath: + path: /dev/dri + type: Directory + - name: drm + hostPath: + path: /sys/class/drm + type: Directory + - name: dshm + emptyDir: + medium: Memory + sizeLimit: {{ $shmLimit }} +{{- end }} +{{- if $isAmdGpu }} + securityContext: + supplementalGroups: + - 44 + - 992 +{{- end }} containers: - name: llamacpp - image: docker.io/beclab/ggml-org-llama.cpp:server-cuda12-b10588 + image: "{{ $engineImage }}" imagePullPolicy: IfNotPresent command: ["/bin/sh", "/llm-init/wrappers/llamacpp.sh"] env: @@ -91,6 +145,8 @@ spec: value: "3600" - name: ENGINE_ARGS value: "{{ $engineArgs }}" + - name: OLARES_GPU_MODE + value: {{ $gpuType | quote }} - name: HF_HUB_CACHE value: /cache/hf/hub {{- if $hfEndpoint }} @@ -125,22 +181,31 @@ spec: - name: wrappers mountPath: /llm-init/wrappers readOnly: true +{{- if $isAmdGpu }} + - name: kfd + mountPath: /dev/kfd +{{- end }} +{{- if $needDri }} + - name: dri + mountPath: /dev/dri + - name: drm + mountPath: /sys/class/drm + - name: dshm + mountPath: /dev/shm +{{- end }} resources: - # nvidia.com/gpumem is the per-instance GPU memory quota read by the - # Olares gpu-inject webhook (accelerator is -1/auto, can't carry it). - # It's a non-overcommittable extended resource, so it must be in BOTH - # requests and limits with the same value. Dropped on gb10 (unified - # memory uses pod memory instead) . + # nvidia.com/gpumem: nvidia only in chart; gb10 via pod memory + + # gpu-inject; amd-gpu / intel-gpu via the pod annotation above. requests: cpu: "{{ $cpuRequest }}" memory: "{{ $memRequest }}" -{{- if not $unifiedMem }} +{{- if $needNvidiaGpumem }} nvidia.com/gpumem: {{ $gpuMiB }} {{- end }} limits: cpu: "{{ $cpuLimit }}" memory: "{{ $memLimit }}" -{{- if not $unifiedMem }} +{{- if $needNvidiaGpumem }} nvidia.com/gpumem: {{ $gpuMiB }} {{- end }} --- diff --git a/llamacppllmbasev3/templates/llm-init.yaml b/llamacppllmbasev3/templates/llm-init.yaml index d069c644f..0f6165ddd 100644 --- a/llamacppllmbasev3/templates/llm-init.yaml +++ b/llamacppllmbasev3/templates/llm-init.yaml @@ -53,7 +53,7 @@ {{- $logLevel := $oe.LOG_LEVEL | default "debug" -}} {{- $hfEndpoint := $oe.HF_ENDPOINT | default "" -}} {{- $hfToken := $oe.HF_TOKEN | default "" -}} -{{- $runStateHostPath := printf "%s/huggingface/llm-init-run/%s" .Values.userspace.appCommon .Release.Name -}} +{{- $runStateHostPath := printf "%s/llm-init-run/%s" .Values.userspace.appCache .Release.Name -}} --- # llm-init Pod (no GPU). Downloads GGUF models, writes sentinel + model_path, # serves progress UI + OpenAI /v1/* proxy on :8090, reverse-proxies the engine @@ -109,7 +109,7 @@ spec: # v1.2.4 defaults to the stable LFS download path (hf_xet disabled # unless HF_ENABLE_XET=true); LFS streams each file to disk with a # bounded RAM footprint, avoiding the hf_xet OOM (huggingface_hub#3300). - image: docker.io/beclab/llm-init:v1.5.0 + image: docker.io/beclab/llm-init:v1.7.12 imagePullPolicy: IfNotPresent env: - name: ENGINE_KIND diff --git a/llamacppllmbasev3/values.yaml b/llamacppllmbasev3/values.yaml index c6133968b..ce9cbb804 100644 --- a/llamacppllmbasev3/values.yaml +++ b/llamacppllmbasev3/values.yaml @@ -7,3 +7,13 @@ workloads: replicaCount: 1 llamacppllmbasev3: replicaCount: 1 + +# llama.cpp engine images by Olares accelerator mode. +engine: + images: + nvidia: docker.io/beclab/ggml-org-llama.cpp:server-cuda12-b10969 + nvidiaArm64: docker.io/beclab/ggml-org-llama.cpp:server-cuda12-b10969 + nvidiaGb10: docker.io/beclab/ggml-org-llama.cpp:server-cuda12-b10969 + amdGpu: docker.io/beclab/ggml-org-llama.cpp:server-rocm-b10731 + intel: docker.io/beclab/ggml-org-llama.cpp:server-intel-b10884 + intelGpu: docker.io/beclab/ggml-org-llama.cpp:server-intel-b10752 From 6f12a4a10c5da52724e6d8f3fec6dc775482ec5f Mon Sep 17 00:00:00 2001 From: Harvey Date: Tue, 15 Sep 2026 12:10:10 +0800 Subject: [PATCH 2/3] Limit update to public baseline CUDA image bump --- llamacppllmbasev3/Chart.yaml | 2 +- llamacppllmbasev3/OlaresManifest.yaml | 55 ++--------- .../i18n/de-DE/OlaresManifest.yaml | 40 -------- .../i18n/en-US/OlaresManifest.yaml | 7 +- .../i18n/es-ES/OlaresManifest.yaml | 40 -------- .../i18n/fr-FR/OlaresManifest.yaml | 40 -------- .../i18n/it-IT/OlaresManifest.yaml | 40 -------- .../i18n/ja-JP/OlaresManifest.yaml | 40 -------- .../i18n/zh-CN/OlaresManifest.yaml | 7 +- llamacppllmbasev3/templates/_helpers.tpl | 50 ---------- llamacppllmbasev3/templates/llamacpp.yaml | 97 +++---------------- llamacppllmbasev3/templates/llm-init.yaml | 4 +- llamacppllmbasev3/values.yaml | 10 -- 13 files changed, 31 insertions(+), 401 deletions(-) delete mode 100644 llamacppllmbasev3/i18n/de-DE/OlaresManifest.yaml delete mode 100644 llamacppllmbasev3/i18n/es-ES/OlaresManifest.yaml delete mode 100644 llamacppllmbasev3/i18n/fr-FR/OlaresManifest.yaml delete mode 100644 llamacppllmbasev3/i18n/it-IT/OlaresManifest.yaml delete mode 100644 llamacppllmbasev3/i18n/ja-JP/OlaresManifest.yaml diff --git a/llamacppllmbasev3/Chart.yaml b/llamacppllmbasev3/Chart.yaml index f9d32c363..53c28ebb3 100644 --- a/llamacppllmbasev3/Chart.yaml +++ b/llamacppllmbasev3/Chart.yaml @@ -3,4 +3,4 @@ appVersion: b10969 description: Generic llama.cpp + llm-init base; the GGUF model is supplied at install time via env name: llamacppllmbasev3 type: application -version: 1.2.47 +version: 1.2.38 \ No newline at end of file diff --git a/llamacppllmbasev3/OlaresManifest.yaml b/llamacppllmbasev3/OlaresManifest.yaml index 3caa50aeb..99e79eb4e 100644 --- a/llamacppllmbasev3/OlaresManifest.yaml +++ b/llamacppllmbasev3/OlaresManifest.yaml @@ -7,7 +7,7 @@ metadata: description: "Generic llama.cpp engine base. Pick any GGUF model at install via env." appid: llamacppllmbasev3 title: llama.cpp Engine Base - version: '1.2.47' + version: '1.2.38' categories: - AI - models @@ -36,9 +36,9 @@ workloadReplicas: llminit: 1 spec: onlyAdmin: true - versionName: 'b10969 / b10731-rocm / b10752-intel / b10884-intel-iGPU' + versionName: 'b10969' upgradeDescription: | - Chart 1.2.47 — Upgrade the NVIDIA / Spark CUDA image from server-cuda12-b10752 to server-cuda12-b10969. + Chart 1.2.38 — Upgrade the llama.cpp CUDA image from server-cuda12-b10588 to server-cuda12-b10969. fullDescription: | **IMPORTANT NOTE** @@ -51,7 +51,7 @@ spec: - `MODEL_SUPPORTS`: Model capability options (Vision / Tools / Thinking / None). Pick what your model supports. This is required. - `ENGINE_ARGS`: `-c 8192 -ngl all -fa on`. Extra llama.cpp options, like `-c 8192 -ngl all -fa on`. To fully use GPU, add `-ngl all`. You can also use values like `LLAMA_ARG_*=val`. - `LOG_LEVEL`: `debug`, `info`, `warn`, or `error`. Log level: choose from `debug`, `info`, `warn`, or `error`. - - `LLAMACPP_REQUIRED_GPU_MEMORY`: `8Gi`, `8192`, or `8192Mi`. Model memory needed. Accepts values like `8Gi`, `8192`, or `8192Mi`. On NVIDIA, this sets `nvidia.com/gpumem` in MiB; on Spark and `intel` integrated GPUs, sets pod memory (include model, KV cache and runtime overhead); on `amd-gpu` and `intel-gpu` declares the discrete VRAM quota through the `gpu.bytetrade.io/required-gpu-memory` pod annotation. + - `LLAMACPP_REQUIRED_GPU_MEMORY`: `8Gi`, `8192`, or `8192Mi`. Model memory needed. Accepts values like `8Gi`, `8192`, or `8192Mi`. On NVIDIA, this sets `nvidia.com/gpumem` in MiB; on Spark, sets pod memory. **How It Works** First, `llm-init` downloads your model and gets it ready, then `llama.cpp` starts serving the model when it’s ready. These two parts run separately but share files and progress info. @@ -65,9 +65,6 @@ spec: |----------------|-------------------|---------------------------------------------------------------------| | `nvidia` | GPU (CUDA) | Set `LLAMACPP_REQUIRED_GPU_MEMORY`; use `-ngl all` for full GPU use | | `nvidia-gb10` | GPU (Spark) | Like above, but memory is managed by the pod | - | `amd-gpu` | GPU (ROCm) | Discrete AMD (e.g. Radeon AI PRO R9700 / gfx1201). Image `server-rocm-b10731`. Use `-ngl all`. Do not use `amd` (Ryzen iGPU). | - | `intel` | Intel integrated GPU (Panther Lake / SYCL) | `server-intel-b10884`; `-ngl all`; LLAMACPP_REQUIRED_GPU_MEMORY = shared pod memory (model + KV + runtime). | - | `intel-gpu` | GPU (SYCL) | Discrete Intel Arc (e.g. Arc Pro B70). Image `server-intel-b10752`. Use `-ngl all`. | **Model Storage** All model files are stored in a shared folder at `appCommon/huggingface`, mounted as `/cache/hf/hub`. All similar apps use this location for models. @@ -83,23 +80,13 @@ spec: locale: - en-US - zh-CN - - de-DE - - es-ES - - it-IT - - fr-FR - - ja-JP license: - text: MIT url: https://github.com/ggml-org/llama.cpp?tab=MIT-1-ov-file accelerator: - # nvidia - discrete GPU; LLAMACPP_REQUIRED_GPU_MEMORY -> nvidia.com/gpumem - # nvidia-gb10 - DGX Spark unified memory; same env -> pod memory - # amd-gpu - discrete AMD ROCm; LLAMACPP_REQUIRED_GPU_MEMORY -> gpu-memory pod annotation - # intel-gpu - discrete Intel Arc SYCL; LLAMACPP_REQUIRED_GPU_MEMORY -> gpu-memory pod annotation - # The two discrete non-NVIDIA modes have no GPU-memory extended resource to - # carry the quota, so the chart puts it on gpu.bytetrade.io/required-gpu-memory - # and app-service reads that to resolve requiredGPUMemory: "-1". + # nvidia — discrete GPU; LLAMACPP_REQUIRED_GPU_MEMORY -> gpumem + # nvidia-gb10 — DGX Spark unified memory; same env -> pod memory - mode: nvidia limitedCpu: "-1" requiredCpu: "-1" @@ -116,37 +103,11 @@ spec: limitedDisk: 500Gi limitedMemory: "-1" requiredMemory: "-1" - - mode: amd-gpu - limitedCpu: "-1" - requiredCpu: "-1" - requiredDisk: 50Mi - limitedDisk: 500Gi - limitedMemory: "-1" - requiredMemory: "-1" - requiredGPUMemory: "-1" - limitedGPUMemory: "-1" - - mode: intel - limitedCpu: "-1" - requiredCpu: "-1" - requiredDisk: 50Mi - limitedDisk: 500Gi - limitedMemory: "-1" - requiredMemory: "-1" - - mode: intel-gpu - limitedCpu: "-1" - requiredCpu: "-1" - requiredDisk: 50Mi - limitedDisk: 500Gi - limitedMemory: "-1" - requiredMemory: "-1" - requiredGPUMemory: "-1" - limitedGPUMemory: "-1" supportArch: - amd64 - arm64 permission: - appCache: true appData: true appCommon: true options: @@ -209,7 +170,7 @@ envs: type: string editable: true applyOnChange: true - description: "Example: -c 8192 -ngl all -fa on. Extra llama.cpp options, to fully use GPU, add `-ngl all`. You can also use values like `LLAMA_ARG_*=val`. Note that `-c` sizes the whole KV cache rather than one request, and omitting `-np` selects llama.cpp's default of 4 slots sharing it — set `-np 1` for a single long conversation, or raise `-c` to match the concurrency you want." + description: "Example: -c 8192 -ngl all -fa on. Extra llama.cpp options, to fully use GPU, add `-ngl all`. You can also use values like `LLAMA_ARG_*=val`." - envName: LOG_LEVEL # single-select: debug | info | warn | error required: false type: string @@ -264,5 +225,5 @@ envs: type: string editable: false applyOnChange: false - description: "Example: `8Gi`, `8192`, or `8192Mi`. GPU memory required for the model. On NVIDIA this sets `nvidia.com/gpumem` in MiB. On Spark and intel integrated GPUs it sets pod memory; include model, KV cache and runtime overhead. On amd-gpu and intel-gpu it declares the discrete VRAM quota through the gpu.bytetrade.io/required-gpu-memory pod annotation." + description: "Example: `8Gi`, `8192`, or `8192Mi`. GPU memory required for the model. On NVIDIA this sets `nvidia.com/gpumem` in MiB. On Spark it sets pod memory." regex: "^[0-9]+(Gi|Mi)?$" diff --git a/llamacppllmbasev3/i18n/de-DE/OlaresManifest.yaml b/llamacppllmbasev3/i18n/de-DE/OlaresManifest.yaml deleted file mode 100644 index 796ca5739..000000000 --- a/llamacppllmbasev3/i18n/de-DE/OlaresManifest.yaml +++ /dev/null @@ -1,40 +0,0 @@ -metadata: - description: "Generische llama.cpp-Engine-Basis. Beliebiges GGUF-Modell bei der Installation per Env wählen" - title: llama.cpp Engine Base - -spec: - upgradeDescription: | - Chart 1.2.47 — NVIDIA-/Spark-CUDA-Image von server-cuda12-b10752 auf server-cuda12-b10969 aktualisiert. - - fullDescription: | - **WICHTIGER HINWEIS** - Diese App ist eine Vorlage und kann nicht allein genutzt werden. Setzen Sie die unten stehenden Umgebungsvariablen, um ein konkretes Modell anzubinden. - - **Einrichtungsleitfaden für Umgebungsvariablen** - - `MODEL_SOURCE`: `hf:// --include .gguf`. Quelle zum Herunterladen Ihres Modells. Bei Split-Dateien nur den ersten Teil angeben; bei Vision-Language-Modellen den Projektor per Komma hinzufügen, z. B. `hf:// --include model.gguf,hf:// --include mmproj-F16.gguf`. - - `MODEL_NAME`: `owner/repo:quant`. Name Ihres Modells im Format `owner/repo:quant`, z. B. `unsloth/Qwen3.5-2B-GGUF:UD-Q4_K_XL`. Wird auch als OpenAI-Alias verwendet. - - `MODEL_MODE`: `chat` oder `embedding`. Bei embedding fügt das Chart automatisch `--embedding` hinzu. - - `MODEL_SUPPORTS`: Modellfähigkeiten (Vision / Tools / Thinking / None). Wählen Sie, was Ihr Modell unterstützt. Erforderlich. - - `ENGINE_ARGS`: `-c 8192 -ngl all -fa on`. Zusätzliche llama.cpp-Optionen wie `-c 8192 -ngl all -fa on`. Für volle GPU-Nutzung `-ngl all` hinzufügen. Auch Werte wie `LLAMA_ARG_*=val` möglich. - - `LOG_LEVEL`: `debug`, `info`, `warn` oder `error`. Log-Level: Wählen Sie zwischen `debug`, `info`, `warn` oder `error`. - - `LLAMACPP_REQUIRED_GPU_MEMORY`: `8Gi`, `8192` oder `8192Mi`. Benötigter Modellspeicher. Akzeptiert Werte wie `8Gi`, `8192` oder `8192Mi`. Auf NVIDIA setzt dies `nvidia.com/gpumem` in MiB; auf Spark die Pod-Memory; auf `amd-gpu` und `intel-gpu` deklariert es das diskrete VRAM-Kontingent über die Pod-Annotation `gpu.bytetrade.io/required-gpu-memory`. - - **Funktionsweise** - Zuerst lädt `llm-init` Ihr Modell herunter und bereitet es vor, dann startet `llama.cpp` den Serving, wenn es bereit ist. Beide Teile laufen getrennt, teilen aber Dateien und Fortschrittsinformationen. - - **Stack** - - **llama.cpp**: Führt das Modell aus und bedient API-Anfragen (Port 8081) - - **llm-init**: Lädt das Modell herunter, zeigt Fortschritt und stellt die OpenAI-API auf Port 8090 bereit, mit Proxy zu `llamacpp`. - - **Beschleuniger-Modi** - | Modus | Gerät | Details | - |----------------|-------------------|---------------------------------------------------------------------| - | `nvidia` | GPU (CUDA) | `LLAMACPP_REQUIRED_GPU_MEMORY` setzen; `-ngl all` für volle GPU-Nutzung | - | `nvidia-gb10` | GPU (Spark) | Wie oben, aber Speicher wird vom Pod verwaltet | - - **Modellspeicher** - Alle Modelldateien liegen im gemeinsamen Ordner `appCommon/huggingface`, gemountet als `/cache/hf/hub`. Ähnliche Apps nutzen denselben Ort für Modelle. - - **Ressourcentipps** - - CPU, RAM und GPU konfigurieren sich automatisch anhand des Charts; Disk-Einstellungen sind fest. - - Für die GPU `-ngl all` in `ENGINE_ARGS` setzen. `-ngl all` für volles GPU-Offload. diff --git a/llamacppllmbasev3/i18n/en-US/OlaresManifest.yaml b/llamacppllmbasev3/i18n/en-US/OlaresManifest.yaml index e958e8d74..cf78b236b 100644 --- a/llamacppllmbasev3/i18n/en-US/OlaresManifest.yaml +++ b/llamacppllmbasev3/i18n/en-US/OlaresManifest.yaml @@ -4,7 +4,7 @@ metadata: spec: upgradeDescription: | - Chart 1.2.47 — Upgrade the NVIDIA / Spark CUDA image from server-cuda12-b10752 to server-cuda12-b10969. + Chart 1.2.38 — Upgrade the llama.cpp CUDA image from server-cuda12-b10588 to server-cuda12-b10969. fullDescription: | **IMPORTANT NOTE** @@ -17,7 +17,7 @@ spec: - `MODEL_SUPPORTS`: Model capability options (Vision / Tools / Thinking / None). Pick what your model supports. This is required. - `ENGINE_ARGS`: `-c 8192 -ngl all -fa on`. Extra llama.cpp options, like `-c 8192 -ngl all -fa on`. To fully use GPU, add `-ngl all`. You can also use values like `LLAMA_ARG_*=val`. - `LOG_LEVEL`: `debug`, `info`, `warn`, or `error`. Log level: choose from `debug`, `info`, `warn`, or `error`. - - `LLAMACPP_REQUIRED_GPU_MEMORY`: `8Gi`, `8192`, or `8192Mi`. Model memory needed. Accepts values like `8Gi`, `8192`, or `8192Mi`. On NVIDIA, this sets `nvidia.com/gpumem` in MiB; on Spark, sets pod memory; on `amd-gpu` and `intel-gpu` declares the discrete VRAM quota through the `gpu.bytetrade.io/required-gpu-memory` pod annotation. + - `LLAMACPP_REQUIRED_GPU_MEMORY`: `8Gi`, `8192`, or `8192Mi`. Model memory needed. Accepts values like `8Gi`, `8192`, or `8192Mi`. On NVIDIA, this sets `nvidia.com/gpumem` in MiB; on Spark, sets pod memory. **How It Works** First, `llm-init` downloads your model and gets it ready, then `llama.cpp` starts serving the model when it’s ready. These two parts run separately but share files and progress info. @@ -31,9 +31,6 @@ spec: |----------------|-------------------|---------------------------------------------------------------------| | `nvidia` | GPU (CUDA) | Set `LLAMACPP_REQUIRED_GPU_MEMORY`; use `-ngl all` for full GPU use | | `nvidia-gb10` | GPU (Spark) | Like above, but memory is managed by the pod | - | `amd-gpu` | GPU (ROCm) | Discrete AMD (R9700 / gfx1201). Image `server-rocm-b10731`. Do not use `amd`. | - | `intel` | Intel integrated GPU (Panther Lake / SYCL) | `server-intel-b10884`; `-ngl all`; LLAMACPP_REQUIRED_GPU_MEMORY = shared pod memory (model + KV + runtime). | - | `intel-gpu` | GPU (SYCL) | Discrete Intel Arc (e.g. Arc Pro B70). Image `server-intel-b10752`. Use `-ngl all`. | **Model Storage** All model files are stored in a shared folder at `appCommon/huggingface`, mounted as `/cache/hf/hub`. All similar apps use this location for models. diff --git a/llamacppllmbasev3/i18n/es-ES/OlaresManifest.yaml b/llamacppllmbasev3/i18n/es-ES/OlaresManifest.yaml deleted file mode 100644 index a3f1b8340..000000000 --- a/llamacppllmbasev3/i18n/es-ES/OlaresManifest.yaml +++ /dev/null @@ -1,40 +0,0 @@ -metadata: - description: "Base genérica del motor llama.cpp. Elija cualquier modelo GGUF en la instalación vía env" - title: llama.cpp Engine Base - -spec: - upgradeDescription: | - Chart 1.2.47 — Actualiza la imagen CUDA de NVIDIA / Spark de server-cuda12-b10752 a server-cuda12-b10969. - - fullDescription: | - **NOTA IMPORTANTE** - Esta app es una plantilla y no puede usarse sola. Para usarla, configure las variables de entorno siguientes para conectar un modelo concreto. - - **Guía de configuración de variables de entorno** - - `MODEL_SOURCE`: `hf:// --include .gguf`. Fuente para descargar su modelo. En archivos partidos, incluya solo la primera parte; en modelos visión-lenguaje, añada el proyector con una coma, p. ej. `hf:// --include model.gguf,hf:// --include mmproj-F16.gguf`. - - `MODEL_NAME`: `owner/repo:quant`. Nombre de su modelo en formato `owner/repo:quant`, p. ej. `unsloth/Qwen3.5-2B-GGUF:UD-Q4_K_XL`. También se usa como alias OpenAI. - - `MODEL_MODE`: `chat` o `embedding`. Si elige embedding, el chart añade `--embedding` automáticamente. - - `MODEL_SUPPORTS`: Capacidades del modelo (Vision / Tools / Thinking / None). Elija lo que su modelo admite. Obligatorio. - - `ENGINE_ARGS`: `-c 8192 -ngl all -fa on`. Opciones extra de llama.cpp, como `-c 8192 -ngl all -fa on`. Para usar la GPU al completo, añada `-ngl all`. También valores como `LLAMA_ARG_*=val`. - - `LOG_LEVEL`: `debug`, `info`, `warn` o `error`. Nivel de log: elija entre `debug`, `info`, `warn` o `error`. - - `LLAMACPP_REQUIRED_GPU_MEMORY`: `8Gi`, `8192` o `8192Mi`. Memoria de modelo necesaria. Admite valores como `8Gi`, `8192` o `8192Mi`. En NVIDIA establece `nvidia.com/gpumem` en MiB; en Spark, la memoria del pod; en `amd-gpu` e `intel-gpu` declara la cuota de VRAM discreta mediante la anotación de pod `gpu.bytetrade.io/required-gpu-memory`. - - **Cómo funciona** - Primero, `llm-init` descarga su modelo y lo prepara; luego `llama.cpp` empieza a servir el modelo cuando está listo. Ambas partes se ejecutan por separado pero comparten archivos e información de progreso. - - **Stack** - - **llama.cpp**: Ejecuta el modelo y atiende peticiones API (puerto 8081) - - **llm-init**: Descarga el modelo, muestra progreso y proporciona la API OpenAI en el puerto 8090, haciendo proxy a `llamacpp`. - - **Modos de acelerador** - | Modo | Dispositivo | Detalles | - |----------------|-------------------|---------------------------------------------------------------------| - | `nvidia` | GPU (CUDA) | Establezca `LLAMACPP_REQUIRED_GPU_MEMORY`; use `-ngl all` para GPU completa | - | `nvidia-gb10` | GPU (Spark) | Como arriba, pero la memoria la gestiona el pod | - - **Almacenamiento de modelos** - Todos los archivos de modelo se guardan en la carpeta compartida `appCommon/huggingface`, montada como `/cache/hf/hub`. Apps similares usan esta ubicación para modelos. - - **Consejos de recursos** - - CPU, RAM y GPU se autoconfiguran según el chart; los ajustes de disco son fijos. - - Para usar la GPU, establezca `-ngl all` en `ENGINE_ARGS`. Use `-ngl all` para offload GPU completo. diff --git a/llamacppllmbasev3/i18n/fr-FR/OlaresManifest.yaml b/llamacppllmbasev3/i18n/fr-FR/OlaresManifest.yaml deleted file mode 100644 index 00d610e5b..000000000 --- a/llamacppllmbasev3/i18n/fr-FR/OlaresManifest.yaml +++ /dev/null @@ -1,40 +0,0 @@ -metadata: - description: "Base générique du moteur llama.cpp. Choisissez n'importe quel modèle GGUF à l'installation via env" - title: llama.cpp Engine Base - -spec: - upgradeDescription: | - Chart 1.2.47 — Mise à jour CUDA NVIDIA / Spark de server-cuda12-b10752 vers server-cuda12-b10969. - - fullDescription: | - **NOTE IMPORTANTE** - Cette app est un modèle et ne peut pas être utilisée seule. Pour l'utiliser, définissez les variables d'environnement ci-dessous pour connecter un modèle spécifique. - - **Guide de configuration des variables d'environnement** - - `MODEL_SOURCE` : `hf:// --include .gguf`. Source pour télécharger votre modèle. Pour les fichiers fractionnés, n'incluez que la première partie ; pour les modèles vision-langage, ajoutez le projecteur avec une virgule, ex. `hf:// --include model.gguf,hf:// --include mmproj-F16.gguf`. - - `MODEL_NAME` : `owner/repo:quant`. Nom de votre modèle au format `owner/repo:quant`, ex. `unsloth/Qwen3.5-2B-GGUF:UD-Q4_K_XL`. Aussi utilisé comme alias OpenAI. - - `MODEL_MODE` : `chat` ou `embedding`. Si vous choisissez embedding, le chart ajoute automatiquement `--embedding`. - - `MODEL_SUPPORTS` : Capacités du modèle (Vision / Tools / Thinking / None). Choisissez ce que votre modèle prend en charge. Obligatoire. - - `ENGINE_ARGS` : `-c 8192 -ngl all -fa on`. Options llama.cpp supplémentaires, comme `-c 8192 -ngl all -fa on`. Pour utiliser pleinement le GPU, ajoutez `-ngl all`. Aussi des valeurs comme `LLAMA_ARG_*=val`. - - `LOG_LEVEL` : `debug`, `info`, `warn` ou `error`. Niveau de log : choisissez parmi `debug`, `info`, `warn` ou `error`. - - `LLAMACPP_REQUIRED_GPU_MEMORY` : `8Gi`, `8192` ou `8192Mi`. Mémoire modèle nécessaire. Accepte des valeurs comme `8Gi`, `8192` ou `8192Mi`. Sur NVIDIA, définit `nvidia.com/gpumem` en MiB ; sur Spark, la mémoire du pod ; sur `amd-gpu` et `intel-gpu`, déclare le quota de VRAM discrète via l'annotation de pod `gpu.bytetrade.io/required-gpu-memory`. - - **Fonctionnement** - D'abord, `llm-init` télécharge votre modèle et le prépare, puis `llama.cpp` commence à servir le modèle quand il est prêt. Les deux parties tournent séparément mais partagent fichiers et infos de progression. - - **Stack** - - **llama.cpp** : Exécute le modèle et sert les requêtes API (port 8081) - - **llm-init** : Télécharge le modèle, affiche la progression et fournit l'API OpenAI sur le port 8090, en proxy vers `llamacpp`. - - **Modes accélérateur** - | Mode | Périphérique | Détails | - |----------------|-------------------|---------------------------------------------------------------------| - | `nvidia` | GPU (CUDA) | Définir `LLAMACPP_REQUIRED_GPU_MEMORY` ; utiliser `-ngl all` pour GPU complet | - | `nvidia-gb10` | GPU (Spark) | Comme ci-dessus, mais la mémoire est gérée par le pod | - - **Stockage des modèles** - Tous les fichiers modèle sont stockés dans le dossier partagé `appCommon/huggingface`, monté comme `/cache/hf/hub`. Des apps similaires utilisent cet emplacement pour les modèles. - - **Conseils ressources** - - CPU, RAM et GPU s'autoconfigurent selon le chart ; les réglages disque sont fixes. - - Pour utiliser le GPU, définissez `-ngl all` dans `ENGINE_ARGS`. Utilisez `-ngl all` pour un offload GPU complet. diff --git a/llamacppllmbasev3/i18n/it-IT/OlaresManifest.yaml b/llamacppllmbasev3/i18n/it-IT/OlaresManifest.yaml deleted file mode 100644 index c65b2eb9c..000000000 --- a/llamacppllmbasev3/i18n/it-IT/OlaresManifest.yaml +++ /dev/null @@ -1,40 +0,0 @@ -metadata: - description: "Base generica del motore llama.cpp. Scegli qualsiasi modello GGUF in installazione via env" - title: llama.cpp Engine Base - -spec: - upgradeDescription: | - Chart 1.2.47 — Aggiornamento CUDA NVIDIA / Spark da server-cuda12-b10752 a server-cuda12-b10969. - - fullDescription: | - **NOTA IMPORTANTE** - Questa app è un template e non può essere usata da sola. Per usarla, imposta le variabili d'ambiente sotto per collegare un modello specifico. - - **Guida alla configurazione delle variabili d'ambiente** - - `MODEL_SOURCE`: `hf:// --include .gguf`. Sorgente da cui scaricare il modello. Per file suddivisi includi solo la prima parte; per modelli vision-language aggiungi il proiettore con una virgola, es. `hf:// --include model.gguf,hf:// --include mmproj-F16.gguf`. - - `MODEL_NAME`: `owner/repo:quant`. Nome del modello in formato `owner/repo:quant`, es. `unsloth/Qwen3.5-2B-GGUF:UD-Q4_K_XL`. Usato anche come alias OpenAI. - - `MODEL_MODE`: `chat` o `embedding`. Se scegli embedding, il chart aggiunge automaticamente `--embedding`. - - `MODEL_SUPPORTS`: Capacità del modello (Vision / Tools / Thinking / None). Scegli ciò che il modello supporta. Obbligatorio. - - `ENGINE_ARGS`: `-c 8192 -ngl all -fa on`. Opzioni extra llama.cpp, come `-c 8192 -ngl all -fa on`. Per usare appieno la GPU, aggiungi `-ngl all`. Anche valori come `LLAMA_ARG_*=val`. - - `LOG_LEVEL`: `debug`, `info`, `warn` o `error`. Livello di log: scegli tra `debug`, `info`, `warn` o `error`. - - `LLAMACPP_REQUIRED_GPU_MEMORY`: `8Gi`, `8192` o `8192Mi`. Memoria modello necessaria. Accetta valori come `8Gi`, `8192` o `8192Mi`. Su NVIDIA imposta `nvidia.com/gpumem` in MiB; su Spark, la memoria del pod; su `amd-gpu` e `intel-gpu` dichiara la quota di VRAM discreta tramite l'annotazione del pod `gpu.bytetrade.io/required-gpu-memory`. - - **Come funziona** - Prima `llm-init` scarica il modello e lo prepara, poi `llama.cpp` inizia a servirlo quando è pronto. Le due parti girano separatemente ma condividono file e info di progresso. - - **Stack** - - **llama.cpp**: Esegue il modello e serve le richieste API (porta 8081) - - **llm-init**: Scarica il modello, mostra il progresso e fornisce l'API OpenAI sulla porta 8090, facendo proxy verso `llamacpp`. - - **Modalità acceleratore** - | Modalità | Dispositivo | Dettagli | - |----------------|-------------------|---------------------------------------------------------------------| - | `nvidia` | GPU (CUDA) | Imposta `LLAMACPP_REQUIRED_GPU_MEMORY`; usa `-ngl all` per GPU piena | - | `nvidia-gb10` | GPU (Spark) | Come sopra, ma la memoria è gestita dal pod | - - **Archiviazione modelli** - Tutti i file modello sono nella cartella condivisa `appCommon/huggingface`, montata come `/cache/hf/hub`. App simili usano questa posizione per i modelli. - - **Suggerimenti sulle risorse** - - CPU, RAM e GPU si autoconfigurano in base al chart; le impostazioni disco sono fisse. - - Per usare la GPU, imposta `-ngl all` in `ENGINE_ARGS`. Usa `-ngl all` per offload GPU completo. diff --git a/llamacppllmbasev3/i18n/ja-JP/OlaresManifest.yaml b/llamacppllmbasev3/i18n/ja-JP/OlaresManifest.yaml deleted file mode 100644 index c70f4d8a4..000000000 --- a/llamacppllmbasev3/i18n/ja-JP/OlaresManifest.yaml +++ /dev/null @@ -1,40 +0,0 @@ -metadata: - description: "汎用 llama.cpp エンジンベース。インストール時に env で任意の GGUF モデルを選択" - title: llama.cpp Engine Base - -spec: - upgradeDescription: | - Chart 1.2.47 — NVIDIA / Spark CUDA イメージを server-cuda12-b10752 から server-cuda12-b10969 に更新。 - - fullDescription: | - **重要なお知らせ** - このアプリはテンプレートであり、単体では使えません。下の環境変数を設定して特定モデルを接続してください。 - - **環境変数セットアップガイド** - - `MODEL_SOURCE`: `hf:// --include .gguf`。モデルのダウンロード元。分割ファイルは先頭パートのみ; ビジョン言語モデルはカンマでプロジェクタを追加。例: `hf:// --include model.gguf,hf:// --include mmproj-F16.gguf`。 - - `MODEL_NAME`: `owner/repo:quant`。`owner/repo:quant` 形式のモデル名。例: `unsloth/Qwen3.5-2B-GGUF:UD-Q4_K_XL`。OpenAI エイリアスとしても使用。 - - `MODEL_MODE`: `chat` または `embedding`。embedding を選ぶとチャートが自動で `--embedding` を追加。 - - `MODEL_SUPPORTS`: モデル能力(Vision / Tools / Thinking / None)。モデルが対応するものを選択。必須。 - - `ENGINE_ARGS`: `-c 8192 -ngl all -fa on`。`-c 8192 -ngl all -fa on` などの追加 llama.cpp オプション。GPU をフルに使うには `-ngl all`。`LLAMA_ARG_*=val` 形式も可。 - - `LOG_LEVEL`: `debug`、`info`、`warn`、または `error`。ログレベルは `debug`、`info`、`warn`、`error` から選択。 - - `LLAMACPP_REQUIRED_GPU_MEMORY`: `8Gi`、`8192`、または `8192Mi`。必要なモデルメモリ。`8Gi`、`8192`、`8192Mi` などの値を使用可能。NVIDIA では MiB 単位の `nvidia.com/gpumem` を設定; Spark ではポッドメモリを設定; `amd-gpu` と `intel-gpu` では Pod アノテーション `gpu.bytetrade.io/required-gpu-memory` でディスクリート VRAM クォータを宣言。 - - **動作の仕組み** - まず `llm-init` がモデルをダウンロードして準備し、準備ができたら `llama.cpp` がサービングを開始します。両者は別々に動作しますが、ファイルと進捗情報を共有します。 - - **スタック** - - **llama.cpp**: モデルを実行し API リクエストを処理(ポート 8081) - - **llm-init**: モデルをダウンロードし、進捗を表示し、ポート 8090 で OpenAI API を提供して `llamacpp` へプロキシ。 - - **アクセラレータモード** - | モード | デバイス | 詳細 | - |----------------|-------------------|---------------------------------------------------------------------| - | `nvidia` | GPU (CUDA) | `LLAMACPP_REQUIRED_GPU_MEMORY` を設定; フル GPU には `-ngl all` | - | `nvidia-gb10` | GPU (Spark) | 上記と同様だがメモリはポッドが管理 | - - **モデルストレージ** - すべてのモデルファイルは共有フォルダ `appCommon/huggingface` に保存され、`/cache/hf/hub` にマウントされます。類似アプリもモデルにこの場所を使います。 - - **リソースのヒント** - - CPU、RAM、GPU はチャートに基づき自動設定; ディスク設定は固定。 - - GPU を使うには `ENGINE_ARGS` に `-ngl all` を設定。フル GPU オフロードには `-ngl all`。 diff --git a/llamacppllmbasev3/i18n/zh-CN/OlaresManifest.yaml b/llamacppllmbasev3/i18n/zh-CN/OlaresManifest.yaml index 85b5dba5e..752f2bb82 100644 --- a/llamacppllmbasev3/i18n/zh-CN/OlaresManifest.yaml +++ b/llamacppllmbasev3/i18n/zh-CN/OlaresManifest.yaml @@ -4,7 +4,7 @@ metadata: spec: upgradeDescription: | - Chart 1.2.47 — NVIDIA / Spark CUDA 镜像从 server-cuda12-b10752 升级到 server-cuda12-b10969。 + Chart 1.2.38 — llama.cpp CUDA 镜像从 server-cuda12-b10588 升级到 server-cuda12-b10969。 fullDescription: | **重要提示** @@ -17,7 +17,7 @@ spec: - `MODEL_SUPPORTS`:模型能力选项(Vision / Tools / Thinking / None)。请选择您的模型支持的功能,必填。 - `ENGINE_ARGS`:`-c 8192 -ngl all -fa on`。如需完全利用 GPU,请加 `-ngl all`。也可用如 `LLAMA_ARG_*=val` 的参数。 - `LOG_LEVEL`:`debug`、`info`、`warn` 或 `error`。日志等级,请从上述选项中选择。 - - `LLAMACPP_REQUIRED_GPU_MEMORY`:`8Gi`、`8192` 或 `8192Mi`。模型(仅 GPU 模式)所需显存。支持 `8Gi`、`8192`、`8192Mi` 等格式;NVIDIA 下设置 `nvidia.com/gpumem`(MiB),Spark 下设置 pod 显存,`amd-gpu` 与 `intel-gpu` 下通过 pod annotation `gpu.bytetrade.io/required-gpu-memory` 声明独立显存配额。 + - `LLAMACPP_REQUIRED_GPU_MEMORY`:`8Gi`、`8192` 或 `8192Mi`。模型(仅 GPU 模式)所需显存。支持 `8Gi`、`8192`、`8192Mi` 等格式;NVIDIA 下设置 `nvidia.com/gpumem`(MiB),Spark 下设置 pod 显存。 **工作流程说明** 首先,`llm-init` 下载并准备模型,然后 `llama.cpp` 在模型准备好后启动服务。这两个部分独立运行但共享文件与进度信息。 @@ -31,9 +31,6 @@ spec: |----------------|------------------|--------------------------------------------------------------| | `nvidia` | GPU (CUDA) | 需设置 `LLAMACPP_REQUIRED_GPU_MEMORY`,用 `-ngl all` 完全利用 GPU | | `nvidia-gb10` | GPU (Spark) | 类似上方,但显存由 pod 管理 | - | `amd-gpu` | GPU (ROCm) | AMD 独显(R9700 / gfx1201)。镜像 `server-rocm-b10731`。不要用 `amd`(核显)。 | - | `intel` | Intel integrated GPU (Panther Lake / SYCL) | `server-intel-b10884`; `-ngl all`; LLAMACPP_REQUIRED_GPU_MEMORY = shared pod memory (model + KV + runtime). | - | `intel-gpu` | GPU (SYCL) | Intel Arc 独显(如 Arc Pro B70)。镜像 `server-intel-b10752`。请用 `-ngl all`。 | **模型存储** 所有模型文件存储在共享目录 `appCommon/huggingface`(挂载为 `/cache/hf/hub`),同类应用均使用此目录。 diff --git a/llamacppllmbasev3/templates/_helpers.tpl b/llamacppllmbasev3/templates/_helpers.tpl index 71f04826f..3dcee99c8 100644 --- a/llamacppllmbasev3/templates/_helpers.tpl +++ b/llamacppllmbasev3/templates/_helpers.tpl @@ -23,53 +23,3 @@ {{- $in := . -}} {{- trim ($in.Args | default "") -}} {{- end -}} -{{- /* Olares GPU mode at install: nvidia | nvidia-gb10 | amd-gpu | intel | intel-gpu. */ -}} -{{- define "llmbase.gpuType" -}} -{{- $gpuObj := .Values.GPU | default dict -}} -{{- $gpuType := .Values.gpu | default "" -}} -{{- if not $gpuType -}} -{{- $gpuType = $gpuObj.Type | default "nvidia" -}} -{{- end -}} -{{- $gpuType -}} -{{- end -}} -{{- /* llama.cpp engine image by accelerator. NVIDIA/Spark CUDA b10752; amd-gpu ROCm b10731; intel-gpu SYCL b10752. */ -}} -{{- define "llamacppllmbasev3.engineImage" -}} -{{- $gpuType := include "llmbase.gpuType" . -}} -{{- $isGb10 := or (eq $gpuType "nvidia-gb10") (eq (include "llmbase.isGb10" .) "true") -}} -{{- $img := .Values.engine.images | default dict -}} -{{- if eq $gpuType "amd-gpu" -}} -{{- $img.amdGpu | default "docker.io/beclab/ggml-org-llama.cpp:server-rocm-b10731" -}} -{{- else if eq $gpuType "intel-gpu" -}} -{{- $img.intelGpu | default "docker.io/beclab/ggml-org-llama.cpp:server-intel-b10752" -}} -{{- else if eq $gpuType "intel" -}} -{{- $img.intel | default "docker.io/beclab/ggml-org-llama.cpp:server-intel-b10884" -}} -{{- else if $isGb10 -}} -{{- $img.nvidiaGb10 | default "docker.io/beclab/ggml-org-llama.cpp:server-cuda12-b10969" -}} -{{- else -}} -{{- $img.nvidia | default "docker.io/beclab/ggml-org-llama.cpp:server-cuda12-b10969" -}} -{{- end -}} -{{- end -}} -{{- /* Spark/GB10: detect from GPU.Type or node hardware (install-time .Values.nodes). */ -}} -{{- define "llmbase.isGb10" -}} -{{- $isGb10 := "false" -}} -{{- if .Values.nodes -}} - {{- range $nodeIndex, $node := .Values.nodes -}} - {{- if eq $nodeIndex 0 -}} - {{- with $node -}} - {{- if .GPUS -}} - {{- range $gpuIndex, $gpu := .GPUS -}} - {{- if eq $gpuIndex 0 -}} - {{- with $gpu -}} - {{- if eq (upper .Model) "GB10" -}} - {{- $isGb10 = "true" -}} - {{- end -}} - {{- end -}} - {{- end -}} - {{- end -}} - {{- end -}} - {{- end -}} - {{- end -}} - {{- end -}} -{{- end -}} -{{- $isGb10 -}} -{{- end -}} diff --git a/llamacppllmbasev3/templates/llamacpp.yaml b/llamacppllmbasev3/templates/llamacpp.yaml index 141bf3f0a..f530357ca 100644 --- a/llamacppllmbasev3/templates/llamacpp.yaml +++ b/llamacppllmbasev3/templates/llamacpp.yaml @@ -13,46 +13,26 @@ {{- $memRequest := trim ($oe.LLAMACPP_MEMORY_REQUEST | default "2Gi") -}} {{- $cpuLimit := trim ($oe.LLAMACPP_CPU_LIMIT | default "6") -}} {{- $memLimit := trim ($oe.LLAMACPP_MEMORY_LIMIT | default "35Gi") -}} -{{- $gpuType := include "llmbase.gpuType" . -}} -{{- $engineImage := include "llamacppllmbasev3.engineImage" . -}} -{{- $gb10FromNodes := eq (include "llmbase.isGb10" .) "true" -}} -{{- $isAmdGpu := eq $gpuType "amd-gpu" -}} -{{- $isIntelGpu := eq $gpuType "intel-gpu" -}} -{{- $isIntel := eq $gpuType "intel" -}} -{{- $needDri := or $isAmdGpu $isIntelGpu $isIntel -}} -{{- $unifiedMem := or $gb10FromNodes (eq $gpuType "nvidia-gb10") $isIntel -}} -{{- /* nvidia.com/gpumem is NVIDIA-HAMi only: HAMi's extender exempts it from - the node capacity check, so it can carry a quota the kubelet never - advertises. The discrete AMD/Intel plugins advertise a card count and - nothing else, so their quota cannot ride on a container resource at all - and travels on a pod annotation instead, which is inert to the - scheduler. app-service reads it to resolve the requiredGPUMemory / - limitedGPUMemory "-1" sentinel this manifest declares for those modes; - without it the sentinel resolves to 0 and the card is dropped from its - own compute binding on resume. */ -}} -{{- $needNvidiaGpumem := and (not $unifiedMem) (not $isAmdGpu) (not $isIntelGpu) -}} -{{- $needGpuMemAnnotation := or $isAmdGpu $isIntelGpu -}} -{{- $shmLimit := "16Gi" -}} -{{- if or $isIntelGpu $isIntel -}} -{{- $shmLimit = "10Gi" -}} +{{- $gpuType := .Values.gpu | default "" -}} +{{- if not $gpuType -}} +{{- $gpuType = .Values.GPU.Type | default "nvidia" -}} {{- end -}} +{{- $unifiedMem := eq $gpuType "nvidia-gb10" -}} {{- $engineArgs = include "llamacppllmbasev3.engineArgs" (dict "Args" $engineArgs) -}} -{{- /* nvidia.com/gpumem needs a BARE MiB int; Spark / Intel integrated unified memory reuses the +{{- /* nvidia.com/gpumem needs a BARE MiB int; Spark unified memory reuses the same value as pod memory ("Mi"). */ -}} {{- $gpuMiB := include "llmbase.gpuMiB" ($oe.LLAMACPP_REQUIRED_GPU_MEMORY | default "4096") -}} {{- if $unifiedMem -}} {{- $memRequest = printf "%sMi" $gpuMiB -}} {{- $memLimit = printf "%sMi" $gpuMiB -}} {{- end -}} -{{- $runStateHostPath := printf "%s/llm-init-run/%s" .Values.userspace.appCache .Release.Name -}} +{{- $runStateHostPath := printf "%s/huggingface/llm-init-run/%s" .Values.userspace.appCommon .Release.Name -}} --- # llama.cpp engine Deployment. wrappers/llamacpp.sh blocks on the sentinel # (download ready), then execs llama-server -hf "$MODEL_NAME". # llm-init reverse-proxies at http://llamacpp:8081 — Service name MUST be -# "llamacpp". GPU base: CUDA (nvidia / nvidia-gb10), ROCm (amd-gpu), or -# Intel SYCL (intel / intel-gpu); put `-ngl all` in ENGINE_ARGS. -# amd-gpu / intel-gpu VRAM quota is the gpu.bytetrade.io/*-gpu-memory pod -# annotation (not nvidia.com/gpumem). +# "llamacpp". GPU base: CUDA server build + gpu-inject; put `-ngl all` in +# ENGINE_ARGS to offload all layers to the GPU. apiVersion: apps/v1 kind: Deployment metadata: @@ -64,7 +44,6 @@ metadata: io.kompose.service: llamacpp annotations: applications.app.bytetrade.io/gpu-inject: "llamacpp" - llamacppllmbasev3.io/engine-image: {{ $engineImage | quote }} spec: replicas: {{ .Values.workloads.llamacppllmbasev3.replicaCount }} selector: @@ -77,13 +56,6 @@ spec: labels: io.kompose.network/chrome-default: "true" io.kompose.service: llamacpp -{{- if $needGpuMemAnnotation }} - # VRAM quota for the modes that cannot express it as a resource. Plain - # Kubernetes quantity (unlike nvidia.com/gpumem, which is a bare MiB int). - annotations: - gpu.bytetrade.io/required-gpu-memory: "{{ $gpuMiB }}Mi" - gpu.bytetrade.io/limited-gpu-memory: "{{ $gpuMiB }}Mi" -{{- end }} spec: restartPolicy: Always volumes: @@ -99,35 +71,9 @@ spec: configMap: name: llm-init-wrappers defaultMode: 0555 -{{- if $isAmdGpu }} - - name: kfd - hostPath: - path: /dev/kfd - type: CharDevice -{{- end }} -{{- if $needDri }} - - name: dri - hostPath: - path: /dev/dri - type: Directory - - name: drm - hostPath: - path: /sys/class/drm - type: Directory - - name: dshm - emptyDir: - medium: Memory - sizeLimit: {{ $shmLimit }} -{{- end }} -{{- if $isAmdGpu }} - securityContext: - supplementalGroups: - - 44 - - 992 -{{- end }} containers: - name: llamacpp - image: "{{ $engineImage }}" + image: docker.io/beclab/ggml-org-llama.cpp:server-cuda12-b10969 imagePullPolicy: IfNotPresent command: ["/bin/sh", "/llm-init/wrappers/llamacpp.sh"] env: @@ -145,8 +91,6 @@ spec: value: "3600" - name: ENGINE_ARGS value: "{{ $engineArgs }}" - - name: OLARES_GPU_MODE - value: {{ $gpuType | quote }} - name: HF_HUB_CACHE value: /cache/hf/hub {{- if $hfEndpoint }} @@ -181,31 +125,22 @@ spec: - name: wrappers mountPath: /llm-init/wrappers readOnly: true -{{- if $isAmdGpu }} - - name: kfd - mountPath: /dev/kfd -{{- end }} -{{- if $needDri }} - - name: dri - mountPath: /dev/dri - - name: drm - mountPath: /sys/class/drm - - name: dshm - mountPath: /dev/shm -{{- end }} resources: - # nvidia.com/gpumem: nvidia only in chart; gb10 via pod memory + - # gpu-inject; amd-gpu / intel-gpu via the pod annotation above. + # nvidia.com/gpumem is the per-instance GPU memory quota read by the + # Olares gpu-inject webhook (accelerator is -1/auto, can't carry it). + # It's a non-overcommittable extended resource, so it must be in BOTH + # requests and limits with the same value. Dropped on gb10 (unified + # memory uses pod memory instead) . requests: cpu: "{{ $cpuRequest }}" memory: "{{ $memRequest }}" -{{- if $needNvidiaGpumem }} +{{- if not $unifiedMem }} nvidia.com/gpumem: {{ $gpuMiB }} {{- end }} limits: cpu: "{{ $cpuLimit }}" memory: "{{ $memLimit }}" -{{- if $needNvidiaGpumem }} +{{- if not $unifiedMem }} nvidia.com/gpumem: {{ $gpuMiB }} {{- end }} --- diff --git a/llamacppllmbasev3/templates/llm-init.yaml b/llamacppllmbasev3/templates/llm-init.yaml index 0f6165ddd..d069c644f 100644 --- a/llamacppllmbasev3/templates/llm-init.yaml +++ b/llamacppllmbasev3/templates/llm-init.yaml @@ -53,7 +53,7 @@ {{- $logLevel := $oe.LOG_LEVEL | default "debug" -}} {{- $hfEndpoint := $oe.HF_ENDPOINT | default "" -}} {{- $hfToken := $oe.HF_TOKEN | default "" -}} -{{- $runStateHostPath := printf "%s/llm-init-run/%s" .Values.userspace.appCache .Release.Name -}} +{{- $runStateHostPath := printf "%s/huggingface/llm-init-run/%s" .Values.userspace.appCommon .Release.Name -}} --- # llm-init Pod (no GPU). Downloads GGUF models, writes sentinel + model_path, # serves progress UI + OpenAI /v1/* proxy on :8090, reverse-proxies the engine @@ -109,7 +109,7 @@ spec: # v1.2.4 defaults to the stable LFS download path (hf_xet disabled # unless HF_ENABLE_XET=true); LFS streams each file to disk with a # bounded RAM footprint, avoiding the hf_xet OOM (huggingface_hub#3300). - image: docker.io/beclab/llm-init:v1.7.12 + image: docker.io/beclab/llm-init:v1.5.0 imagePullPolicy: IfNotPresent env: - name: ENGINE_KIND diff --git a/llamacppllmbasev3/values.yaml b/llamacppllmbasev3/values.yaml index ce9cbb804..c6133968b 100644 --- a/llamacppllmbasev3/values.yaml +++ b/llamacppllmbasev3/values.yaml @@ -7,13 +7,3 @@ workloads: replicaCount: 1 llamacppllmbasev3: replicaCount: 1 - -# llama.cpp engine images by Olares accelerator mode. -engine: - images: - nvidia: docker.io/beclab/ggml-org-llama.cpp:server-cuda12-b10969 - nvidiaArm64: docker.io/beclab/ggml-org-llama.cpp:server-cuda12-b10969 - nvidiaGb10: docker.io/beclab/ggml-org-llama.cpp:server-cuda12-b10969 - amdGpu: docker.io/beclab/ggml-org-llama.cpp:server-rocm-b10731 - intel: docker.io/beclab/ggml-org-llama.cpp:server-intel-b10884 - intelGpu: docker.io/beclab/ggml-org-llama.cpp:server-intel-b10752 From 47742574b955363b59328e73b93ba431a042421e Mon Sep 17 00:00:00 2001 From: Harvey Date: Tue, 15 Sep 2026 12:13:36 +0800 Subject: [PATCH 3/3] Document upstream b10588 to b10969 engine changes --- llamacppllmbasev3/OlaresManifest.yaml | 22 ++++++++++++++++++- .../i18n/en-US/OlaresManifest.yaml | 22 ++++++++++++++++++- .../i18n/zh-CN/OlaresManifest.yaml | 22 ++++++++++++++++++- 3 files changed, 63 insertions(+), 3 deletions(-) diff --git a/llamacppllmbasev3/OlaresManifest.yaml b/llamacppllmbasev3/OlaresManifest.yaml index 99e79eb4e..2e43bef8e 100644 --- a/llamacppllmbasev3/OlaresManifest.yaml +++ b/llamacppllmbasev3/OlaresManifest.yaml @@ -38,7 +38,27 @@ spec: onlyAdmin: true versionName: 'b10969' upgradeDescription: | - Chart 1.2.38 — Upgrade the llama.cpp CUDA image from server-cuda12-b10588 to server-cuda12-b10969. + Chart 1.2.38 — Upgrade llama.cpp from `server-cuda12-b10588` to `server-cuda12-b10969`. + + **Model compatibility** + - Add Qwen3.8-Flash-Next (`qwen4exp`) text/vision architecture support ([#27742](https://github.com/ggml-org/llama.cpp/pull/27742)), with subsequent recurrent-state rollback and inference fixes. + - Add NVIDIA Nemotron-3-Puzzle-75B-A9B ([#25444](https://github.com/ggml-org/llama.cpp/pull/25444)) and Tencent Hy 4 preview architecture support ([#28127](https://github.com/ggml-org/llama.cpp/pull/28127)). These require matching model weights and sufficient resources. + + **Speculative decoding and CUDA** + - Add DFlash2 draft-model support ([#27342](https://github.com/ggml-org/llama.cpp/pull/27342)); requires a compatible draft checkpoint. Upstream notes that DFlash2 GGUF files converted before August 27, 2026 need reconversion for vision correctness. + - Add CUDA sparse Flash Attention for DeepSeek-V4/GLM ([#27970](https://github.com/ggml-org/llama.cpp/pull/27970)), extend MoE fusion to speculative decoding ([#27621](https://github.com/ggml-org/llama.cpp/pull/27621)), and optimize Q4_K/Q5_K decoding with DGX Spark prefetch improvements ([#26705](https://github.com/ggml-org/llama.cpp/pull/26705)). Gains depend on model and hardware. + + **Server and chat behavior** + - Add `--kv-unified-per-slot` to cap each request slot when using a unified KV cache; `-c` controls the total pool ([#24124](https://github.com/ggml-org/llama.cpp/pull/24124)). This update does not change the configured context size automatically. + - Enable `preserve_reasoning` by default for chat templates that support it; use `--no-reasoning-preserve` to disable ([#28174](https://github.com/ggml-org/llama.cpp/pull/28174)). + - Accept `data:` URLs in `input_audio` and `input_video` requests, fixing media decoding errors ([#27735](https://github.com/ggml-org/llama.cpp/pull/27735)); requires a compatible multimodal model and client. + + **Stability and memory** + - Reduce peak RAM during model loading when tensor repacking is enabled ([#27483](https://github.com/ggml-org/llama.cpp/pull/27483)). + - Fix the model-router LRU hang with concurrent requests ([#28539](https://github.com/ggml-org/llama.cpp/pull/28539)), speculative decoding after image input ([#28715](https://github.com/ggml-org/llama.cpp/pull/28715)), and CUDA Flash Attention synchronization ([#27870](https://github.com/ggml-org/llama.cpp/pull/27870)). + + These are upstream engine changes; model-specific capabilities are not all validated by this chart update. llm-init remains at `v1.5.0`. + Full upstream changes: https://github.com/ggml-org/llama.cpp/compare/b10588...b10969 fullDescription: | **IMPORTANT NOTE** diff --git a/llamacppllmbasev3/i18n/en-US/OlaresManifest.yaml b/llamacppllmbasev3/i18n/en-US/OlaresManifest.yaml index cf78b236b..6dac468a5 100644 --- a/llamacppllmbasev3/i18n/en-US/OlaresManifest.yaml +++ b/llamacppllmbasev3/i18n/en-US/OlaresManifest.yaml @@ -4,7 +4,27 @@ metadata: spec: upgradeDescription: | - Chart 1.2.38 — Upgrade the llama.cpp CUDA image from server-cuda12-b10588 to server-cuda12-b10969. + Chart 1.2.38 — Upgrade llama.cpp from `server-cuda12-b10588` to `server-cuda12-b10969`. + + **Model compatibility** + - Add Qwen3.8-Flash-Next (`qwen4exp`) text/vision architecture support ([#27742](https://github.com/ggml-org/llama.cpp/pull/27742)), with subsequent recurrent-state rollback and inference fixes. + - Add NVIDIA Nemotron-3-Puzzle-75B-A9B ([#25444](https://github.com/ggml-org/llama.cpp/pull/25444)) and Tencent Hy 4 preview architecture support ([#28127](https://github.com/ggml-org/llama.cpp/pull/28127)). These require matching model weights and sufficient resources. + + **Speculative decoding and CUDA** + - Add DFlash2 draft-model support ([#27342](https://github.com/ggml-org/llama.cpp/pull/27342)); requires a compatible draft checkpoint. Upstream notes that DFlash2 GGUF files converted before August 27, 2026 need reconversion for vision correctness. + - Add CUDA sparse Flash Attention for DeepSeek-V4/GLM ([#27970](https://github.com/ggml-org/llama.cpp/pull/27970)), extend MoE fusion to speculative decoding ([#27621](https://github.com/ggml-org/llama.cpp/pull/27621)), and optimize Q4_K/Q5_K decoding with DGX Spark prefetch improvements ([#26705](https://github.com/ggml-org/llama.cpp/pull/26705)). Gains depend on model and hardware. + + **Server and chat behavior** + - Add `--kv-unified-per-slot` to cap each request slot when using a unified KV cache; `-c` controls the total pool ([#24124](https://github.com/ggml-org/llama.cpp/pull/24124)). This update does not change the configured context size automatically. + - Enable `preserve_reasoning` by default for chat templates that support it; use `--no-reasoning-preserve` to disable ([#28174](https://github.com/ggml-org/llama.cpp/pull/28174)). + - Accept `data:` URLs in `input_audio` and `input_video` requests, fixing media decoding errors ([#27735](https://github.com/ggml-org/llama.cpp/pull/27735)); requires a compatible multimodal model and client. + + **Stability and memory** + - Reduce peak RAM during model loading when tensor repacking is enabled ([#27483](https://github.com/ggml-org/llama.cpp/pull/27483)). + - Fix the model-router LRU hang with concurrent requests ([#28539](https://github.com/ggml-org/llama.cpp/pull/28539)), speculative decoding after image input ([#28715](https://github.com/ggml-org/llama.cpp/pull/28715)), and CUDA Flash Attention synchronization ([#27870](https://github.com/ggml-org/llama.cpp/pull/27870)). + + These are upstream engine changes; model-specific capabilities are not all validated by this chart update. llm-init remains at `v1.5.0`. + Full upstream changes: https://github.com/ggml-org/llama.cpp/compare/b10588...b10969 fullDescription: | **IMPORTANT NOTE** diff --git a/llamacppllmbasev3/i18n/zh-CN/OlaresManifest.yaml b/llamacppllmbasev3/i18n/zh-CN/OlaresManifest.yaml index 752f2bb82..b6eaeaab1 100644 --- a/llamacppllmbasev3/i18n/zh-CN/OlaresManifest.yaml +++ b/llamacppllmbasev3/i18n/zh-CN/OlaresManifest.yaml @@ -4,7 +4,27 @@ metadata: spec: upgradeDescription: | - Chart 1.2.38 — llama.cpp CUDA 镜像从 server-cuda12-b10588 升级到 server-cuda12-b10969。 + Chart 1.2.38 — llama.cpp 镜像从 `server-cuda12-b10588` 升级到 `server-cuda12-b10969`。 + + **模型兼容性** + - 新增 Qwen3.8-Flash-Next(`qwen4exp`)文本与视觉架构支持([#27742](https://github.com/ggml-org/llama.cpp/pull/27742)),并包含后续循环状态回滚和推理修复。 + - 新增 NVIDIA Nemotron-3-Puzzle-75B-A9B([#25444](https://github.com/ggml-org/llama.cpp/pull/25444))和 Tencent Hy 4 预览架构支持([#28127](https://github.com/ggml-org/llama.cpp/pull/28127));使用时需准备匹配的模型权重和足够的资源。 + + **推测解码与 CUDA** + - 新增 DFlash2 草稿模型支持([#27342](https://github.com/ggml-org/llama.cpp/pull/27342)),需配置兼容的草稿权重。上游提示:2026 年 8 月 27 日前转换的 DFlash2 GGUF 文件需重新转换,以保证视觉功能正确。 + - 新增 DeepSeek-V4/GLM 的 CUDA 稀疏 Flash Attention([#27970](https://github.com/ggml-org/llama.cpp/pull/27970)),将 MoE 融合优化扩展到推测解码([#27621](https://github.com/ggml-org/llama.cpp/pull/27621)),优化 Q4_K/Q5_K 解码和 DGX Spark 预取([#26705](https://github.com/ggml-org/llama.cpp/pull/26705))。实际收益取决于模型与硬件。 + + **服务接口与对话行为** + - 新增 `--kv-unified-per-slot`,可在统一 KV 缓存模式下限制每个请求槽的上下文,`-c` 控制总缓存池([#24124](https://github.com/ggml-org/llama.cpp/pull/24124))。本次升级不会自动修改实例的上下文配置。 + - 对支持的聊天模板,默认启用 `preserve_reasoning`;可通过 `--no-reasoning-preserve` 关闭([#28174](https://github.com/ggml-org/llama.cpp/pull/28174))。 + - `input_audio`、`input_video` 请求支持 `data:` URL,修复媒体解码错误([#27735](https://github.com/ggml-org/llama.cpp/pull/27735));需使用兼容的多模态模型与客户端。 + + **稳定性与内存** + - 降低启用张量重排时模型加载阶段的内存峰值([#27483](https://github.com/ggml-org/llama.cpp/pull/27483))。 + - 修复模型路由器并发请求的 LRU 卡住问题([#28539](https://github.com/ggml-org/llama.cpp/pull/28539))、图片输入后的推测解码([#28715](https://github.com/ggml-org/llama.cpp/pull/28715))和 CUDA Flash Attention 同步问题([#27870](https://github.com/ggml-org/llama.cpp/pull/27870))。 + + 以上为上游引擎变化,各模型专属能力尚未全部由本次 Chart 升级验证。llm-init 保持 `v1.5.0`。 + 完整上游变更:https://github.com/ggml-org/llama.cpp/compare/b10588...b10969 fullDescription: | **重要提示**