From 23e60f73f6bdfab9498822bb18a403e65de0b50a Mon Sep 17 00:00:00 2001 From: Ivan Despot <66276597+g-despot@users.noreply.github.com> Date: Sun, 19 Jul 2026 21:46:32 +0200 Subject: [PATCH 1/5] docs: correct deprecated ChunkSize and document backup chunking variables ChunkSize has had no effect since its removal in core, but the page documented it as a working option with a default, minimum and maximum. Mark it deprecated and point to BACKUP_CHUNK_TARGET_SIZE. Document BACKUP_MIN_CHUNK_SIZE, BACKUP_CHUNK_TARGET_SIZE and BACKUP_SPLIT_FILE_SIZE, which were undocumented, including how their values are raised against each other. Document the location parameter for text2vec-google. No version markers: these all landed across several stable lines, so no single version is correct for every supported release. --- docs/deploy/configuration/backups.md | 24 ++++++++++++++++++- .../model-providers/google/embeddings.md | 3 +++ 2 files changed, 26 insertions(+), 1 deletion(-) diff --git a/docs/deploy/configuration/backups.md b/docs/deploy/configuration/backups.md index de8ac6b1..5b9eddd9 100644 --- a/docs/deploy/configuration/backups.md +++ b/docs/deploy/configuration/backups.md @@ -316,7 +316,7 @@ The `*` character matches any sequence of characters. For example, `Article*` ma | name | type | required | default | description | | ---- | ---- | ---- | ---- |---- | | `CPUPercentage` | number | no | `50%` | An optional integer to set the desired CPU core utilization ranging from 1%-80%. | -| `ChunkSize` | number | no | `128MB` | An optional integer represents the desired size for chunks. Weaviate will attempt to come close the specified size, with a minimum of 2MB, default of 128MB, and a maximum of 512MB.| +| `ChunkSize` | number | no | - | **Deprecated. This option has no effect.** Weaviate ignores any value sent here, so it neither sets nor caps the chunk size. Chunk sizing is now controlled by the [`BACKUP_CHUNK_TARGET_SIZE`](#how-it-works) environment variable, which replaced it. | | `CompressionLevel`| string | no | `DefaultCompression` | An optional [compression level](#compression-levels) to be used. | | `Path` | string | no | `""` | An optional string to manually set the backup location. If not provided, the backup will be stored in the default location. Introduced in Weaviate `v1.27.2`. | | `incremental_base_backup_id` | string | no | `None` | The ID of a previous backup to use as the base for an [incremental backup](#incremental-backups). Files unchanged since the base backup are stored as references rather than copied. Introduced in Weaviate `v1.37`. | @@ -468,6 +468,28 @@ This can result in dramatically smaller backups and much faster backup times. When creating a backup, Weaviate splits large files into individual chunks. During an incremental backup, Weaviate compares each file against the base backup. Files that haven't changed are stored as pointers to the base backup rather than being copied again. On restore, Weaviate automatically fetches the referenced files from the base backup. +Only files that are large enough to get a chunk of their own can be referenced individually. Smaller files are packed together into shared chunks, so a change to any one of them means the whole chunk is written again. Three environment variables control the packing: `BACKUP_MIN_CHUNK_SIZE` sets which files count as large, `BACKUP_CHUNK_TARGET_SIZE` sets how much data is packed into one shared chunk, and `BACKUP_SPLIT_FILE_SIZE` sets when a single very large file is spread across several chunks. + +Lowering `BACKUP_MIN_CHUNK_SIZE` allows more files to be referenced individually, at the cost of more chunks per backup. + +The following environment variables control chunking. All three accept a plain number of bytes or a number with a unit suffix (`B`, `KB`, `MB`, `GB`, `TB`, `KiB`, `MiB`, `GiB`, `TiB`), for example `4MiB`, and all three are applied at startup, so a restart is required to change them. + +| Environment variable | Required | Description | +| --- | --- | --- | +| `BACKUP_MIN_CHUNK_SIZE` | no | The minimum size a file must reach before it is stored in its own chunk and can be referenced individually by a later incremental backup. Defaults to `1MiB`. | +| `BACKUP_CHUNK_TARGET_SIZE` | no | The size Weaviate aims for when packing several small files into a single chunk. Defaults to `10MiB`. | +| `BACKUP_SPLIT_FILE_SIZE` | no | The size above which a single file is split across multiple chunks instead of being written to one. Defaults to `50GiB`. | + +:::note Minimum values are raised automatically + +These three settings are lower bounds, not exact values, and Weaviate raises them when a smaller value would have no useful effect: + +- The threshold for treating a file as large is the larger of `BACKUP_MIN_CHUNK_SIZE` and the size of the 100th largest file in the shard, so roughly the 100 largest files in a shard can get their own chunk, and lowering `BACKUP_MIN_CHUNK_SIZE` below that point has no further effect. +- `BACKUP_CHUNK_TARGET_SIZE` is raised to that threshold if you set it lower, since a chunk that packs small files must still be able to hold one large file. +- `BACKUP_SPLIT_FILE_SIZE` is likewise raised to that threshold if you set it lower; otherwise no file would ever be large enough to split. + +::: + #### Create a full (base) backup First, create a regular backup that will serve as the base: diff --git a/docs/weaviate/model-providers/google/embeddings.md b/docs/weaviate/model-providers/google/embeddings.md index 802b5d0d..1b84e150 100644 --- a/docs/weaviate/model-providers/google/embeddings.md +++ b/docs/weaviate/model-providers/google/embeddings.md @@ -234,9 +234,12 @@ The following examples show how to configure Google-specific options. **Vertex AI parameters:** - `projectId` (Required): Your Google Cloud project ID, e.g. `cloud-large-language-models` +- `location` (Optional): The Google Cloud region to send requests to, e.g. `europe-west1`. Defaults to `us-central1`. - `apiEndpoint` (Optional): Regional endpoint, e.g. `us-central1-aiplatform.googleapis.com` - `modelId` (Optional): e.g. `gemini-embedding-001`, `text-embedding-005` +Set `location` together with a matching `apiEndpoint` to keep data in a specific region. + Date: Thu, 30 Jul 2026 14:11:47 +0200 Subject: [PATCH 2/5] Remove default region --- docs/weaviate/model-providers/google/embeddings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/weaviate/model-providers/google/embeddings.md b/docs/weaviate/model-providers/google/embeddings.md index 1b84e150..be5cb647 100644 --- a/docs/weaviate/model-providers/google/embeddings.md +++ b/docs/weaviate/model-providers/google/embeddings.md @@ -234,7 +234,7 @@ The following examples show how to configure Google-specific options. **Vertex AI parameters:** - `projectId` (Required): Your Google Cloud project ID, e.g. `cloud-large-language-models` -- `location` (Optional): The Google Cloud region to send requests to, e.g. `europe-west1`. Defaults to `us-central1`. +- `location` (Optional): The Google Cloud region to send requests to, e.g. `europe-west1`. - `apiEndpoint` (Optional): Regional endpoint, e.g. `us-central1-aiplatform.googleapis.com` - `modelId` (Optional): e.g. `gemini-embedding-001`, `text-embedding-005` From 557dfb27280479d80c44ea31499facb4092036a4 Mon Sep 17 00:00:00 2001 From: Ivan Despot <66276597+g-despot@users.noreply.github.com> Date: Thu, 30 Jul 2026 14:49:34 +0200 Subject: [PATCH 3/5] docs(backups): move chunking into Technical Considerations, add BACKUP_MAX_INDIVIDUAL_FILES Chunking runs on every backup, not only incremental ones, so the block no longer belongs under Incremental Backups > How it works. Move it to a new Technical Considerations > Chunking and file splitting section, modelled on Skip the storage access check. - Document BACKUP_MAX_INDIVIDUAL_FILES (default 100, added in v1.37.14 and v1.38.7, absent from 1.36) and note it is runtime-configurable via the backup_max_individual_files override key, unlike the three size variables which are startup-only. - Reframe BACKUP_MIN_CHUNK_SIZE as a floor: lowering it has no effect once at least BACKUP_MAX_INDIVIDUAL_FILES files already exceed it. - Record that chunks carrying a split-file part are not topped up, so large-file backups produce chunks smaller than BACKUP_CHUNK_TARGET_SIZE. - Note that unlimited/nolimit are accepted, which is how splitting is disabled. - Note that an incremental backup shrinks the individual-file budget by the number of files reused from the base, so it spans a backup chain. - Repoint the deprecated ChunkSize row at the new section anchor. - Add backup_max_individual_files to the runtime overrides table. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01VLMB1FhdGFGcdg63uVHxNY --- docs/deploy/configuration/backups.md | 55 +++++++++++-------- .../configuration/env-vars/runtime-config.md | 1 + 2 files changed, 34 insertions(+), 22 deletions(-) diff --git a/docs/deploy/configuration/backups.md b/docs/deploy/configuration/backups.md index 5b9eddd9..18161a1d 100644 --- a/docs/deploy/configuration/backups.md +++ b/docs/deploy/configuration/backups.md @@ -316,7 +316,7 @@ The `*` character matches any sequence of characters. For example, `Article*` ma | name | type | required | default | description | | ---- | ---- | ---- | ---- |---- | | `CPUPercentage` | number | no | `50%` | An optional integer to set the desired CPU core utilization ranging from 1%-80%. | -| `ChunkSize` | number | no | - | **Deprecated. This option has no effect.** Weaviate ignores any value sent here, so it neither sets nor caps the chunk size. Chunk sizing is now controlled by the [`BACKUP_CHUNK_TARGET_SIZE`](#how-it-works) environment variable, which replaced it. | +| `ChunkSize` | number | no | - | **Deprecated. This option has no effect.** Weaviate ignores any value sent here, so it neither sets nor caps the chunk size. Chunk sizing is now controlled by the [`BACKUP_CHUNK_TARGET_SIZE`](#chunking-and-file-splitting) environment variable, which replaced it. | | `CompressionLevel`| string | no | `DefaultCompression` | An optional [compression level](#compression-levels) to be used. | | `Path` | string | no | `""` | An optional string to manually set the backup location. If not provided, the backup will be stored in the default location. Introduced in Weaviate `v1.27.2`. | | `incremental_base_backup_id` | string | no | `None` | The ID of a previous backup to use as the base for an [incremental backup](#incremental-backups). Files unchanged since the base backup are stored as references rather than copied. Introduced in Weaviate `v1.37`. | @@ -468,27 +468,7 @@ This can result in dramatically smaller backups and much faster backup times. When creating a backup, Weaviate splits large files into individual chunks. During an incremental backup, Weaviate compares each file against the base backup. Files that haven't changed are stored as pointers to the base backup rather than being copied again. On restore, Weaviate automatically fetches the referenced files from the base backup. -Only files that are large enough to get a chunk of their own can be referenced individually. Smaller files are packed together into shared chunks, so a change to any one of them means the whole chunk is written again. Three environment variables control the packing: `BACKUP_MIN_CHUNK_SIZE` sets which files count as large, `BACKUP_CHUNK_TARGET_SIZE` sets how much data is packed into one shared chunk, and `BACKUP_SPLIT_FILE_SIZE` sets when a single very large file is spread across several chunks. - -Lowering `BACKUP_MIN_CHUNK_SIZE` allows more files to be referenced individually, at the cost of more chunks per backup. - -The following environment variables control chunking. All three accept a plain number of bytes or a number with a unit suffix (`B`, `KB`, `MB`, `GB`, `TB`, `KiB`, `MiB`, `GiB`, `TiB`), for example `4MiB`, and all three are applied at startup, so a restart is required to change them. - -| Environment variable | Required | Description | -| --- | --- | --- | -| `BACKUP_MIN_CHUNK_SIZE` | no | The minimum size a file must reach before it is stored in its own chunk and can be referenced individually by a later incremental backup. Defaults to `1MiB`. | -| `BACKUP_CHUNK_TARGET_SIZE` | no | The size Weaviate aims for when packing several small files into a single chunk. Defaults to `10MiB`. | -| `BACKUP_SPLIT_FILE_SIZE` | no | The size above which a single file is split across multiple chunks instead of being written to one. Defaults to `50GiB`. | - -:::note Minimum values are raised automatically - -These three settings are lower bounds, not exact values, and Weaviate raises them when a smaller value would have no useful effect: - -- The threshold for treating a file as large is the larger of `BACKUP_MIN_CHUNK_SIZE` and the size of the 100th largest file in the shard, so roughly the 100 largest files in a shard can get their own chunk, and lowering `BACKUP_MIN_CHUNK_SIZE` below that point has no further effect. -- `BACKUP_CHUNK_TARGET_SIZE` is raised to that threshold if you set it lower, since a chunk that packs small files must still be able to hold one large file. -- `BACKUP_SPLIT_FILE_SIZE` is likewise raised to that threshold if you set it lower; otherwise no file would ever be large enough to split. - -::: +Only files that are large enough to get a chunk of their own can be referenced individually. Smaller files are packed together into shared chunks, so a change to any one of them means the whole chunk is written again. How Weaviate decides which files get their own chunk, and the environment variables that control it, are described in [Chunking and file splitting](#chunking-and-file-splitting). #### Create a full (base) backup @@ -800,6 +780,37 @@ The backup API is built in a way that no long-running network requests are requi If you would like your application to wait for the background backup process to complete, you can use the "wait for completion" feature that is present in all language clients. The clients will poll the status endpoint in the background and block until the status is either `SUCCESS` or `FAILED`. This makes it easy to write simple synchronous backup scripts, even with the async nature of the API. +### Chunking and file splitting + +Weaviate does not upload a shard's files one by one. It packs them into chunks, and the way files are grouped into chunks determines how much of a backup a later [incremental backup](#incremental-backups) can reuse. Chunking runs on every backup, not only on incremental ones. + +Weaviate gives each of a shard's biggest files a chunk of its own, and only a file that gets its own chunk can be referenced individually by a later incremental backup. All remaining files are packed together into shared chunks, so changing any one of them means the whole chunk is written again. A file above the split threshold is spread across several chunks instead of being written into one. + +The size a file must reach to get its own chunk is the larger of `BACKUP_MIN_CHUNK_SIZE` and the size of the Nth largest file in the shard, where N is `BACKUP_MAX_INDIVIDUAL_FILES`. `BACKUP_MIN_CHUNK_SIZE` is therefore a floor on that size rather than a dial. Lowering it never reduces how many files get their own chunk, but it only increases that number while the shard holds fewer than `BACKUP_MAX_INDIVIDUAL_FILES` files above its current value. At the defaults, a shard with 100 or more files of at least `1MiB` is unaffected by a lower `BACKUP_MIN_CHUNK_SIZE`, because the size of the 100th largest file wins the comparison. To have more files referenced individually in that case, raise `BACKUP_MAX_INDIVIDUAL_FILES` instead, at the cost of more chunks per backup. + +A chunk that carries part of a split file holds nothing else, because Weaviate deliberately leaves the rest of that chunk empty rather than mixing split parts with regular files. Backups dominated by very large files therefore produce chunks noticeably smaller than `BACKUP_CHUNK_TARGET_SIZE`. + +The three size variables below accept a plain number of bytes or a number with a unit suffix (`B`, `KB`, `MB`, `GB`, `TB`, `KiB`, `MiB`, `GiB`, `TiB`), for example `4MiB`. The decimal and binary suffixes are distinct: `MB` is 1,000,000 bytes while `MiB` is 1,048,576 bytes. They also accept `unlimited` or `nolimit`, which is how you disable file splitting through `BACKUP_SPLIT_FILE_SIZE`. All three are read at startup, so a restart is required to change them. `BACKUP_MAX_INDIVIDUAL_FILES` is a plain count instead of a size, and it is read for each backup, so it can also be changed without a restart. + +| Environment variable | Required | Description | +| --- | --- | --- | +| `BACKUP_MIN_CHUNK_SIZE` | no | The floor on the size a file must reach before it is stored in its own chunk and can be referenced individually by a later incremental backup. Defaults to `1MiB`. | +| `BACKUP_CHUNK_TARGET_SIZE` | no | The size Weaviate aims for when packing several smaller files into a single chunk. Defaults to `10MiB`. | +| `BACKUP_SPLIT_FILE_SIZE` | no | The size above which a single file is split across multiple chunks instead of being written to one. Set it to `unlimited` to disable splitting. Defaults to `50GiB`. | +| `BACKUP_MAX_INDIVIDUAL_FILES` | no | How many of a shard's biggest files are targeted to get their own chunk, and can therefore be referenced individually by a later incremental backup. A count rather than a size, and it must be greater than `0`. Defaults to `100`. Also settable without a restart through the `backup_max_individual_files` [runtime configuration](./env-vars/runtime-config.md) key.

Added in `v1.37.14` and `v1.38.7`. Not available in `v1.36`. | + +:::note Values are raised automatically + +These settings are lower bounds, not exact values, and Weaviate raises them when a smaller value would have no useful effect: + +- `BACKUP_CHUNK_TARGET_SIZE` is raised to the size that qualifies a file for its own chunk if you set it lower, since a chunk that packs smaller files must still be able to hold one big file. +- `BACKUP_SPLIT_FILE_SIZE` is likewise raised to that size if you set it lower; otherwise no file would ever be big enough to split. +- If a shard holds fewer files than `BACKUP_MAX_INDIVIDUAL_FILES`, the qualifying size falls back to the size of the shard's smallest file, still raised to `BACKUP_MIN_CHUNK_SIZE` if that is larger. + +::: + +On an incremental backup, the `BACKUP_MAX_INDIVIDUAL_FILES` budget is reduced by the number of files that are reused from the base backup, so it is shared across a whole backup chain rather than being renewed for each backup in it. + ### Skip the storage access check When a cloud backup backend (`backup-s3`, `backup-gcs`, or `backup-azure`) initializes, Weaviate verifies that the configured credentials can write to and delete from the target bucket. It does this by writing a temporary `access-check` object and then removing it. This probe fails on immutable (write-once / WORM) buckets, or with least-privilege credentials that are not permitted to delete objects. diff --git a/docs/deploy/configuration/env-vars/runtime-config.md b/docs/deploy/configuration/env-vars/runtime-config.md index 137ead48..a44ea2c8 100644 --- a/docs/deploy/configuration/env-vars/runtime-config.md +++ b/docs/deploy/configuration/env-vars/runtime-config.md @@ -61,6 +61,7 @@ The following overrides are currently supported: | `async_replication_hashtree_init_concurrency` | `ASYNC_REPLICATION_HASHTREE_INIT_CONCURRENCY`| | `async_replication_cluster_max_workers` _(removed in `v1.38`)_ | `ASYNC_REPLICATION_CLUSTER_MAX_WORKERS` _(removed in `v1.38`)_ | | `autoschema_enabled` | `AUTOSCHEMA_ENABLED` | +| `backup_max_individual_files` | `BACKUP_MAX_INDIVIDUAL_FILES` | | `debug_endpoints_enabled` | `DEBUG_ENDPOINTS_ENABLED` | | `default_quantization` | `DEFAULT_QUANTIZATION` | | `default_sharding_count` | `DEFAULT_SHARDING_COUNT` | From 04b56ae5b429ba4ff8d5d911216888bc90778cc7 Mon Sep 17 00:00:00 2001 From: Ivan Despot <66276597+g-despot@users.noreply.github.com> Date: Thu, 30 Jul 2026 15:54:21 +0200 Subject: [PATCH 4/5] docs(backups): fix split-chunk size claim, apply editor and verifier feedback Correct one factual error and apply the review gates' required changes to the Chunking and file splitting section. - Split-file chunks: their size derives from BACKUP_SPLIT_FILE_SIZE, not from BACKUP_CHUNK_TARGET_SIZE. Each part is larger than half the split size and no larger than it, so at the defaults these chunks are far larger than the target, not smaller as previously stated. - Shared chunks are re-uploaded on every incremental backup regardless of whether their contents changed, because only single big-file chunks can be skipped. - Note that getting an own chunk is necessary but not sufficient for reuse: Weaviate only reuses files it treats as immutable. - Standardize on "qualifying size" for the size threshold, name BACKUP_SPLIT_FILE_SIZE where the split threshold is described, and split the floor-vs-dial paragraph so the definition stands on its own. - State that BACKUP_MAX_INDIVIDUAL_FILES is changed without a restart through the backup_max_individual_files runtime configuration key. - Move the backup-chain budget paragraph above the admonition so the section ends on it, and note that the shard-file-count fallback compares against the reduced budget on an incremental backup. - Link shard on first mention, retitle the admonition to cover the fallback bullet, drop the negative version marker, and rewrite the How it works remnant so it no longer conflicts with or duplicates the new section. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01VLMB1FhdGFGcdg63uVHxNY --- docs/deploy/configuration/backups.md | 30 +++++++++++++++------------- 1 file changed, 16 insertions(+), 14 deletions(-) diff --git a/docs/deploy/configuration/backups.md b/docs/deploy/configuration/backups.md index 18161a1d..c85ec2b5 100644 --- a/docs/deploy/configuration/backups.md +++ b/docs/deploy/configuration/backups.md @@ -466,9 +466,9 @@ This can result in dramatically smaller backups and much faster backup times. #### How it works -When creating a backup, Weaviate splits large files into individual chunks. During an incremental backup, Weaviate compares each file against the base backup. Files that haven't changed are stored as pointers to the base backup rather than being copied again. On restore, Weaviate automatically fetches the referenced files from the base backup. +When creating a backup, Weaviate packs a shard's files into chunks. During an incremental backup, Weaviate compares each file against the base backup. Files that haven't changed are stored as pointers to the base backup rather than being copied again. On restore, Weaviate automatically fetches the referenced files from the base backup. -Only files that are large enough to get a chunk of their own can be referenced individually. Smaller files are packed together into shared chunks, so a change to any one of them means the whole chunk is written again. How Weaviate decides which files get their own chunk, and the environment variables that control it, are described in [Chunking and file splitting](#chunking-and-file-splitting). +Only a file large enough to get a chunk of its own can be referenced individually, so the way Weaviate groups files into chunks determines how much an incremental backup can reuse. For how Weaviate decides which files get their own chunk, and the environment variables that control chunking, see [Chunking and file splitting](#chunking-and-file-splitting). #### Create a full (base) backup @@ -782,35 +782,37 @@ If you would like your application to wait for the background backup process to ### Chunking and file splitting -Weaviate does not upload a shard's files one by one. It packs them into chunks, and the way files are grouped into chunks determines how much of a backup a later [incremental backup](#incremental-backups) can reuse. Chunking runs on every backup, not only on incremental ones. +Weaviate does not upload a [shard's](/weaviate/concepts/storage.md#logical-storage-units-indexes-shards-stores) files one by one. It packs them into chunks, and the way files are grouped into chunks determines how much of that backup a later [incremental backup](#incremental-backups) can reuse. Chunking runs on every backup, not only on incremental ones. -Weaviate gives each of a shard's biggest files a chunk of its own, and only a file that gets its own chunk can be referenced individually by a later incremental backup. All remaining files are packed together into shared chunks, so changing any one of them means the whole chunk is written again. A file above the split threshold is spread across several chunks instead of being written into one. +Weaviate gives each of a shard's biggest files a chunk of its own, and only a file that gets its own chunk can be referenced individually by a later incremental backup. A chunk of its own is a requirement rather than a guarantee, because Weaviate only reuses files that it treats as immutable, so a large file that it keeps rewriting is uploaded again with every backup. All remaining files are packed together into shared chunks, and a shared chunk is uploaded again on every incremental backup whether or not anything in it changed, because only a chunk that holds a single big file can be skipped. A file larger than `BACKUP_SPLIT_FILE_SIZE` is spread across several chunks instead of being written into one. -The size a file must reach to get its own chunk is the larger of `BACKUP_MIN_CHUNK_SIZE` and the size of the Nth largest file in the shard, where N is `BACKUP_MAX_INDIVIDUAL_FILES`. `BACKUP_MIN_CHUNK_SIZE` is therefore a floor on that size rather than a dial. Lowering it never reduces how many files get their own chunk, but it only increases that number while the shard holds fewer than `BACKUP_MAX_INDIVIDUAL_FILES` files above its current value. At the defaults, a shard with 100 or more files of at least `1MiB` is unaffected by a lower `BACKUP_MIN_CHUNK_SIZE`, because the size of the 100th largest file wins the comparison. To have more files referenced individually in that case, raise `BACKUP_MAX_INDIVIDUAL_FILES` instead, at the cost of more chunks per backup. +A file gets its own chunk once it reaches the qualifying size: the larger of `BACKUP_MIN_CHUNK_SIZE` and the size of the Nth largest file in the shard, where N is `BACKUP_MAX_INDIVIDUAL_FILES`. -A chunk that carries part of a split file holds nothing else, because Weaviate deliberately leaves the rest of that chunk empty rather than mixing split parts with regular files. Backups dominated by very large files therefore produce chunks noticeably smaller than `BACKUP_CHUNK_TARGET_SIZE`. +Because the larger of the two wins, `BACKUP_MIN_CHUNK_SIZE` is a floor on the qualifying size rather than a dial for it. Lowering it never reduces how many files get their own chunk, and it only increases that number while the shard holds fewer than `BACKUP_MAX_INDIVIDUAL_FILES` files above its current value. At the defaults, a shard with 100 or more files of at least `1MiB` is unaffected by a lower `BACKUP_MIN_CHUNK_SIZE`, because the size of the 100th largest file wins the comparison. To have more files referenced individually in that case, raise `BACKUP_MAX_INDIVIDUAL_FILES` instead, at the cost of more chunks per backup. -The three size variables below accept a plain number of bytes or a number with a unit suffix (`B`, `KB`, `MB`, `GB`, `TB`, `KiB`, `MiB`, `GiB`, `TiB`), for example `4MiB`. The decimal and binary suffixes are distinct: `MB` is 1,000,000 bytes while `MiB` is 1,048,576 bytes. They also accept `unlimited` or `nolimit`, which is how you disable file splitting through `BACKUP_SPLIT_FILE_SIZE`. All three are read at startup, so a restart is required to change them. `BACKUP_MAX_INDIVIDUAL_FILES` is a plain count instead of a size, and it is read for each backup, so it can also be changed without a restart. +On an incremental backup, the `BACKUP_MAX_INDIVIDUAL_FILES` budget is reduced by the number of files that are reused from the base backup, so it is shared across a whole backup chain rather than being renewed for each backup in it. + +A chunk that carries part of a split file holds nothing else, because Weaviate deliberately leaves the rest of that chunk empty rather than mixing split parts with regular files. The size of such a chunk is governed by `BACKUP_SPLIT_FILE_SIZE` rather than by `BACKUP_CHUNK_TARGET_SIZE`: Weaviate divides the file into equal parts that are each larger than half of the split size and no larger than the split size, so at the defaults these chunks are far larger than the target, not smaller. + +The three size variables below accept a plain number of bytes or a number with a unit suffix (`B`, `KB`, `MB`, `GB`, `TB`, `KiB`, `MiB`, `GiB`, `TiB`), for example `4MiB`. The decimal and binary suffixes are distinct: `MB` is 1,000,000 bytes while `MiB` is 1,048,576 bytes. They also accept `unlimited` or `nolimit`, which is how you disable file splitting through `BACKUP_SPLIT_FILE_SIZE`. All three are read at startup, so a restart is required to change them. `BACKUP_MAX_INDIVIDUAL_FILES` is a plain count instead of a size, and Weaviate reads it for each backup, so you can change it without a restart through the `backup_max_individual_files` [runtime configuration](./env-vars/runtime-config.md) key. | Environment variable | Required | Description | | --- | --- | --- | -| `BACKUP_MIN_CHUNK_SIZE` | no | The floor on the size a file must reach before it is stored in its own chunk and can be referenced individually by a later incremental backup. Defaults to `1MiB`. | +| `BACKUP_MIN_CHUNK_SIZE` | no | The floor on the qualifying size a file must reach before it is stored in its own chunk and becomes eligible to be referenced individually by a later incremental backup. Defaults to `1MiB`. | | `BACKUP_CHUNK_TARGET_SIZE` | no | The size Weaviate aims for when packing several smaller files into a single chunk. Defaults to `10MiB`. | | `BACKUP_SPLIT_FILE_SIZE` | no | The size above which a single file is split across multiple chunks instead of being written to one. Set it to `unlimited` to disable splitting. Defaults to `50GiB`. | -| `BACKUP_MAX_INDIVIDUAL_FILES` | no | How many of a shard's biggest files are targeted to get their own chunk, and can therefore be referenced individually by a later incremental backup. A count rather than a size, and it must be greater than `0`. Defaults to `100`. Also settable without a restart through the `backup_max_individual_files` [runtime configuration](./env-vars/runtime-config.md) key.

Added in `v1.37.14` and `v1.38.7`. Not available in `v1.36`. | +| `BACKUP_MAX_INDIVIDUAL_FILES` | no | How many of a shard's biggest files Weaviate aims to give a chunk of their own, and therefore how many become eligible to be referenced individually by a later incremental backup. This is a count rather than a size, and it must be greater than `0`. Defaults to `100`. Also settable without a restart through the `backup_max_individual_files` [runtime configuration](./env-vars/runtime-config.md) key.

Added in `v1.37.14` and `v1.38.7`. | -:::note Values are raised automatically +:::note How Weaviate adjusts these values These settings are lower bounds, not exact values, and Weaviate raises them when a smaller value would have no useful effect: -- `BACKUP_CHUNK_TARGET_SIZE` is raised to the size that qualifies a file for its own chunk if you set it lower, since a chunk that packs smaller files must still be able to hold one big file. +- `BACKUP_CHUNK_TARGET_SIZE` is raised to the qualifying size if you set it lower, since a chunk that packs smaller files must still be able to hold one big file. - `BACKUP_SPLIT_FILE_SIZE` is likewise raised to that size if you set it lower; otherwise no file would ever be big enough to split. -- If a shard holds fewer files than `BACKUP_MAX_INDIVIDUAL_FILES`, the qualifying size falls back to the size of the shard's smallest file, still raised to `BACKUP_MIN_CHUNK_SIZE` if that is larger. +- If a shard holds fewer files than `BACKUP_MAX_INDIVIDUAL_FILES`, or than the reduced budget on an incremental backup, the qualifying size falls back to the size of the shard's smallest file, still raised to `BACKUP_MIN_CHUNK_SIZE` if that is larger. ::: -On an incremental backup, the `BACKUP_MAX_INDIVIDUAL_FILES` budget is reduced by the number of files that are reused from the base backup, so it is shared across a whole backup chain rather than being renewed for each backup in it. - ### Skip the storage access check When a cloud backup backend (`backup-s3`, `backup-gcs`, or `backup-azure`) initializes, Weaviate verifies that the configured credentials can write to and delete from the target bucket. It does this by writing a temporary `access-check` object and then removing it. This probe fails on immutable (write-once / WORM) buckets, or with least-privilege credentials that are not permitted to delete objects. From dc0e3204df4965373ff0e31da663adc07246771d Mon Sep 17 00:00:00 2001 From: Ivan Despot <66276597+g-despot@users.noreply.github.com> Date: Thu, 30 Jul 2026 16:08:19 +0200 Subject: [PATCH 5/5] docs(backups): tighten the split-part size bounds The lower bound on a split part was stated with a strict inequality. Because the part count is a ceiling division and the final part carries only the remainder, the last part can land on exactly half of the split size: a file of split size plus one byte splits into two parts of 26843545601 and 26843545600 bytes, the second being exactly 25GiB at the default 50GiB split. Say "at least half of the split size" instead of "larger than half", and soften "equal parts" to "roughly equal", since parts can differ by up to one byte per part. The upper bound and the paragraph's conclusion are unchanged. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01VLMB1FhdGFGcdg63uVHxNY --- docs/deploy/configuration/backups.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deploy/configuration/backups.md b/docs/deploy/configuration/backups.md index c85ec2b5..bbb001fc 100644 --- a/docs/deploy/configuration/backups.md +++ b/docs/deploy/configuration/backups.md @@ -792,7 +792,7 @@ Because the larger of the two wins, `BACKUP_MIN_CHUNK_SIZE` is a floor on the qu On an incremental backup, the `BACKUP_MAX_INDIVIDUAL_FILES` budget is reduced by the number of files that are reused from the base backup, so it is shared across a whole backup chain rather than being renewed for each backup in it. -A chunk that carries part of a split file holds nothing else, because Weaviate deliberately leaves the rest of that chunk empty rather than mixing split parts with regular files. The size of such a chunk is governed by `BACKUP_SPLIT_FILE_SIZE` rather than by `BACKUP_CHUNK_TARGET_SIZE`: Weaviate divides the file into equal parts that are each larger than half of the split size and no larger than the split size, so at the defaults these chunks are far larger than the target, not smaller. +A chunk that carries part of a split file holds nothing else, because Weaviate deliberately leaves the rest of that chunk empty rather than mixing split parts with regular files. The size of such a chunk is governed by `BACKUP_SPLIT_FILE_SIZE` rather than by `BACKUP_CHUNK_TARGET_SIZE`: Weaviate divides the file into roughly equal parts that are each at least half of the split size and no larger than the split size, so at the defaults these chunks are far larger than the target, not smaller. The three size variables below accept a plain number of bytes or a number with a unit suffix (`B`, `KB`, `MB`, `GB`, `TB`, `KiB`, `MiB`, `GiB`, `TiB`), for example `4MiB`. The decimal and binary suffixes are distinct: `MB` is 1,000,000 bytes while `MiB` is 1,048,576 bytes. They also accept `unlimited` or `nolimit`, which is how you disable file splitting through `BACKUP_SPLIT_FILE_SIZE`. All three are read at startup, so a restart is required to change them. `BACKUP_MAX_INDIVIDUAL_FILES` is a plain count instead of a size, and Weaviate reads it for each backup, so you can change it without a restart through the `backup_max_individual_files` [runtime configuration](./env-vars/runtime-config.md) key.