Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions docs/deploy/configuration/env-vars/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,11 +83,11 @@ import APITable from '@site/src/components/APITable';
| `OPERATIONAL_MODE` | Sets the [mode of operation](../status.md#operational-modes) for the instance. Options: `READ_WRITE` (default), `READ_ONLY`, `WRITE_ONLY`, `SCALE_OUT`. Limits available operations based on the mode selected. | `string` | `READ_WRITE` |
| `ORIGIN` | Set the http(s) origin for Weaviate | `string - HTTP origin` | `https://my-weaviate-deployment.com` |
| `PERSISTENCE_DATA_PATH` | Path to the Weaviate data store.<br/>[Note about file systems and performance](/weaviate/concepts/resources.md#file-system). | `string - file path` | `/var/lib/weaviate` <br/> Defaults to `./data`|
| `PERSISTENCE_HNSW_DISABLE_SNAPSHOTS` | **Deprecated in `v1.39`.** Ignored, and logs a startup warning. [In `v1.31` through `v1.38`](/weaviate/configuration/hnsw-snapshots.md#pre-v1-39-configuration): if set, HNSW snapshotting will be disabled. Default: `false` (enabled) as of `v1.36`; `true` (disabled) in `v1.31` through `v1.35`<br/>Added in `v1.31` | `boolean` | `false` |
| `PERSISTENCE_HNSW_SNAPSHOT_INTERVAL_SECONDS` | **Deprecated in `v1.39`.** Ignored, and logs a startup warning. [In `v1.31` through `v1.38`](/weaviate/configuration/hnsw-snapshots.md#pre-v1-39-configuration): the minimum time in seconds that must pass before the next snapshot is created. Default: `21600` seconds (6 hours)<br/>Added in `v1.31` | `string - number` | `3600` |
| `PERSISTENCE_HNSW_SNAPSHOT_MIN_DELTA_COMMITLOGS_NUMBER` | **Deprecated in `v1.39`.** Ignored, and logs a startup warning. [In `v1.31` through `v1.38`](/weaviate/configuration/hnsw-snapshots.md#pre-v1-39-configuration): the minimum number of new commit log files created since the last snapshot. Default: `1`<br/>Added in `v1.31` | `string - number` | `100` |
| `PERSISTENCE_HNSW_SNAPSHOT_MIN_DELTA_COMMITLOGS_SIZE_PERCENTAGE` | **Deprecated in `v1.39`.** Ignored, and logs a startup warning. [In `v1.31` through `v1.38`](/weaviate/configuration/hnsw-snapshots.md#pre-v1-39-configuration): the minimum total size of new commit logs (as a percentage of the previous snapshot's size) required to trigger a new snapshot. Default: `5` (meaning 5% of the previous snapshot's size in new commit logs)<br/>Added in `v1.31` | `string - number` | `15` |
| `PERSISTENCE_HNSW_SNAPSHOT_ON_STARTUP` | **Deprecated in `v1.39`.** Ignored, and logs a startup warning. [In `v1.31` through `v1.38`](/weaviate/configuration/hnsw-snapshots.md#pre-v1-39-configuration): if set, Weaviate will try to create a new snapshot during startup if there are changes in the commit log since the last snapshot. If there are no changes, then the existing snapshot will be loaded. Default: `true`<br/>Added in `v1.31` | `boolean` | `false` |
| `PERSISTENCE_HNSW_DISABLE_SNAPSHOTS` | **Deprecated in `v1.39`.** Ignored, and logs a startup warning. In `v1.31` through `v1.38`: if set, HNSW snapshotting is disabled. Default: `false` (enabled) as of `v1.36`; `true` (disabled) in `v1.31` through `v1.35`. See [snapshot configuration before `v1.39`](/weaviate/concepts/storage.md#pre-v1-39-configuration). <br/>Added in `v1.31` | `boolean` | `false` |
| `PERSISTENCE_HNSW_SNAPSHOT_INTERVAL_SECONDS` | **Deprecated in `v1.39`.** Ignored, and logs a startup warning. In `v1.31` through `v1.38`: the minimum time in seconds that must pass before the next snapshot is created. Default: `21600` seconds (6 hours). See [snapshot configuration before `v1.39`](/weaviate/concepts/storage.md#pre-v1-39-configuration). <br/>Added in `v1.31` | `string - number` | `3600` |
| `PERSISTENCE_HNSW_SNAPSHOT_MIN_DELTA_COMMITLOGS_NUMBER` | **Deprecated in `v1.39`.** Ignored, and logs a startup warning. In `v1.31` through `v1.38`: the minimum number of new commit log files created since the last snapshot. Default: `1`. See [snapshot configuration before `v1.39`](/weaviate/concepts/storage.md#pre-v1-39-configuration). <br/>Added in `v1.31` | `string - number` | `100` |
| `PERSISTENCE_HNSW_SNAPSHOT_MIN_DELTA_COMMITLOGS_SIZE_PERCENTAGE` | **Deprecated in `v1.39`.** Ignored, and logs a startup warning. In `v1.31` through `v1.38`: the minimum total size of new commit logs (as a percentage of the previous snapshot's size) required to trigger a new snapshot. Default: `5` (meaning 5% of the previous snapshot's size in new commit logs). See [snapshot configuration before `v1.39`](/weaviate/concepts/storage.md#pre-v1-39-configuration). <br/>Added in `v1.31` | `string - number` | `15` |
| `PERSISTENCE_HNSW_SNAPSHOT_ON_STARTUP` | **Deprecated in `v1.39`.** Ignored, and logs a startup warning. In `v1.31` through `v1.38`: if set, Weaviate tries to create a new snapshot during startup if there are changes in the commit log since the last snapshot. If there are no changes, the existing snapshot is loaded. Default: `true`. See [snapshot configuration before `v1.39`](/weaviate/concepts/storage.md#pre-v1-39-configuration). <br/>Added in `v1.31` | `boolean` | `false` |
Comment on lines +86 to +90
| `PERSISTENCE_HNSW_MAX_LOG_SIZE` | Maximum size of the HNSW [write-ahead-log](/weaviate/concepts/storage.md#hnsw-vector-index-storage). Increase this to improve log compaction efficiency, or decrease to reduce memory requirements. Default: 500MiB | `string` | `4GiB` (IEC units), `4GB` (SI units), `4000000000` (bytes) |
| `PERSISTENCE_LSM_ACCESS_STRATEGY` | Function used to access disk data in virtual memory. Default: `mmap` | `string` | `mmap` or `pread` |
| `PERSISTENCE_LSM_MAX_SEGMENT_SIZE` | Maximum size of a segment in the [LSM store](/weaviate/concepts/storage.md#object-and-inverted-index-store). Set this to limit disk usage spikes during compaction to ~2x the segment size. Default: no limit | `string` | `4GiB` (IEC units), `4GB` (SI units), `4000000000` (bytes) |
Expand Down
1 change: 1 addition & 0 deletions docs/deploy/configuration/persistence.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,7 @@ You can configure automatic deletion of objects after a specified time period us

## Related pages
- [Configuration: Backups](/deploy/configuration/backups.md)
- [Concepts: Storage - HNSW snapshots](/weaviate/concepts/storage.md#hnsw-snapshots)

## Questions and feedback

Expand Down
41 changes: 29 additions & 12 deletions docs/weaviate/concepts/storage.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: Storage
sidebar_position: 18
description: "Persistent, fault-tolerant storage architecture for objects, vectors, and inverted index management."
description: "Persistent, fault-tolerant storage architecture for objects, vectors, and inverted index management, including HNSW snapshots and commit log compaction."
image: og/docs/concepts.jpg
# tags: ['architecture', 'storage']
---
Expand Down Expand Up @@ -79,7 +79,7 @@ This change improves reliability during rolling restarts and upgrades. Eager loa

#### Vector cache prefill behavior

The [`HNSW_STARTUP_WAIT_FOR_VECTOR_CACHE`](/deploy/configuration/env-vars#hnsw_startup_wait_for_vector_cache) environment variable controls whether vector cache prefill is synchronous (blocking) or asynchronous (background) at startup. Its default changed to `true` in v1.36.6.
The [`HNSW_STARTUP_WAIT_FOR_VECTOR_CACHE`](/deploy/configuration/env-vars#HNSW_STARTUP_WAIT_FOR_VECTOR_CACHE) environment variable controls whether vector cache prefill is synchronous (blocking) or asynchronous (background) at startup. Its default changed to `true` in v1.36.6.

For collections where lazy shard loading is active, vector cache prefill is always **asynchronous** — the `HNSW_STARTUP_WAIT_FOR_VECTOR_CACHE` value is overridden to `false` regardless of the configured value. For eagerly-loaded collections, the configured value applies (default: `true`, meaning synchronous prefill).

Expand All @@ -101,35 +101,52 @@ For the HNSW vector index, the Write-Ahead-Log (WAL) is a critical component for

The entire HNSW index state can be reconstructed by replaying these WAL entries.

For very large indexes of tens or hundreds of millions of objects, this can be time-consuming. To avoid replaying the entire WAL on every restart, Weaviate writes **[HNSW snapshots](../configuration/hnsw-snapshots.md)**.
For very large indexes of tens or hundreds of millions of objects, this can be time-consuming. To avoid replaying the entire commit log on every restart, Weaviate writes **[HNSW snapshots](#hnsw-snapshots)**.

### HNSW snapshots

import HnswSnapshots from '/_includes/feature-notes/hnsw-snapshots.mdx';

<HnswSnapshots/>

For very large HNSW vector indexes, HNSW snapshots significantly reduce the startup time.

A snapshot represents a point-in-time state of the HNSW index. When Weaviate starts, it loads the most recent snapshot and replays only the commit log entries written after it. This significantly reduces startup time, because the number of entries that have to be replayed no longer grows with the age of the index.

The commit log still persists every change immediately, guaranteeing that any acknowledged write is durable. Even with a fresh snapshot, the server typically still has to load at least one subsequent commit log file.

Starting in `v1.39`, snapshots are part of how the vector index is stored rather than an optional speedup, and Weaviate manages them automatically. A background compactor owns the on-disk lifecycle of the index: it compacts newly flushed commit logs, merges them together, and writes a new snapshot when doing so is worthwhile. Snapshots and commit logs live in the same directory, and a snapshot replaces the commit logs it covers rather than duplicating them, which keeps the disk footprint proportional to the size of the index.
Starting in `v1.39`, snapshots are part of how the vector index is stored rather than an optional speedup. A background process called the commit log compactor owns the on-disk lifecycle of the index: it compacts newly flushed commit logs, merges them together, and writes a new snapshot when doing so is worthwhile. Snapshots and commit logs live in the same directory, and a snapshot replaces the commit logs it covers rather than duplicating them, which keeps the disk footprint proportional to the size of the index.

Upgrading to `v1.39` therefore reduces the disk space the vector index uses, in some cases substantially. Earlier versions keep the full commit log alongside the snapshot, and a snapshot is a more compact representation of the same index than the commit logs it replaces, because compaction keeps only the final state of each vector's connections instead of every change made to them.

Two caveats apply. The saving appears once the compactor has run its first cycles on each loaded shard rather than at the moment you upgrade, and inactive tenants do not shrink until they are next activated. Keep planning for the same headroom as before, because peak disk usage still rises while a snapshot is being written.

Because the merge reads its inputs as sorted streams from disk, only the most recently flushed commit log is processed in memory. That step has a fixed cost regardless of how large the graph is, so snapshot creation no longer requires enough memory to hold the previous snapshot plus the commit log delta.

Weaviate protects this on-disk state in several ways. New files are written to a temporary path and atomically renamed into place, so an interrupted write can never be mistaken for a complete file, and orphaned temporary files are cleaned up on the next startup.

Commit logs, including compacted ones, are self-healing. If one cannot be read in full, whether because a crash tore its tail or because the file was damaged on disk, it is truncated back to its last valid entry. The entries written before the damage are retained and the file becomes valid again for later compaction, so only the damaged tail is lost.
Commit logs, including compacted ones, are self-healing. If one cannot be read in full, whether because a crash cut it short or because the file was damaged on disk, it is truncated back to its last valid entry. The entries written before the damage are retained and the file becomes valid again for later compaction, so only the damaged tail is lost.

Snapshots are handled differently. A snapshot is stored in a checksummed block format, and it is never truncated or repaired. If the current snapshot cannot be read, the affected vector index fails to start rather than loading partial data. Because the commit logs a snapshot covers are deleted once it has been written, nothing remains on the node to replay in its place, so recovering that shard means restoring the data, for example from a [backup](/deploy/configuration/backups.md).
Snapshots are handled differently. A snapshot is stored in a checksummed block format and every block is verified when it is read, but unlike a commit log, a snapshot is not truncated or repaired.

See **[the HNSW snapshots configuration](../configuration/hnsw-snapshots.md)** for version-specific details.
If the current snapshot cannot be read, Weaviate does not load a partial index. The shard that owns the snapshot fails to load, and so does every other vector index on that shard. Because the commit logs a snapshot covers are deleted once the snapshot has been written, the index cannot be rebuilt from the node's remaining files. If that shard uses [dynamic lazy shard loading](#dynamic-lazy-shard-loading), the node stays up and requests to the shard return an error. If the shard is loaded eagerly, which is the default for single-tenant collections and for multi-tenant collections below the auto-detection thresholds, node startup fails instead. Restore the affected data from a [backup](/deploy/configuration/backups.md), which includes the snapshot.

:::note Behavior in `v1.31` through `v1.38`
In these versions, snapshots are an optional feature layered on top of the commit log, and are configured with the `PERSISTENCE_HNSW_SNAPSHOT_*` environment variables. They are enabled by default starting in `v1.36`, and disabled by default in `v1.31` through `v1.35`. Weaviate creates a snapshot at startup if the commit log changed since the last snapshot, and periodically once a configured time interval has passed and enough new commit log data has accumulated. If a snapshot cannot be loaded, it is removed and Weaviate falls back to loading the full commit log from the beginning.
:::
The environment variables that configured snapshots before `v1.39` are deprecated. That version and later still recognize `PERSISTENCE_HNSW_DISABLE_SNAPSHOTS` and the `PERSISTENCE_HNSW_SNAPSHOT_*` variables, so an existing deployment starts without a configuration error, but their values are ignored. For each of these variables that is set, Weaviate logs a warning at startup stating that the variable has no effect and will be removed in a future version; variables that are not set produce no warning. If your deployment passes a configuration file with `--config-file` (`weaviate.conf.json` by default), the equivalent camelCase fields in that file, such as `hnswDisableSnapshots`, are accepted and ignored as well, and unlike the environment variables they produce no startup warning. Remove the variables from your deployment configuration, and the equivalent fields from any configuration file, to clear the warnings.

#### Snapshot configuration before `v1.39` {#pre-v1-39-configuration}

In `v1.31` through `v1.38`, snapshots are an optional feature layered on top of the commit log rather than part of it, and the `PERSISTENCE_HNSW_SNAPSHOT_*` environment variables control when Weaviate creates them. Snapshots are enabled by default starting in `v1.36`, and disabled by default in `v1.31` through `v1.35`. Weaviate creates one at startup if the commit log changed since the last snapshot, and periodically thereafter. If a snapshot cannot be read in these versions, it is discarded and Weaviate replays the full commit log instead. For the variables themselves, including their defaults and deprecation status, see [`PERSISTENCE_HNSW_DISABLE_SNAPSHOTS`](/deploy/configuration/env-vars/index.md#PERSISTENCE_HNSW_DISABLE_SNAPSHOTS) and the rows that follow it.

<details>
<summary>Periodic snapshot conditions and memory requirements</summary>

Periodic snapshot creation is governed by three variables, and **all** of the following conditions must be met before Weaviate creates a snapshot:

- `PERSISTENCE_HNSW_SNAPSHOT_INTERVAL_SECONDS` — the minimum time since the previous snapshot has elapsed.
- `PERSISTENCE_HNSW_SNAPSHOT_MIN_DELTA_COMMITLOGS_NUMBER` — enough new commit log files have been created since the last snapshot.
- `PERSISTENCE_HNSW_SNAPSHOT_MIN_DELTA_COMMITLOGS_SIZE_PERCENTAGE` — the new commit logs are large enough, measured as a percentage of the previous snapshot's size.

Before creating a new snapshot, Weaviate loads the previous snapshot and the commit log difference into memory, so the node needs enough memory to accommodate both. This requirement does not apply in `v1.39` and later, where the merge streams its inputs from disk.

</details>

## Conclusions

Expand Down
Loading
Loading