Skip to content

feat(cache): add file-tree and download-link caches (DB by default) - #75

Open
Wudarensheng wants to merge 3 commits into
OpenListTeam:mainfrom
Wudarensheng:feat/cache
Open

Wudarensheng wants to merge 3 commits into
OpenListTeam:mainfrom
Wudarensheng:feat/cache

Conversation

@Wudarensheng

Copy link
Copy Markdown
Member

feat(cache): add file-tree and download-link caches (DB by default)

Summary / 摘要

Browsing a directory and starting a download both hit the remote drive on every
single request. The download-link exchange (driver.get() → presigned / raw URL)
is usually the most expensive and most rate-limited step of the two. The upstream
OpenList.ts reference solves this with a two-level cache (file tree + download
links); this PR ports that design to this repository.

浏览目录和开始下载这两个动作,此前每次请求都会打到远端存储;其中「换链」
(driver.get() → 预签名 / 直链)通常是最贵、最容易被网盘限流的一步。
本 PR 参考上游 OpenList.ts 的实现,把它的两级缓存(文件树 + 下载链接)移植过来。

User-visible behavior / 用户可感知的变化

  • Directory listings are served from cache (default TTL 30 min, following the
    per-storage cache_expiration and custom_cache_policies). Empty directories
    are cached too, so repeated browsing stops hitting the remote drive.
    / 目录列表改为走缓存(默认 30 分钟,跟随存储级 cache_expiration 与
    custom_cache_policies);空目录同样会被缓存。
  • Download links returned by drivers are reused for a short window (default
    5 min), so repeated downloads / previews skip the link exchange.
    / 驱动换来的直链会在短时间内复用(默认 5 分钟),重复下载 / 预览不再重复换链。
  • Writes (mkdir / rename / remove / move / copy / put) invalidate the
    affected path plus its parent directory for the file tree, and the path
    itself for links. / 写操作后自动失效:文件树连带父目录,链接只失效自身。
  • Storage create / update / enable / disable / delete clears that storage's cache
    entirely. / 存储的新增 / 修改 / 启停 / 删除会清空该存储的全部缓存。
  • 4 new admin endpoints: GET /cache/status, POST /cache/clear,
    POST /storage/refresh, POST /storage/refresh_one.
    / 新增 4 个管理接口(含对齐参考实现的 refresh / refresh_one)。
  • GET /env_check now also reports the resolved cache config and the actual
    backends in use. / /env_check 额外回显生效的缓存配置与实际后端。

Implementation / 重要实现变化

  • New module src/backend/internal/cache/ — config.ts (env switches),
    store.ts (backend resolution + key encoding + envelope get/set/list/clear),
    filetree.ts, link.ts, index.ts (barrel + invalidation orchestration),
    plus cache.test.ts (22 cases).
  • Backend selection is entirely env-driven: CACHE_DRIVER=db (the default)
    resolves to the same backend as DB_DRIVER via getStorageBackend(), so
    zero configuration means "cache in the database".
  • Dedicated kv / blob / cfkv / do / memory backends must be opted into
    explicitly and are never auto-detected or auto-substituted — consistent with
    the existing rule that an explicitly configured driver never falls back. An
    unavailable backend is skipped with a one-time warning; if none are available
    the cache silently degrades to a no-op.
  • Cache entries are written through the driver's raw put / get / delete /
    list (deliberately not via saveDb(), which would trigger whole-config
    serialization, write guards and field encryption), under an isolated key prefix
    <CACHE_PREFIX>_<kind>_<storageId>_<encodedPath> (kind = ft / ln), so
    business-data keys are never touched. Paths longer than the EdgeOne KV key limit
    are folded with a double FNV-1a hash.
  • Safety boundary: only driver-layer results are cached (raw FileItem[] and
    raw links). Permissions, meta passwords, hide rules and signatures are still
    computed per request in server/fs.ts / server/raw.ts, so a cache hit cannot
    leak privileges across users. Any cache error degrades to a no-op and never
    fails the request.

Config / 配置 (all optional — defaults preserve "DB-only"):

Variable Default Meaning
CACHE_ENABLED true Master switch for both caches.
CACHE_DRIVER db Backend list, comma-separated: db / kv / blob / cfkv / do / memory / none.
CACHE_FILE_TREE true Toggle the file-tree cache.
CACHE_DOWNLOAD_LINK true Toggle the download-link cache.
CACHE_TTL 0 File-tree TTL in minutes; 0 = follow per-storage cache_expiration (30).
CACHE_LINK_TTL 5 Download-link TTL in minutes.
CACHE_EXCLUDE_DRIVERS virtual,alias,url_tree,strm,chunk Drivers excluded from caching.
CACHE_PREFIX openlist_cache Key prefix isolating cache entries from business data.

Compatibility / 兼容性

  • Purely additive: no existing endpoint, config key, storage format or
    migration path is modified.

  • Caching is enabled by default. Set CACHE_ENABLED=false (or
    CACHE_DRIVER=none) to restore the previous behavior exactly; a single storage
    can opt out with cache_expiration=0 or a custom_cache_policies rule.
    / 缓存默认开启;CACHE_ENABLED=false(或 CACHE_DRIVER=none)可完全恢复原行为,
    单个存储也可用 cache_expiration=0 退出。

  • This PR has breaking changes.
    / 此 PR 包含破坏性变更。

  • This PR changes public API, config, storage format, or migration behavior.
    / 此 PR 修改了公开 API、配置、存储格式或迁移行为。

  • This PR requires corresponding changes in related repositories.
    / 此 PR 需要关联仓库同步修改。

Related repository PRs / 关联仓库 PR:

  • OpenList: — (no change required; the 4 new admin endpoints have no frontend UI yet, follow-up optional)
  • OpenList-Docs: to be opened — documents the 8 new CACHE_* variables and the cache admin endpoints

Testing / 测试

Platform: Windows 10/11, Node.js 22.22.2, tsx --test.

  • go test ./... — N/A: this repository is the TypeScript / Cloudflare Workers
    port; no Go sources are changed.
  • Type check: npx tsc -p tsconfig.json --noEmit — 0 errors in every file
    changed by this PR. (Repo-wide, 10 errors remain, all pre-existing and
    unrelated: pkg/validators.ts cannot resolve zod in this sandbox, and
    internal/model/db_cipher.test.ts has two pre-existing typing errors.)
  • npm run test:cache — 22/22 pass
  • npm run test:model — 39/39 pass
  • npm run test:store — 12/12 pass
  • npm run test:drivers — 111/111 pass
  • npm run test:189 — 20/20 pass
  • npm run test:server — 125/129 pass

The 4 test:server failures are pre-existing and unrelated to this PR. This was
verified by a control run: with this PR's four touched source files reverted to
HEAD, the suite produces the identical result (129 tests / 125 pass / 4 fail, the
same four cases):

  • default_credentials.test.ts (3) — admin bootstrap / password-reset semantics.
    Its import closure (db, auth, user, password, middlewares, op/sshkey)
    does not contain any file this PR touches.

  • seed.test.ts (1) — "CAS codec matches casmeta base64 JSON field names": the
    codec emits cloud / slice_md5s / slice_size beyond the field set the test
    asserts.

  • Manual test / 手动测试: not performed — and here is why. This environment has
    no credentials for a live remote storage (网盘 / object storage), so the cache
    could not be exercised end-to-end against a real drive. Coverage is unit-level,
    against the in-memory driver. Suggested manual check before merge:
    1. Attach a real storage, list a directory twice — the second listing should be
    served from cache (no remote request).
    2. Download a file twice — the second download should reuse the cached link.
    3. Rename / delete a file — both caches for that path and its parent should be
    invalidated, and the next listing should reflect the change.
    4. GET /api/admin/cache/status should report non-zero file_tree /
    download_link entry counts.

Checklist / 检查清单

  • I have read CONTRIBUTING.
    / 我已阅读 CONTRIBUTING.
  • I confirm this contribution follows the repository license, contribution policy, and code of conduct.
    / 我确认此贡献符合仓库许可证、贡献规范和行为准则。
  • I have formatted the changed code with gofmt, go fmt, or prettier where applicable.
    / 我已按适用情况使用 gofmt、go fmt 或 prettier 格式化变更代码。
  • I have requested review from relevant maintainers or code owners where applicable.
    / 我已在适用情况下请求相关维护者或代码所有者审查。

AI Disclosure / AI 使用声明

  • This PR includes AI-assisted content.
    / 此 PR 包含 AI 辅助内容。

Tools used / 使用工具:

  • ChatGPT
  • Codex
  • GitHub Copilot
  • Claude
  • Gemini
  • Other (please specify) / 其他(请注明): WorkBuddy AI (DeepSeek-V4.1-Flash)

Usage scope / 使用范围:

  • Code generation / 代码生成

  • Refactoring / 重构

  • Documentation / 文档

  • Tests / 测试

  • Translation / 翻译

  • Review assistance / 审查辅助

  • I have reviewed and validated all AI-assisted content included in this PR.
    / 我已审核并验证此 PR 中的所有 AI 辅助内容。

  • I have ensured that all AI-assisted commits include Co-Authored-By attribution.
    / 我已确保所有 AI 辅助提交都包含 Co-Authored-By 归属信息。

  • I can reproduce all AI-assisted content included in this PR without any AI tools.
    / 我可以在没有任何 AI 工具的情况下重现此 PR 中包含的所有 AI 辅助内容。

Wudarensheng and others added 3 commits September 24, 2026 12:40
Implement the two-level cache from the OpenList.ts reference:

- File-tree cache (`ft`): directory listings are served from cache, keyed
  by storage + virtual path. Empty directories are cached too, so repeated
  browsing no longer hits the remote drive every time.
- Download-link cache (`ln`): the raw_url returned by the driver is reused
  instead of being re-signed/re-exchanged on every download, which is
  usually the most expensive and most rate-limited step.

Caching is DB-only by default. `CACHE_DRIVER=db` (the default) reuses the
backend resolved by `getStorageBackend()`, i.e. the same one as
`DB_DRIVER`, so zero configuration means "cache in the database".
Dedicated KV / Blob / cfkv / do / memory backends must be opted into
explicitly via `CACHE_DRIVER` (e.g. `db,kv`, `kv`, `blob`) and are never
auto-detected or auto-substituted, matching the existing DB_DRIVER rule
that an explicitly configured driver never falls back.

Safety boundary: only driver-layer results are cached (raw FileItem lists
and raw links). Permissions, meta passwords, hide rules and signatures are
still computed per request in server/fs.ts and server/raw.ts, so a cache
hit cannot leak privileges. Any cache error degrades to a no-op and never
breaks the request.

Invalidation: writes (mkdir/rename/remove/move/copy/put) invalidate the
affected path plus its parent directory for the file tree, and the path
itself for links. Storage create/update/enable/disable/delete clears that
storage's cache entirely.

New environment variables (all optional): CACHE_ENABLED, CACHE_DRIVER,
CACHE_FILE_TREE, CACHE_DOWNLOAD_LINK, CACHE_TTL, CACHE_LINK_TTL,
CACHE_EXCLUDE_DRIVERS, CACHE_PREFIX.

New admin endpoints: GET /cache/status, POST /cache/clear,
POST /storage/refresh, POST /storage/refresh_one.
The local assistant keeps its project memory under .workbuddy-ai/, which
must never be committed. Ignore it next to the existing .codebuddy entry.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant