feat(cache): add file-tree and download-link caches (DB by default) - #75
Open
Wudarensheng wants to merge 3 commits into
Open
Wudarensheng wants to merge 3 commits into
Wudarensheng wants to merge 3 commits into
Conversation
Implement the two-level cache from the OpenList.ts reference: - File-tree cache (`ft`): directory listings are served from cache, keyed by storage + virtual path. Empty directories are cached too, so repeated browsing no longer hits the remote drive every time. - Download-link cache (`ln`): the raw_url returned by the driver is reused instead of being re-signed/re-exchanged on every download, which is usually the most expensive and most rate-limited step. Caching is DB-only by default. `CACHE_DRIVER=db` (the default) reuses the backend resolved by `getStorageBackend()`, i.e. the same one as `DB_DRIVER`, so zero configuration means "cache in the database". Dedicated KV / Blob / cfkv / do / memory backends must be opted into explicitly via `CACHE_DRIVER` (e.g. `db,kv`, `kv`, `blob`) and are never auto-detected or auto-substituted, matching the existing DB_DRIVER rule that an explicitly configured driver never falls back. Safety boundary: only driver-layer results are cached (raw FileItem lists and raw links). Permissions, meta passwords, hide rules and signatures are still computed per request in server/fs.ts and server/raw.ts, so a cache hit cannot leak privileges. Any cache error degrades to a no-op and never breaks the request. Invalidation: writes (mkdir/rename/remove/move/copy/put) invalidate the affected path plus its parent directory for the file tree, and the path itself for links. Storage create/update/enable/disable/delete clears that storage's cache entirely. New environment variables (all optional): CACHE_ENABLED, CACHE_DRIVER, CACHE_FILE_TREE, CACHE_DOWNLOAD_LINK, CACHE_TTL, CACHE_LINK_TTL, CACHE_EXCLUDE_DRIVERS, CACHE_PREFIX. New admin endpoints: GET /cache/status, POST /cache/clear, POST /storage/refresh, POST /storage/refresh_one.
The local assistant keeps its project memory under .workbuddy-ai/, which must never be committed. Ignore it next to the existing .codebuddy entry.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(cache): add file-tree and download-link caches (DB by default)
Summary / 摘要
Browsing a directory and starting a download both hit the remote drive on every
single request. The download-link exchange (
driver.get()→ presigned / raw URL)is usually the most expensive and most rate-limited step of the two. The upstream
OpenList.tsreference solves this with a two-level cache (file tree + downloadlinks); this PR ports that design to this repository.
浏览目录和开始下载这两个动作,此前每次请求都会打到远端存储;其中「换链」
(
driver.get()→ 预签名 / 直链)通常是最贵、最容易被网盘限流的一步。本 PR 参考上游
OpenList.ts的实现,把它的两级缓存(文件树 + 下载链接)移植过来。User-visible behavior / 用户可感知的变化
per-storage
cache_expirationandcustom_cache_policies). Empty directoriesare cached too, so repeated browsing stops hitting the remote drive.
/ 目录列表改为走缓存(默认 30 分钟,跟随存储级
cache_expiration与custom_cache_policies);空目录同样会被缓存。5 min), so repeated downloads / previews skip the link exchange.
/ 驱动换来的直链会在短时间内复用(默认 5 分钟),重复下载 / 预览不再重复换链。
mkdir/rename/remove/move/copy/put) invalidate theaffected path plus its parent directory for the file tree, and the path
itself for links. / 写操作后自动失效:文件树连带父目录,链接只失效自身。
entirely. / 存储的新增 / 修改 / 启停 / 删除会清空该存储的全部缓存。
GET /cache/status,POST /cache/clear,POST /storage/refresh,POST /storage/refresh_one./ 新增 4 个管理接口(含对齐参考实现的 refresh / refresh_one)。
GET /env_checknow also reports the resolved cache config and the actualbackends in use. /
/env_check额外回显生效的缓存配置与实际后端。Implementation / 重要实现变化
src/backend/internal/cache/—config.ts(env switches),store.ts(backend resolution + key encoding + envelope get/set/list/clear),filetree.ts,link.ts,index.ts(barrel + invalidation orchestration),plus
cache.test.ts(22 cases).CACHE_DRIVER=db(the default)resolves to the same backend as
DB_DRIVERviagetStorageBackend(), sozero configuration means "cache in the database".
kv/blob/cfkv/do/memorybackends must be opted intoexplicitly and are never auto-detected or auto-substituted — consistent with
the existing rule that an explicitly configured driver never falls back. An
unavailable backend is skipped with a one-time warning; if none are available
the cache silently degrades to a no-op.
put/get/delete/list(deliberately not viasaveDb(), which would trigger whole-configserialization, write guards and field encryption), under an isolated key prefix
<CACHE_PREFIX>_<kind>_<storageId>_<encodedPath>(kind=ft/ln), sobusiness-data keys are never touched. Paths longer than the EdgeOne KV key limit
are folded with a double FNV-1a hash.
FileItem[]andraw links). Permissions, meta passwords, hide rules and signatures are still
computed per request in
server/fs.ts/server/raw.ts, so a cache hit cannotleak privileges across users. Any cache error degrades to a no-op and never
fails the request.
Config / 配置 (all optional — defaults preserve "DB-only"):
CACHE_ENABLEDtrueCACHE_DRIVERdbdb/kv/blob/cfkv/do/memory/none.CACHE_FILE_TREEtrueCACHE_DOWNLOAD_LINKtrueCACHE_TTL00= follow per-storagecache_expiration(30).CACHE_LINK_TTL5CACHE_EXCLUDE_DRIVERSvirtual,alias,url_tree,strm,chunkCACHE_PREFIXopenlist_cacheCompatibility / 兼容性
Purely additive: no existing endpoint, config key, storage format or
migration path is modified.
Caching is enabled by default. Set
CACHE_ENABLED=false(orCACHE_DRIVER=none) to restore the previous behavior exactly; a single storagecan opt out with
cache_expiration=0or acustom_cache_policiesrule./ 缓存默认开启;
CACHE_ENABLED=false(或CACHE_DRIVER=none)可完全恢复原行为,单个存储也可用
cache_expiration=0退出。This PR has breaking changes.
/ 此 PR 包含破坏性变更。
This PR changes public API, config, storage format, or migration behavior.
/ 此 PR 修改了公开 API、配置、存储格式或迁移行为。
This PR requires corresponding changes in related repositories.
/ 此 PR 需要关联仓库同步修改。
Related repository PRs / 关联仓库 PR:
CACHE_*variables and the cache admin endpointsTesting / 测试
Platform: Windows 10/11, Node.js 22.22.2,
tsx --test.go test ./...— N/A: this repository is the TypeScript / Cloudflare Workersport; no Go sources are changed.
npx tsc -p tsconfig.json --noEmit— 0 errors in every filechanged by this PR. (Repo-wide, 10 errors remain, all pre-existing and
unrelated:
pkg/validators.tscannot resolvezodin this sandbox, andinternal/model/db_cipher.test.tshas two pre-existing typing errors.)npm run test:cache— 22/22 passnpm run test:model— 39/39 passnpm run test:store— 12/12 passnpm run test:drivers— 111/111 passnpm run test:189— 20/20 passnpm run test:server— 125/129 passThe 4
test:serverfailures are pre-existing and unrelated to this PR. This wasverified by a control run: with this PR's four touched source files reverted to
HEAD, the suite produces the identical result (129 tests / 125 pass / 4 fail, thesame four cases):
default_credentials.test.ts(3) — admin bootstrap / password-reset semantics.Its import closure (
db,auth,user,password,middlewares,op/sshkey)does not contain any file this PR touches.
seed.test.ts(1) — "CAS codec matches casmeta base64 JSON field names": thecodec emits
cloud/slice_md5s/slice_sizebeyond the field set the testasserts.
Manual test / 手动测试: not performed — and here is why. This environment has
no credentials for a live remote storage (网盘 / object storage), so the cache
could not be exercised end-to-end against a real drive. Coverage is unit-level,
against the in-memory driver. Suggested manual check before merge:
1. Attach a real storage, list a directory twice — the second listing should be
served from cache (no remote request).
2. Download a file twice — the second download should reuse the cached link.
3. Rename / delete a file — both caches for that path and its parent should be
invalidated, and the next listing should reflect the change.
4.
GET /api/admin/cache/statusshould report non-zerofile_tree/download_linkentry counts.Checklist / 检查清单
/ 我已阅读 CONTRIBUTING.
/ 我确认此贡献符合仓库许可证、贡献规范和行为准则。
gofmt,go fmt, orprettierwhere applicable./ 我已按适用情况使用
gofmt、go fmt或prettier格式化变更代码。/ 我已在适用情况下请求相关维护者或代码所有者审查。
AI Disclosure / AI 使用声明
/ 此 PR 包含 AI 辅助内容。
Tools used / 使用工具:
Usage scope / 使用范围:
Code generation / 代码生成
Refactoring / 重构
Documentation / 文档
Tests / 测试
Translation / 翻译
Review assistance / 审查辅助
I have reviewed and validated all AI-assisted content included in this PR.
/ 我已审核并验证此 PR 中的所有 AI 辅助内容。
I have ensured that all AI-assisted commits include
Co-Authored-Byattribution./ 我已确保所有 AI 辅助提交都包含
Co-Authored-By归属信息。I can reproduce all AI-assisted content included in this PR without any AI tools.
/ 我可以在没有任何 AI 工具的情况下重现此 PR 中包含的所有 AI 辅助内容。