Fix/improve load in memory paths

[AlexisMora]

What is Changed / Added

This PR improves the startup placeholder refresh path, especially the local placeholder scan performed by loadInMemoryPaths.

The file placeholder loading was also extracted into its own helper and changed to workers. This avoids creating one promise per file upfront when a folder contains many files. statReaddir now follows the same idea for filesystem stats, so a large directory does not launch one stat operation per entry at once

The recursive folder walk has been replaced with an iterative depth-first traversal helper under backend/features/virtual-drive/tree-traversal. I have made this function reusable so that we can use it in both loadInMemoryPaths and traverse keeping the traversal mechanics in one place while preserving the existing behavior.

---

Why

Users with very large drives have reported startup freezes and long periods where the app appears unresponsive. loadInMemoryPaths is part of that startup path, and it has to inspect a large local placeholder tree before comparing local state with remote state.

Summary by CodeRabbit

• Improvements
  • Improved remote file synchronization for nested folders and large directory structures.
  • Added bounded processing to reduce resource usage during file and folder scans.
  • Improved handling of missing or unreadable files so processing can continue when individual entries fail.
  • Added support for stopping long-running folder scans when cancellation is requested.
  • Improved reliability when processing deeply nested folder hierarchies.

[github-actions]

This PR changes 651 lines.

If it contains multiple concerns, consider splitting it. Large PRs are fine when the changes are mechanical, intentionally grouped, or belong together as part of the same feature.

Please make sure the PR description gives reviewers enough context about what matters most to review.

[coderabbitai]

📝 Walkthrough

Walkthrough

The change adds iterative depth-first traversal, bounds filesystem stat and placeholder-loading concurrency, refactors remote-sync path loading to use the traversal helper, and updates virtual-drive traversal to use shared worker-count logic. Tests cover ordering, cancellation, deep trees, metadata propagation, concurrency, and errors.

Changes

Filesystem traversal and concurrency

Layer / File(s)	Summary
Bounded directory entry processing src/core/utils/concurrency.ts, src/core/utils/concurrency.test.ts, src/infra/file-system/services/stat-readdir.*	Adds getWorkerCount and applies a 20-worker bound to fileSystem.stat calls while retaining per-entry error handling and expanded coverage.
Iterative depth-first traversal src/backend/features/virtual-drive/tree-traversal/traverse-depth-first.*	Adds stack-based asynchronous traversal with ordered children, subtree skipping, abort handling, and deep-tree tests.
Remote-sync path loading src/backend/features/remote-sync/sync-items-by-checkpoint/load-in-memory-paths.*, src/backend/features/remote-sync/sync-items-by-checkpoint/load-file-explorer-files.*	Refactors path loading around depth-first callbacks and loads file placeholders concurrently with propagated parent and metadata fields.
Virtual-drive traversal integration src/context/virtual-drive/items/application/Traverser.ts	Replaces manual folder stack processing with traverseDepthFirst, extracts child processing, and uses shared worker-count sizing.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related PRs

• internxt/drive-desktop#1446: Refactors related folder placeholder traversal and file-processing concurrency in Traverser.ts.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is concise and points to the main area changed, even though it is somewhat broad.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/improve-loadInMemoryPaths

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

src/backend/features/remote-sync/sync-items-by-checkpoint/load-in-memory-paths.ts (1)

22-72: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Cross-callback state via item mutation is implicit.

loadFolder mutates item.uuid in place (Line 52) so that processChildren's later getFolderUuid (Line 42) can read it. This is correct today only because traverseDepthFirst happens to pass the same object reference to both processNode and processChildren for a given stack item — an implicit contract not documented on either side. Works correctly, but worth a comment noting the dependency so a future change to the traversal helper doesn't silently break this.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@src/backend/features/remote-sync/sync-items-by-checkpoint/load-in-memory-paths.ts`
around lines 22 - 72, Document the shared-state dependency between loadFolder
and processChildren in loadInMemoryPaths: loadFolder mutates item.uuid for
placeholder folders, and processChildren later reads that value through
getFolderUuid. Add a concise comment near the mutation or callback wiring noting
that traverseDepthFirst must pass the same StackItem reference to both
callbacks; do not change the existing behavior.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In
`@src/backend/features/remote-sync/sync-items-by-checkpoint/load-in-memory-paths.ts`:
- Around line 22-72: Document the shared-state dependency between loadFolder and
processChildren in loadInMemoryPaths: loadFolder mutates item.uuid for
placeholder folders, and processChildren later reads that value through
getFolderUuid. Add a concise comment near the mutation or callback wiring noting
that traverseDepthFirst must pass the same StackItem reference to both
callbacks; do not change the existing behavior.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: a1a22b44-b937-4d9d-9e99-e417ac27d47a

📥 Commits

Reviewing files that changed from the base of the PR and between 7f6d1b0 and 535da6f.

📒 Files selected for processing (11)

• src/backend/features/remote-sync/sync-items-by-checkpoint/load-file-explorer-files.test.ts
• src/backend/features/remote-sync/sync-items-by-checkpoint/load-file-explorer-files.ts
• src/backend/features/remote-sync/sync-items-by-checkpoint/load-in-memory-paths.test.ts
• src/backend/features/remote-sync/sync-items-by-checkpoint/load-in-memory-paths.ts
• src/backend/features/virtual-drive/tree-traversal/traverse-depth-first.test.ts
• src/backend/features/virtual-drive/tree-traversal/traverse-depth-first.ts
• src/context/virtual-drive/items/application/Traverser.ts
• src/core/utils/concurrency.test.ts
• src/core/utils/concurrency.ts
• src/infra/file-system/services/stat-readdir.test.ts
• src/infra/file-system/services/stat-readdir.ts

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
96.6% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

The merge-base changed after approval.