fix: harden local replica synchronization

.vscodeignore

@@ -4,6 +4,7 @@ src/**
 patches/**
 docs/**
 .gitignore
+AGENTS.md
 .yarnrc
 **/tsconfig.json
 **/.eslintrc.json

AGENTS.md

@@ -0,0 +1,25 @@
+# Overleaf Workshop Local Fork
+
+## Local Replica Sync
+
+- Runtime local changes are event-driven through VS Code `FileSystemWatcher`; do not add periodic directory or hash polling.
+- `.overleaf/sync-state.json` stores the remote history version and SHA-256 content baseline used only for startup/reconnect reconciliation.
+- `.overleaf/settings.json` is the authoritative local-replica association. It must contain enough project URI and SCM settings metadata to rebuild the transient per-login project/SCM state after authentication expires.
+- Batch startup state changes into one write and skip writes when serialized state is unchanged; do not rewrite the state file once per synchronized path. The file is a disposable cache, so write it in place to avoid delete/create watcher events.
+- History API failures such as HTTP 429 must not be interpreted as a missing version. Leave the checkpoint unchanged and retry on a later reconnect.
+- Reuse recent history updates for both version discovery and changed-path collection. Serialize unavoidable history requests and honor `Retry-After` instead of issuing immediate parallel probes.
+- Route every HTTP request from every project through one process-wide queue. A 429 response pauses the entire queue for `Retry-After`; local watcher bursts are debounced before upload.
+- A failed path must make incremental startup sync fail explicitly; never log completion or advance `remoteVersion` after partial failure.
+- Report final user-actionable failures through a deduplicated VS Code notification with access to the Output log; individual retries remain log-only.
+- Connection and SCM creation logs must include project ID, connection scheme, retry attempt, local base URI, and the original structured error. Never swallow `joinProject` or trigger-initialization errors behind a generic reconnecting message.
+- Disposing a cached VFS is terminal: disconnect handlers must not schedule reconnects after disposal. Successful `Open Project Locally` creation must leave its provider-owned VFS alive.
+- A background project used by `Open Project Locally` must not register workspace-global commands, views, status items, or compile actions. Those features belong only to the project identified by the active workspace; deterministic feature-registration failures must not enter the connection retry loop.
+- Ignore every path containing a dot-prefixed component, including `.output`, before any stat/read/write work.
+- Ignore symbolic links and paths below symbolic-link directories in both directions. Never upload them, overwrite/delete them during a pull, or include them in sync state.
+- Never choose a winner or synthesize a merge when both sides changed. Pause that path, preserve both sides, notify the user, and require manual resolution followed by a window reload. Do not create conflict-copy files.
+- Full remote-to-local sync is automatic only for an empty replica or when all local hashes still match the checkpoint. Missing/invalid history plus uncheckpointed local changes pauses all replica watchers.
+
+## Verification
+
+- Run `npm run compile`, `npm run lint`, and `git diff --check` after synchronization changes.
+- Package local builds with `npx @vscode/vsce package --out overleaf-workshop-local-0.15.10.vsix` and install with `code --install-extension <vsix> --force`.

docs/anatomy.md

@@ -283,10 +283,11 @@ It also proxies commands for `class HistoryDataProvider` via `(get) triggers`.
 #### `src/scm/localReplicaSCM.ts`
 The exported `LocalReplicaSCMProvider` implements the `BaseSCM` interface and supports the ["Open Project Locally"](wiki.md#open-project-locally) feature.
 
-Since the local filesystem keeps no version information, at the start of project open, it always apply `this.overwrite` to overwrite local changes with overleaf server version.
-Therefore, if there is a network disturbance, your local changes or remote changes (by other collaborators) will be lost depending on `syncFromVFS` is called via `this.vfsWatcher` firstly, or `syncToVFS` is called via `this.localWatcher` firstly.
+The provider persists the last synchronized Overleaf project version and SHA-256 hashes in `.overleaf/sync-state.json`. On project open or reconnect, it reuses recent project-history updates to determine both the current version and changed paths. A separate file-tree diff request is only made when those updates do not cover the stored checkpoint. All HTTP requests share one process-wide queue, and rate-limit responses pause that queue according to `Retry-After`. A full remote-to-local synchronization is allowed only when the local replica is empty or still matches its checkpoint. If history is unavailable while local files changed, or if the same path changed on both sides, synchronization pauses without modifying either side.
 
-A smarter solution is proposed in `(private async) overwrite`, but is not applied in the `this.writeFile`.
+The local workspace association is stored in `.overleaf/settings.json`, independently of authentication credentials. When a local replica is reopened after login expiry, the extension rebuilds its transient project/SCM entry from this file and reconnects after the user logs in again; the user does not need to repeat `Open Project Locally`.
+
+If the same text file changed locally and remotely, the existing `diff-match-patch` merge strategy is applied using the previous remote version as the base. If a reliable base cannot be retrieved, the remote version remains authoritative, matching the previous initialization behavior.
 
 #### `src/scm/localGitBridgeSCM.ts`
 > Not completed now. Target to provide local git bridge via [isomorphic-git](https://github.com/isomorphic-git/isomorphic-git) without local file system or git client binary needed.

docs/wiki.md

@@ -323,6 +323,10 @@ In the Local Replica configuration, you can choose to enable/disable the Local R
 
 ![screenshot-config-local-replica](assets/screenshot-config-local-replica.png)
 
+Changes made to files in the local replica by other applications are also detected and uploaded automatically, including while the VS Code window is in the background. Synchronization is driven by VS Code file-system events and does not require an editor save or window focus. Files matching the Local Replica ignore patterns are excluded. Synchronization diagnostics are available in the `Overleaf Workshop` channel in the Output view.
+
+After the first synchronization, `.overleaf/sync-state.json` records the last Overleaf project version and SHA-256 hashes of synchronized files. Subsequent project opens and reconnects reuse the latest history update response for both the current version and changed paths when possible, so unchanged file contents are not downloaded and an additional file-tree diff request is avoided. History requests are serialized and rate-limit responses honor `Retry-After`. State changes produced by startup reconciliation are batched into one write, unchanged state is not rewritten, and the disposable cache is written in place so it does not repeatedly appear as delete/create events. If synchronization ultimately fails, a deduplicated VS Code error notification links to the `Overleaf Workshop` output log. Local symbolic links and paths below symbolic-link directories are excluded from upload, download, deletion, and sync state.
+
 The project-related metadata for local replica are located in `.overleaf/settings.json` in the following format:
 ```json
 {

package.nls.json

@@ -69,4 +69,4 @@
     "views.explorer.overleaf-workshop.chatWebview.contextualTitle": "Overleaf Chat",
 
     "customEditors.overleaf-workshop.pdfViewer.displayName": "Overleaf Workshop PDF Viewer"
-}
+}

src/api/base.ts

@@ -1,17 +1,64 @@
 /* eslint-disable @typescript-eslint/naming-convention */
-import * as stream from 'stream';
-import * as FormData from 'form-data';
+import { Blob } from 'buffer';
 import { v4 as uuidv4 } from 'uuid';
-import { fetch } from 'undici';
+import { fetch, FormData } from 'undici';
 import { FileEntity, FileType, FolderEntity, OutputFileEntity } from '../core/remoteFileSystemProvider';
+import { log } from '../utils/outputChannel';
+
+// Overleaf rate limits are applied to the authenticated user, not to a single
+// BaseAPI instance. Keep every HTTP request in one process-wide queue so
+// several projects cannot collectively trigger a burst of 429 responses.
+let globalRequestChain: Promise<void> = Promise.resolve();
+let globalNextRequestAt = 0;
+let globalRateLimitedUntil = 0;
+const GLOBAL_MIN_REQUEST_INTERVAL_MS = 300;
+
+function retryAfterHeaderMs(response: any): number {
+    const retryAfter = response.headers?.get?.('retry-after');
+    if (retryAfter!==undefined && retryAfter!==null) {
+        const seconds = Number(retryAfter);
+        if (Number.isFinite(seconds)) {
+            return Math.max(1000, seconds*1000);
+        }
+        const retryAt = Date.parse(retryAfter);
+        if (Number.isFinite(retryAt)) { return Math.max(1000, retryAt-Date.now()); }
+    }
+    return 5000;
+}
+
+async function queuedFetch(url: string, init: any): Promise<any> {
+    const previous = globalRequestChain;
+    let release!: () => void;
+    globalRequestChain = new Promise<void>(resolve => { release = resolve; });
+    await previous;
+    try {
+        const delayMs = Math.max(
+            0,
+            globalNextRequestAt-Date.now(),
+            globalRateLimitedUntil-Date.now(),
+        );
+        if (delayMs>0) { await new Promise(resolve => setTimeout(resolve, delayMs)); }
+        globalNextRequestAt = Date.now() + GLOBAL_MIN_REQUEST_INTERVAL_MS;
+        const response = await fetch(url, init);
+        if (response.status===429) {
+            globalRateLimitedUntil = Math.max(globalRateLimitedUntil, Date.now() + retryAfterHeaderMs(response));
+            log(`Global HTTP request queue entered rate-limit cooldown until ${new Date(globalRateLimitedUntil).toISOString()}.`);
+        }
+        return response;
+    } finally {
+        release();
+    }
+}
 
 /** Extract set-cookie headers from an undici/Response object. */
 function getSetCookie(res: any): string[] {
     if (typeof res.headers?.getSetCookie === 'function') {
         return res.headers.getSetCookie();
     }
     const raw = res.headers?.raw?.()?.['set-cookie'];
-    if (raw) return raw;
+    if (raw) {
+        return raw;
+    }
     return [];
 }
 
@@ -177,6 +224,7 @@ export interface ProjectSettingsSchema {
 
 export interface ResponseSchema {
     type: 'success' | 'error';
+    statusCode?: number;
     raw?: ArrayBuffer;
     message?: string;
     userInfo?: {userId:string, userEmail:string};
@@ -202,13 +250,51 @@ export interface ResponseSchema {
 export class BaseAPI {
     private url: string;
     private identity?: Identity;
+    private historyRequestChain: Promise<void> = Promise.resolve();
+    private lastHistoryRequestAt = 0;
+
+    private isHistoryRoute(route: string): boolean {
+        return route.includes('/updates?') || route.includes('/filetree/diff?') || route.includes('/diff?');
+    }
+
+    private async waitForHistoryRequest() {
+        const previous = this.historyRequestChain;
+        let release!: () => void;
+        this.historyRequestChain = new Promise<void>(resolve => { release = resolve; });
+        await previous;
+        try {
+            const minimumIntervalMs = 3000;
+            const delayMs = Math.max(0, minimumIntervalMs - (Date.now() - this.lastHistoryRequestAt));
+            if (delayMs>0) {
+                await new Promise(resolve => setTimeout(resolve, delayMs));
+            }
+            this.lastHistoryRequestAt = Date.now();
+        } finally {
+            release();
+        }
+    }
+
+    private retryAfterMs(response: any, attempt: number): number {
+        const retryAfter = response.headers?.get?.('retry-after');
+        if (retryAfter!==undefined && retryAfter!==null) {
+            const seconds = Number(retryAfter);
+            if (Number.isFinite(seconds)) {
+                return Math.max(1000, seconds*1000);
+            }
+            const retryAt = Date.parse(retryAfter);
+            if (Number.isFinite(retryAt)) {
+                return Math.max(1000, retryAt-Date.now());
+            }
+        }
+        return 5000 * Math.pow(2, attempt);
+    }
 
     constructor(url:string) {
         this.url = url;
     }
 
     private async getCsrfToken(): Promise<Identity> {
-        const res = await fetch(this.url+'login', {
+        const res = await queuedFetch(this.url+'login', {
             method: 'GET', redirect: 'manual',
         });
         const body = await res.text();
@@ -223,7 +309,7 @@ export class BaseAPI {
     }
 
     private async getUserId(cookies:string) {
-        const res = await fetch(this.url+'project', {
+        const res = await queuedFetch(this.url+'project', {
             method: 'GET', redirect:'manual',
             headers: {
                 'Connection': 'keep-alive',
@@ -267,7 +353,7 @@ export class BaseAPI {
 
     async passportLogin(email:string, password:string): Promise<ResponseSchema> {
         const identity = await this.getCsrfToken();
-        const res = await fetch(this.url+'login', {
+        const res = await queuedFetch(this.url+'login', {
             method: 'POST', redirect: 'manual',
             headers: {
                 'Accept': '*/*',
@@ -329,7 +415,7 @@ export class BaseAPI {
     }
 
     async updateCookies(identity: Identity) {
-        const res = await fetch(this.url + 'socket.io/socket.io.js', {
+        const res = await queuedFetch(this.url + 'socket.io/socket.io.js', {
             method: 'GET',
             redirect: 'manual',
             headers: {
@@ -383,10 +469,13 @@ export class BaseAPI {
 
         for (let attempt = 0; attempt <= MAX_HTTP_RETRIES; attempt++) {
             try {
+                if (this.isHistoryRoute(route)) {
+                    await this.waitForHistoryRequest();
+                }
                 let res = undefined;
                 switch(type) {
                     case 'GET':
-                        res = await fetch(this.url+route, {
+                        res = await queuedFetch(this.url+route, {
                             method: 'GET', redirect: 'manual',
                             headers: {
                                 'Connection': 'keep-alive',
@@ -401,7 +490,7 @@ export class BaseAPI {
                             _csrf: this.identity!.csrfToken,
                             ...body
                         });
-                        res = await fetch(this.url+route, {
+                        res = await queuedFetch(this.url+route, {
                             method: 'POST', redirect: 'manual',
                             headers: {
                                 'Connection': 'keep-alive',
@@ -415,7 +504,7 @@ export class BaseAPI {
                     case 'PUT':
                         break;
                     case 'DELETE':
-                        res = await fetch(this.url+route, {
+                        res = await queuedFetch(this.url+route, {
                             method: 'DELETE', redirect: 'manual',
                             headers: {
                                 'Connection': 'keep-alive',
@@ -436,8 +525,8 @@ export class BaseAPI {
                     } as ResponseSchema;
                 } else if (res && this.isTransientError(res.status) && attempt < MAX_HTTP_RETRIES) {
                     // Transient error: retry with backoff
-                    const delayMs = Math.min(1000 * Math.pow(2, attempt), 4000);
-                    console.log(`HTTP ${res.status} on ${route}, retrying in ${delayMs}ms (attempt ${attempt + 1}/${MAX_HTTP_RETRIES})`);
+                    const delayMs = res.status===429 ? this.retryAfterMs(res, attempt) : Math.min(1000 * Math.pow(2, attempt), 4000);
+                    log(`HTTP ${res.status} on ${route}, retrying in ${delayMs}ms (attempt ${attempt + 1}/${MAX_HTTP_RETRIES})`);
                     lastError = {statusCode: res.status, message: await res.text().catch(() => '')};
                     await new Promise(r => setTimeout(r, delayMs));
                     continue;
@@ -447,19 +536,21 @@ export class BaseAPI {
                     try { errorBody = await resOrFallback.text(); } catch { errorBody = ''; }
                     return {
                         type: 'error',
+                        statusCode: typeof resOrFallback.status==='number' ? resOrFallback.status : undefined,
                         message: `${resOrFallback.status}: ${errorBody}`
                     };
                 }
             } catch (err: any) {
                 const errMsg = err?.message || String(err);
                 if (this.isTransientError(undefined, errMsg) && attempt < MAX_HTTP_RETRIES) {
                     const delayMs = Math.min(1000 * Math.pow(2, attempt), 4000);
-                    console.log(`HTTP fetch error on ${route}: ${errMsg}, retrying in ${delayMs}ms (attempt ${attempt + 1}/${MAX_HTTP_RETRIES})`);
+                    log(`HTTP fetch error on ${route}: ${errMsg}, retrying in ${delayMs}ms (attempt ${attempt + 1}/${MAX_HTTP_RETRIES})`);
                     await new Promise(r => setTimeout(r, delayMs));
                     continue;
                 }
                 return {
                     type: 'error',
+                    statusCode: undefined,
                     message: errMsg
                 };
             }
@@ -468,6 +559,7 @@ export class BaseAPI {
         // All retries exhausted
         return {
             type: 'error',
+            statusCode: lastError.statusCode,
             message: lastError.message || `Request failed after ${MAX_HTTP_RETRIES + 1} attempts`
         };
     }
@@ -477,7 +569,7 @@ export class BaseAPI {
 
         let content: Buffer[] = [];
         while(true) {
-            const res = await fetch(this.url+route, {
+            const res = await queuedFetch(this.url+route, {
                 method: 'GET', redirect: 'manual',
                 headers: {
                     'Connection': 'keep-alive',
@@ -598,13 +690,14 @@ export class BaseAPI {
     }
 
     async uploadFile(identity:Identity, projectId:string, parentFolderId:string, filename:string, fileContent:Uint8Array) {
-        const fileStream = stream.Readable.from(fileContent);
         const formData = new FormData();
         const mimeType = require('mime-types').lookup(filename);
         formData.append('targetFolderId', parentFolderId);
         formData.append('name', filename);
         formData.append('type', mimeType? mimeType : 'text/plain');
-        formData.append('qqfile', fileStream, {filename});
+        formData.append('qqfile', new Blob([Buffer.from(fileContent)], {
+            type: mimeType ? mimeType : 'application/octet-stream',
+        }), filename);
 
         this.setIdentity(identity);
         return this.request('POST', `project/${projectId}/upload?folder_id=${parentFolderId}`, formData, (res) => {
@@ -616,9 +709,10 @@ export class BaseAPI {
 
     async uploadProject(identity:Identity, filename:string, fileContent:Uint8Array) {
         const uuid = uuidv4();
-        const fileStream = stream.Readable.from(fileContent);
         const formData = new FormData();
-        formData.append('qqfile', fileStream, {filename});
+        formData.append('qqfile', new Blob([Buffer.from(fileContent)], {
+            type: 'application/zip',
+        }), filename);
 
         this.setIdentity(identity);
         return this.request('POST', `project/new/upload?_csrf=${identity.csrfToken}&qquuid=${uuid}&qqfilename=${filename}&qqtotalfilesize=${fileContent.length}`, formData, (res) => {
@@ -786,7 +880,7 @@ export class BaseAPI {
         }
         let content: Buffer[] = [];
         while (true) {
-            const res = await fetch(absoluteUrl, {
+            const res = await queuedFetch(absoluteUrl, {
                 method: 'GET', redirect: 'manual',
                 headers
             });