Skip to content

[WRONG BRANCH] fix(lab): retain grace period for stale ledger locks #297

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
luvs01 wants to merge 1 commit into from
Closed

[WRONG BRANCH] fix(lab): retain grace period for stale ledger locks #297

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions src/lab/ledger/store.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ function isLockHolderAlive(pid: number): boolean {
}
}

/** Return true when a ledger lock file has dead metadata and can be recovered. */
/** Return true when a ledger lock file has old, dead metadata and can be recovered. */
function isLedgerLockStale(lockPath: string): boolean {
const meta = readLedgerLockMeta(lockPath);
if (!meta) {
Expand All @@ -86,7 +86,12 @@ function isLedgerLockStale(lockPath: string): boolean {
return false;
}
}
return !isLockHolderAlive(meta.pid);
// A dead PID alone is not enough: between checking its liveness and
// unlinking, another process can replace the path with its own live lock.
// Keeping the same grace period as malformed locks prevents a freshly
// replaced lock from being selected for stale recovery.
return Date.now() - meta.createdAt > LEDGER_LOCK_STALE_MS
&& !isLockHolderAlive(meta.pid);
Comment on lines +93 to +94

@chatgpt-codex-connector chatgpt-codex-connector Bot Aug 16, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Recover recently orphaned locks without waiting a minute

When a ledger owner crashes or is killed less than 60 seconds after acquiring the lock, this new age condition treats its lock as live even though the PID check would prove the owner is gone. tryAcquireLedgerLock only waits 5 seconds (LEDGER_LOCK_WAIT_MS), so every Lab mutation during the remaining grace period synchronously blocks for five seconds and then fails with EEXIST, preventing experiment results and other ledger updates from being persisted after a normal crash/restart. Preserve the ABA protection by making recovery conditional on the identity of the lock that was inspected, rather than delaying recovery of every dead owner beyond the acquisition deadline.

Useful? React with 👍 / 👎.

}

/** Write lock ownership metadata to a newly created exclusive lock file. */
Expand Down Expand Up @@ -529,4 +534,4 @@ export function openLedgerStore(configDir?: string): LedgerStore {

export function defaultLedgerPath(configDir?: string): string {
return labLedgerPath(configDir);
}
}
4 changes: 2 additions & 2 deletions tests/lab-ledger-mutation-lock.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@ test("appendLabEvent waits for the shared ledger mutation lock", async () => {
}
});

test("appendLabEventIfAbsent immediately recovers a lock owned by an exited process", async () => {
test("appendLabEventIfAbsent recovers an aged lock owned by an exited process", async () => {
const home = tempHome();
const ledgerPath = join(home, "lab", "compatibility.jsonl");
const lockPath = `${ledgerPath}.lock`;
Expand All @@ -158,7 +158,7 @@ test("appendLabEventIfAbsent immediately recovers a lock owned by an exited proc
import { writeFileSync } from "node:fs";
writeFileSync(
${JSON.stringify(lockPath)},
JSON.stringify({ pid: process.pid, createdAt: Date.now(), token: "dead-holder" }),
JSON.stringify({ pid: process.pid, createdAt: Date.now() - 61_000, token: "dead-holder" }),
{ mode: 0o600 },
);
writeFileSync(${JSON.stringify(readyPath)}, "ready");
Expand Down
Loading