feat(api-db): count work-lock failures

[chet]

WorkLock now exposes release and keepalive failures through one bounded counter while retaining each existing diagnostic.

• Replaces four error boundaries with Events sharing carbide_work_lock_failures_total.
• Distinguishes database, command-dispatch, command-reply, and lock-loss failures; releasing an expired lock stays a lock-loss condition rather than a database failure.
• Keeps work keys, worker identifiers, and error details in correlated log context instead of metric labels.
• Covers all seven reachable operation/failure series, the KeepAliveError mapping, and the production expiry path.

Related issues

This supports #3910

Type of Change

• Add - New feature or capability
• Change - Changes in existing functionality
• Fix - Bug fixes
• Remove - Removed features or deprecated functionality
• Internal - Internal changes (refactoring, tests, docs, etc.)

Breaking Changes

• This PR contains breaking changes

Testing

• Unit tests added/updated

• Integration tests added/updated

• Manual testing performed

• No testing required (docs, internal refactor, etc.)

• cargo test -p carbide-api-db work_lock_manager --lib

• cargo make format-nightly

• cargo make clippy

• cargo make carbide-lints

• cargo make check-event-names

• cargo xtask check-metric-docs

• cargo xtask check-workspace-deps

• cargo make check-licenses

• cargo make check-bans

Additional Notes

Expected lock contention and connection-health records remain plain tracing. An expired lock that cannot be released is labeled lock_lost, not database.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 5b770803-8226-41ab-b52d-ae26911012aa

📥 Commits

Reviewing files that changed from the base of the PR and between 2de970b and 69b08c7.

⛔ Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (3)

• crates/api-db/Cargo.toml
• crates/api-db/src/work_lock_manager.rs
• docs/observability/core_metrics.md

---

Summary by CodeRabbit

• Observability
  • Added monitoring for work-lock lifecycle failures, including release, keepalive, dispatch, database, and lost-lock errors.
  • Added the carbide_work_lock_failures_total counter with operation and failure-type labels.
• Documentation
  • Documented the new work-lock failure metric in the core metrics reference.

Walkthrough

Work-lock release and keepalive failures now emit structured carbide_instrument events, increment carbide_work_lock_failures_total with bounded labels, and are covered by log and metric assertions. The metric is also added to the core observability documentation.

Changes

Work-lock failure instrumentation

Layer / File(s)	Summary
Failure event contracts and dependencies crates/api-db/Cargo.toml, crates/api-db/src/work_lock_manager.rs	Adds the instrumentation dependency and defines operation, failure, and event types for work-lock failures.
Runtime failure emission crates/api-db/src/work_lock_manager.rs	Emits structured events for release dispatch, release command, lock loss, and other keepalive failures while preserving existing loop behavior.
Metrics validation and documentation crates/api-db/src/work_lock_manager.rs, docs/observability/core_metrics.md	Tests emitted records and labeled counter increments, and documents carbide_work_lock_failures_total as a core counter.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant WorkLockManager
  participant WorkLock
  participant KeepAliveLoop
  participant carbide_instrument
  WorkLockManager->>carbide_instrument: Emit release failure event
  WorkLock->>carbide_instrument: Emit release dispatch failure event
  KeepAliveLoop->>carbide_instrument: Emit lock-lost or keepalive failure event
  carbide_instrument-->>carbide_work_lock_failures_total: Increment labeled counter

Loading

Possibly related issues

• feat(api-db): count work-lock failures #3910 — Directly covers the work-lock failure instrumentation, metric, dependency, and test changes in this pull request.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 46.15% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly states the main change: counting work-lock failures in api-db.
Description check	✅ Passed	The description matches the changeset by describing the bounded counter, failure classes, and test coverage.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[chet]

@coderabbitai full_review, thanks!

[coderabbitai]

@chet Understood — I’ll perform a complete re-review of #3961, including the WorkLock failure classification, metric cardinality, error mappings, and test coverage.

🐇

✅ Action performed

Full review finished.

---

Your included review limit is currently reached under our Fair Usage Limits Policy. This review may still proceed through usage-based billing if eligible. Your next included review will be available in 6 minutes.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pull-request-3961.docs.buildwithfern.com/infra-controller

[chet]

@coderabbitai full_review, thanks!

[coderabbitai]

@chet Understood — I’ll perform a complete re-review of #3961, with particular attention to failure classification, metric cardinality, error mappings, and coverage of the production expiry path.

🐇

✅ Action performed

Full review finished.