feat(dpu-remediation): count executor failures

[chet]

DPU remediation now exposes executor failures as a bounded per-stage counter while retaining the existing retry diagnostics.

• Covers status decoding, local apply/report, response validation, remediation fetch, and Forge client creation failures.
• Keeps machine and remediation identifiers, error text, and response booleans in log context rather than metric labels.
• Leaves delivered script failures and timeouts on the existing result-reporting path instead of counting them as executor failures.
• Covers every failure_stage with table-driven log and counter assertions.

Related issues

This supports #3911

Type of Change

• Add - New feature or capability
• Change - Changes in existing functionality
• Fix - Bug fixes
• Remove - Removed features or deprecated functionality
• Internal - Internal changes (refactoring, tests, docs, etc.)

Breaking Changes

• This PR contains breaking changes

Testing

• Unit tests added/updated

• Integration tests added/updated

• Manual testing performed

• No testing required (docs, internal refactor, etc.)

• cargo test -p carbide-dpu-remediation

• cargo make format-nightly

• cargo make clippy

• cargo make carbide-lints

• cargo make check-event-names

• cargo xtask check-metric-docs

• cargo xtask check-workspace-deps

• cargo make check-licenses

• cargo make check-bans

Additional Notes

A remediation script reporting failure remains a delivered script result and does not increment the executor-failure counter by itself.

This replaces #3956 after GitHub stopped synchronizing that PR's fork head following its hosted-review amendment. The two CodeRabbit nitpicks from that review are included here.

Closes #3911

[chet]

@coderabbitai full_review, thanks!

[coderabbitai]

@chet Understood — I’ll perform a complete re-review of the pull request. 🐇

✅ Action performed

Full review finished.

---

Your included review limit is currently reached under our Fair Usage Limits Policy. This review may still proceed through usage-based billing if eligible. Your next included review will be available in 5 minutes.

[coderabbitai]

Walkthrough

The DPU remediation executor adds stage-labeled failure metrics and structured events for status decoding, application, response validation, fetching, and client creation. Event emission replaces selected error logging, with unit tests and core metric documentation added.

Changes

DPU remediation observability

Layer / File(s)	Summary
Failure event contract and validation crates/dpu-remediation/Cargo.toml, crates/dpu-remediation/src/lib.rs, crates/dpu-remediation/src/metrics.rs	Defines five structured failure events sharing the failure_stage label, registers the module, adds instrumentation dependencies, and tests emitted records and counter increments.
Executor failure emission crates/dpu-remediation/src/remediation.rs, docs/observability/core_metrics.md	Emits structured events for remediation execution failures and status decoding failures, and documents the new counter metric.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related issues

• NVIDIA/infra-controller issue 3911: Directly covers the DPU remediation executor failure metrics and structured events implemented here.

Possibly related PRs

• NVIDIA/infra-controller#3956: Directly corresponds to the failure-stage counter and remediation executor event instrumentation.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly and concisely summarizes the main change: counting DPU remediation executor failures.
Description check	✅ Passed	The description is directly aligned with the changeset and accurately describes the new failure counter behavior and test coverage.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[chet]

/ok-to-test

[chet]

@coderabbitai full_review, thanks!

[coderabbitai]

@chet Understood — I’ll perform a complete re-review of the pull request, including the executor-failure metric paths and test coverage.

🐇

✅ Action performed

Full review finished.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pull-request-3962.docs.buildwithfern.com/infra-controller