fix(machine-controller): report DPU restart outcome accurately

[behroozrafii]

DPU restart handling logged success and queued the reboot-request timestamp before the Redfish restart completed. When the restart failed, logs could falsely imply success, and the lower-level failure log did not identify the affected DPU.

This change:

• Logs the restart attempt before calling Redfish.
• Logs failures with both dpu_machine_id and the returned error.
• Records the reboot-request timestamp and logs success only after the restart succeeds.
• Removes the redundant lower-level failure log while preserving the original error.

This gives operators an accurate restart outcome and identifies the failed DPU without changing any API or configuration.

Related issues

Fixes #3713

Type of Change

• Add - New feature or capability
• Change - Changes in existing functionality
• Fix - Bug fixes
• Remove - Removed features or deprecated functionality
• Internal - Internal changes (refactoring, tests, docs, etc.)

Breaking Changes

• This PR contains breaking changes

Testing

• Unit tests added/updated
• Integration tests added/updated
• Manual testing performed
• No testing required (docs, internal refactor, etc.)
• Existing unit tests run:
  • cargo test -p carbide-machine-controller --lib
  • 56 passed, 0 failed, 0 ignored, 0 measured, 0 filtered out
• Formatting verified:
  • cargo fmt --all -- --check

Additional Notes

No API, configuration, or database-schema changes.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 0fa5ee27-c810-46b2-8bab-603a9ba0f2a7

📥 Commits

Reviewing files that changed from the base of the PR and between 4fa5d42 and e930b6e.

📒 Files selected for processing (1)

• crates/machine-controller/src/handler.rs

🚧 Files skipped from review as they are similar to previous changes (1)

• crates/machine-controller/src/handler.rs

---

Summary by CodeRabbit

• Bug Fixes
  • DPU restart requests now update reboot status only after the restart succeeds.
  • If a restart fails, the system reports the failure immediately and avoids recording an incorrect reboot-request timestamp.
  • Simplified restart error handling to improve consistency of failure reporting.

Walkthrough

The DPU restart handler now performs the Redfish restart before recording reboot-request state or logging success. Restart errors propagate directly, preventing database updates and success logs when the operation fails.

Changes

DPU restart sequencing

Layer / File(s)	Summary
Restart execution and success-gated persistence crates/machine-controller/src/handler.rs	restart_dpu directly propagates Redfish restart errors, while handler_restart_dpu records UpdateRebootRequestedTime and logs success only after a successful restart.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• NVIDIA/infra-controller#3978: Updates another DPU restart path that conditionally invokes reboot helpers, overlapping with this change’s restart execution flow.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: fixing DPU restart outcome reporting.
Description check	✅ Passed	The description matches the code changes and explains the restart logging and state update fix.
Linked Issues check	✅ Passed	The changes align with #3713 by logging before restart, delaying success until after success, and preserving the DPU ID on failure.
Out of Scope Changes check	✅ Passed	The patch stays focused on DPU restart handling in a single handler file and introduces no unrelated changes.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[wminckler]

pending_db_writes is suppose to be discarded when restart_dpu fails (the result from restart_dpu is the result for this function).

I feel like there's a bigger problem that is not fixed by this change.

[wminckler]

so my issue isn't with the code apparently. the description says:

Records the reboot-request timestamp and logs success only after the restart succeeds.

which the timestamp update doesn't happen. that's what the pending_db_writes vector is for

[wminckler]

as I think about it more, the timestamp update should be done unconditionally if the restart works, right? if something else fails, do we not want the timestamp update?