Skip to content

fix: Always write sync status instead of comparing against stale cache - #261

Merged
wikoion merged 2 commits into
mainfrom
wikoion/always-write-sync-status
Sep 2, 2026
Merged

fix: Always write sync status instead of comparing against stale cache#261
wikoion merged 2 commits into
mainfrom
wikoion/always-write-sync-status

Conversation

@wikoion

@wikoion wikoion commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Follow-up to #260. The status patch was still gated on comparing the computed status against the cached object, which races with our own writes: a retry can start before the cache has seen the failure patch, compute False equal to the stale cached status, skip the patch, and leave the condition latched at True. We're seeing exactly this in prod, a reconcile fails, succeeds 90ms later on retry, and the condition stays True indefinitely.

  • reconcile now always writes the status for both success and failure. No-op patches don't bump resourceVersion, and the generation predicate on the watch means status writes don't retrigger reconciles, so there's no churn or loop risk.
  • reconcile_status returns None for a successful reconcile of a deleted resource, meaning don't write, since the finalizer may already be gone and the patch could 404.

@wikoion
wikoion requested a review from a team as a code owner September 2, 2026 22:50
@wikoion
wikoion requested a balanced review from Copilot September 2, 2026 22:52

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Stale cached status can still produce an incorrect lastTransitionTime.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Ensures reconciliation status patches are always written to avoid stale-cache races.

Changes:

  • Always patches computed status.
  • Skips status writes after successful deletion reconciliation.
  • Updates the deleted-resource test.
File summaries
File Description
src/controller.rs Revises status patching and deletion behavior.
Review details
  • Files reviewed: 1/1 changed files
  • Comments generated: 1
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/controller.rs Outdated
// Always write the status. Deciding off the cached resource_sync.status races with our own
// patches: a retry can start before the cache has seen the previous write, skip the patch, and
// leave the condition latched at the wrong value. A no-op patch does not bump resourceVersion.
if let Some(status) = reconcile_status(&resource_sync, &result) {
@wikoion
wikoion enabled auto-merge September 2, 2026 23:29
@wikoion
wikoion added this pull request to the merge queue Sep 2, 2026
Merged via the queue into main with commit 3e28e75 Sep 2, 2026
3 checks passed
@wikoion
wikoion deleted the wikoion/always-write-sync-status branch September 2, 2026 23:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants