Skip to content

CNTRLPLANE-3791: make pre-commit hooks resilient to mock generation failures#9016

Open
bryan-cox wants to merge 1 commit into
openshift:mainfrom
bryan-cox:CNTRLPLANE-3791
Open

CNTRLPLANE-3791: make pre-commit hooks resilient to mock generation failures#9016
bryan-cox wants to merge 1 commit into
openshift:mainfrom
bryan-cox:CNTRLPLANE-3791

Conversation

@bryan-cox

@bryan-cox bryan-cox commented Jul 15, 2026

Copy link
Copy Markdown
Member

What this PR does / why we need it:

The periodic review agent fails when it modifies test files that depend on generated mocks. The root cause is three interacting issues in the repo's build tooling:

  1. The generate target destructively deletes all *_mock.go files before regenerating — if go generate fails for any reason, all mocks are gone and nothing compiles
  2. PULL_BASE_SHA resolution fails in environments without an openshift/hypershift remote, causing api-lint-fix to error out
  3. fail_fast: true aborts all hooks on the first failure, leaving the working tree in a partially-modified state

This PR fixes all three:

  • Remove destructive find -delete from generate targetgo generate overwrites mock files in place, so deleting first is unnecessary and fragile
  • Make PULL_BASE_SHA fall back gracefully — upstream remote/main → local main → empty (omit --new-from-rev, lint all files)
  • Split make-lint-fix into two scoped hooksapi-lint-fix (API Go files only) and main-lint-fix (main module Go files only), so API import reordering doesn't fire on unrelated commits
  • Change fail_fast to false — all hooks run so all problems are visible in one pass

Which issue(s) this PR fixes:

Fixes https://redhat.atlassian.net/browse/CNTRLPLANE-3791

Special notes for your reviewer:

The Jira description originally cited git clean -fx -- '*_mock.go' in a pre-commit hook — this command doesn't exist in the repo. The actual destructive step is find . -name '*_mock.go' -delete in the Makefile generate target, which is transitively invoked by the pre-commit hook via make lint-fixmake generate.

The main-lint-fix target includes $(GOLANGCI_LINT) as a prerequisite so it works on fresh checkouts where the binary hasn't been built yet.

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Summary by CodeRabbit

  • Documentation

    • Clarified that make generate regenerates mock files in place and that generated files shouldn’t be edited manually.
  • Developer Experience

    • Improved linting workflows for API and main code, including safer behavior when the comparison base isn’t available.
    • Added a dedicated main-code lint-fix target.
    • Updated pre-commit linting to use separate Go import-sorting hooks.
    • Mock generation no longer deletes existing generated mock files before regeneration.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 15, 2026
@openshift-ci-robot

openshift-ci-robot commented Jul 15, 2026

Copy link
Copy Markdown

@bryan-cox: This pull request references CNTRLPLANE-3791 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "5.0.0" version, but no target version was set.

Details

In response to this:

What this PR does / why we need it:

The periodic review agent fails when it modifies test files that depend on generated mocks. The root cause is three interacting issues in the repo's build tooling:

  1. The generate target destructively deletes all *_mock.go files before regenerating — if go generate fails for any reason, all mocks are gone and nothing compiles
  2. PULL_BASE_SHA resolution fails in environments without an openshift/hypershift remote, causing api-lint-fix to error out
  3. fail_fast: true aborts all hooks on the first failure, leaving the working tree in a partially-modified state

This PR fixes all three:

  • Remove destructive find -delete from generate targetgo generate overwrites mock files in place, so deleting first is unnecessary and fragile
  • Make PULL_BASE_SHA fall back gracefully — upstream remote/main → local main → empty (omit --new-from-rev, lint all files)
  • Split make-lint-fix into two scoped hooksapi-lint-fix (API Go files only) and main-lint-fix (main module Go files only), so API import reordering doesn't fire on unrelated commits
  • Change fail_fast to false — all hooks run so all problems are visible in one pass

Which issue(s) this PR fixes:

Fixes https://redhat.atlassian.net/browse/CNTRLPLANE-3791

Special notes for your reviewer:

The Jira description originally cited git clean -fx -- '*_mock.go' in a pre-commit hook — this command doesn't exist in the repo. The actual destructive step is find . -name '*_mock.go' -delete in the Makefile generate target, which is transitively invoked by the pre-commit hook via make lint-fixmake generate.

The main-lint-fix target includes $(GOLANGCI_LINT) as a prerequisite so it works on fresh checkouts where the binary hasn't been built yet.

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 15, 2026
@openshift-ci

openshift-ci Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@coderabbitai

coderabbitai Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 7242ef6c-d33d-43db-8c19-c214e2e74694

📥 Commits

Reviewing files that changed from the base of the PR and between 06e2b44 and fa3dde5.

📒 Files selected for processing (3)
  • .pre-commit-config.yaml
  • DEVELOPMENT.md
  • Makefile
🚧 Files skipped from review as they are similar to previous changes (3)
  • DEVELOPMENT.md
  • .pre-commit-config.yaml
  • Makefile

📝 Walkthrough

Walkthrough

The pre-commit configuration separates API and main Go lint-fix hooks and allows later hooks to run after failures. Makefile lint targets now tolerate missing base references, conditionally pass revision options, and add main-lint-fix. Mock generation no longer deletes existing mock files before running go generate, and the development documentation reflects this behavior.

Suggested reviewers: devguyio, muraee

🚥 Pre-merge checks | ✅ 11
✅ Passed checks (11 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly captures the main change: making pre-commit hooks more resilient around mock generation failures.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The PR only changes pre-commit, docs, and Makefile files; no Ginkgo test titles were added or modified.
Test Structure And Quality ✅ Passed PR only changes pre-commit, docs, and Makefile; no Ginkgo test code was modified.
Topology-Aware Scheduling Compatibility ✅ Passed Only Makefile/docs/pre-commit config changed; no deployment manifests, operator code, or controllers were modified.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PR only changes Makefile/docs/pre-commit config; no new Ginkgo e2e tests or network assumptions were added.
No-Weak-Crypto ✅ Passed Touched files only change pre-commit, docs, and Makefile lint/generate tooling; no weak crypto, custom crypto, or secret comparisons are introduced.
Container-Privileges ✅ Passed The PR only changes pre-commit/docs/Makefile files; no container/K8s manifests or privilege settings (privileged, hostPID, allowPrivilegeEscalation, etc.) are present.
No-Sensitive-Data-In-Logs ✅ Passed No new logging or output of secrets/PII/internal hostnames was added; the only echoed text was a removed benign status message.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@openshift-ci

openshift-ci Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bryan-cox

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 15, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
Makefile (1)

193-194: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Keep the mock cleanup step in generate
Removing it can leave stale *_mock.go files behind when interfaces are renamed or deleted, because go generate ./... won’t prune orphaned outputs.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@Makefile` around lines 193 - 194, Restore the mock cleanup step in the
Makefile generate target before running $(GO) generate ./..., ensuring stale
*_mock.go files are removed when interfaces change while preserving the existing
mock generation command.
🧹 Nitpick comments (3)
Makefile (1)

127-135: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

lint-fix duplicates the main-lint-fix command instead of reusing it.

Now that main-lint-fix (Lines 127-129) encapsulates the exact main-module fix command, lint-fix (Lines 132-135) could call $(MAKE) main-lint-fix instead of repeating the raw $(GOLANGCI_LINT) run --config ./.golangci.yml --fix -v invocation, avoiding two places to keep in sync.

♻️ Proposed refactor
 lint-fix: generate
-	$(MAKE) api-lint-fix; api_rc=$$?; \
-	$(GOLANGCI_LINT) run --config ./.golangci.yml --fix -v; main_rc=$$?; \
+	$(MAKE) api-lint-fix; api_rc=$$?; \
+	$(MAKE) main-lint-fix; main_rc=$$?; \
 	exit $$(( api_rc > main_rc ? api_rc : main_rc ))
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@Makefile` around lines 127 - 135, Update the lint-fix target to invoke
$(MAKE) main-lint-fix instead of duplicating the raw golangci-lint command,
while preserving the existing api_rc/main_rc status aggregation and exit
behavior. Keep main-lint-fix as the single definition of the main-module fix
command.
.pre-commit-config.yaml (2)

38-38: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Hook names/descriptions understate actual scope.

Both hooks are named/described as sorting imports, but they run golangci-lint run --fix, which applies fixes for every enabled linter/formatter with autofix support, not just import sorting. Consider renaming/describing them to reflect the broader auto-fix behavior so contributors aren't surprised by unrelated changes appearing in their diffs.

Also applies to: 45-45

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.pre-commit-config.yaml at line 38, Rename and update the descriptions of
both hooks around “Sort API imports” and the corresponding hook at the
additional location to reflect that they run golangci-lint autofixes broadly,
not only import sorting. Ensure the wording clearly communicates that enabled
linter and formatter fixes may modify unrelated code.

37-51: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Hook files patterns don't exclude vendor/.

api-lint-fix (^api/.*\.go$) and main-lint-fix (\.go$ excluding ^api/) will also match staged .go files under vendor/ (main module) or api/vendor/ (API module), if such files are ever staged (e.g. after go mod vendor). DEVELOPMENT.md explicitly states vendor directories should not be modified directly and are managed by make update; auto-fixing vendored files here would work against that guidance.

♻️ Proposed fix
       - id: api-lint-fix
         ...
         files: '^api/.*\.go$'
+        exclude: '(^|/)vendor/'
       - id: main-lint-fix
         ...
         files: '\.go$'
-        exclude: '^api/'
+        exclude: '(^api/|(^|/)vendor/)'
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.pre-commit-config.yaml around lines 37 - 51, Update the files patterns for
the api-lint-fix and main-lint-fix hooks so they exclude vendor directories,
including api/vendor/ and the main module’s vendor/. Preserve the existing
module separation and ensure staged Go files under any vendor path are not
auto-fixed.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@Makefile`:
- Around line 193-194: Restore the mock cleanup step in the Makefile generate
target before running $(GO) generate ./..., ensuring stale *_mock.go files are
removed when interfaces change while preserving the existing mock generation
command.

---

Nitpick comments:
In @.pre-commit-config.yaml:
- Line 38: Rename and update the descriptions of both hooks around “Sort API
imports” and the corresponding hook at the additional location to reflect that
they run golangci-lint autofixes broadly, not only import sorting. Ensure the
wording clearly communicates that enabled linter and formatter fixes may modify
unrelated code.
- Around line 37-51: Update the files patterns for the api-lint-fix and
main-lint-fix hooks so they exclude vendor directories, including api/vendor/
and the main module’s vendor/. Preserve the existing module separation and
ensure staged Go files under any vendor path are not auto-fixed.

In `@Makefile`:
- Around line 127-135: Update the lint-fix target to invoke $(MAKE)
main-lint-fix instead of duplicating the raw golangci-lint command, while
preserving the existing api_rc/main_rc status aggregation and exit behavior.
Keep main-lint-fix as the single definition of the main-module fix command.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 4708708c-1e4f-4cd5-ba07-d7209768157a

📥 Commits

Reviewing files that changed from the base of the PR and between 4fa09b1 and 06e2b44.

📒 Files selected for processing (3)
  • .pre-commit-config.yaml
  • DEVELOPMENT.md
  • Makefile

@codecov

codecov Bot commented Jul 15, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 44.13%. Comparing base (7eb0da0) to head (fa3dde5).
⚠️ Report is 6 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9016      +/-   ##
==========================================
+ Coverage   44.11%   44.13%   +0.01%     
==========================================
  Files         772      772              
  Lines       96226    96255      +29     
==========================================
+ Hits        42448    42478      +30     
+ Misses      50832    50831       -1     
  Partials     2946     2946              

see 6 files with indirect coverage changes

Flag Coverage Δ
cmd-support 38.24% <ø> (ø)
cpo-hostedcontrolplane 46.17% <ø> (+<0.01%) ⬆️
cpo-other 45.21% <ø> (+0.08%) ⬆️
hypershift-operator 54.09% <ø> (ø)
other 32.19% <ø> (+0.07%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@bryan-cox
bryan-cox marked this pull request as ready for review July 16, 2026 12:17
@bryan-cox

Copy link
Copy Markdown
Member Author

/area ci-tooling

@openshift-ci openshift-ci Bot added area/ci-tooling Indicates the PR includes changes for CI or tooling and removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. do-not-merge/needs-area labels Jul 16, 2026
@openshift-ci
openshift-ci Bot requested review from devguyio and muraee July 16, 2026 12:24
Remove the destructive find-delete of *_mock.go files from the generate
target — go generate overwrites in place, so deleting first creates a
fragile window where any failure leaves all mocks gone.

Split the monolithic make-lint-fix pre-commit hook into two scoped hooks
(api-lint-fix and main-lint-fix) so API import reordering doesn't fire
on unrelated commits, and add the missing $(GOLANGCI_LINT) prerequisite
to the new main-lint-fix target.

Make PULL_BASE_SHA fall back gracefully: upstream remote/main → local
main → empty (lint all files). When empty, --new-from-rev is omitted so
api-lint targets work in any environment.

Change fail_fast from true to false so all hook problems are visible in
one pass.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@openshift-ci

openshift-ci Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

@bryan-cox: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@bryan-cox

Copy link
Copy Markdown
Member Author

Local Verification Results

Six tests verifying each change in this PR. All output captured from local runs.


Test 1: Mock file survival through make generate

Problem on main: generate target runs find -delete on all *_mock.go files before go generate. If regeneration fails, mocks are gone and builds break with undefined: awsapi.NewMockEC2API.

Evidence: Ran make generate on this branch — 14 mock files before, 14 after, all checksums identical:

=== Mock file count BEFORE make generate ===
      14
=== Running make generate on our branch ===
GO111MODULE=on GOWORK=off GOFLAGS=-mod=vendor go generate ./...
=== Mock file count AFTER ===
      14

All 14 checksums match before/after (e.g. ec2_mock.go = bddb602d30083105a55e785ed17f9988).


Test 2: PULL_BASE_SHA fallback

Problem on main: api-lint-fix always passes --new-from-rev=${PULL_BASE_SHA}, even when empty (fork with no upstream remote).

Evidence: Dry-run comparison of the three modes:

# Empty PULL_BASE_SHA — our branch omits the flag entirely:
$ make -n api-lint-fix PULL_BASE_SHA=
golangci-lint run --config ./.golangci.yml --fix -v
# (no --new-from-rev, lints all files, no crash)

# Pre-commit mode — scoped to staged changes:
$ make -n api-lint-fix PULL_BASE_SHA=HEAD
golangci-lint run --config ./.golangci.yml --fix -v --new-from-rev=HEAD --whole-files

Test 3: fail_fast: false allows all hooks to run

Problem on main: fail_fast: true means if api-lint-fix fails, gitlint and all subsequent hooks are skipped.

Evidence: Created two identical test configs differing only in fail_fast:

=== fail_fast: true ===
Hook that always fails...................................................Failed
- hook id: always-fail
- exit code: 1
(second hook never appears — skipped entirely)

=== fail_fast: false ===
Hook that always fails...................................................Failed
- hook id: always-fail
- exit code: 1
Should this hook run?....................................................Passed
(second hook runs despite first failure)

Test 4: File-scoped hooks skip non-Go commits

Problem on main: make-lint-fix hook has no files: pattern — runs make lint-fix on every commit, even when only .md or .yaml files are staged.

Evidence: Staged only a .md file and ran pre-commit on our branch:

Sort API imports.....................................(no files to check)Skipped
Sort imports.........................................(no files to check)Skipped

Both lint hooks correctly skipped because no Go files were staged.


Test 5: --new-from-rev=HEAD --whole-files scopes lint-fix to staged files only

Problem on main: make lint-fix processes the entire repo, modifying files outside the staged set.

Evidence: Created identical gci violations in two files (corev1 "k8s.io/api/core/v1" placed in the wrong import group). Staged only one. Simulated pre-commit's stash behavior, then ran the scoped lint-fix:

=== Step 1: Stash unstaged changes (simulating pre-commit) ===
Saved working directory and index state On CNTRLPLANE-3791: test-stash

=== Step 3: Run our hook command ===
Command: make main-lint-fix PULL_BASE_SHA=HEAD
0 issues.

=== RESULTS ===
✓ hypershift-operator/main.go: MODIFIED (staged file was fixed by gci)
  (checksum changed: 00d13d... → 061f99...)
✓ control-plane-operator/main.go: unchanged (unstaged file correctly left alone)
  (checksum stayed: 4639a7... → 4639a7...)

Test 6: End-to-end pre-commit workflow

Created a Go file with intentionally wrong import ordering (k8s.io import mixed into stdlib group), staged it, ran pre-commit run:

Check for merge conflict strings.........................................Passed
Check yaml syntax....................................(no files to check)Skipped
Trim all whitespace from end of lines....................................Passed
Verify codespell.........................................................Passed
Keep cpo-containerfiles-in-sync..........................................Passed
Sort API imports.....................................(no files to check)Skipped
Sort imports.............................................................Failed
- hook id: main-lint-fix
- exit code: 2
- files were modified by this hook
Run gitlint..............................................................Passed

The diff shows gci correctly reordered the imports into proper groups:

 import (
+	"context"
 	"fmt"
-	"k8s.io/apimachinery/pkg/types"
+
 	"github.com/openshift/hypershift/support/config"
-	"context"
+
+	"k8s.io/apimachinery/pkg/types"
 )

Key observations:

  • Sort API importsSkipped (non-API file, file scoping works)
  • Sort importsFailed - files were modified (gci fixed imports, expected behavior)
  • Run gitlintPassed (ran after the "failure" because fail_fast: false)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/ci-tooling Indicates the PR includes changes for CI or tooling jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants