Skip to content

OCPBUGS-99014: Authenticate Konnectivity agents with cluster CA#9031

Open
hlipsig wants to merge 3 commits into
openshift:mainfrom
hlipsig:fix-konnectivity-agent-auth
Open

OCPBUGS-99014: Authenticate Konnectivity agents with cluster CA#9031
hlipsig wants to merge 3 commits into
openshift:mainfrom
hlipsig:fix-konnectivity-agent-auth

Conversation

@hlipsig

@hlipsig hlipsig commented Jul 17, 2026

Copy link
Copy Markdown

What this PR does / why we need it:

The Konnectivity proxy-server currently accepts agent connections without validating client certificates because its agent listener is not configured with a cluster CA. A reachable client can therefore connect without a certificate signed by the hosted cluster's Konnectivity CA.

Configure --cluster-ca-cert with the existing mounted Konnectivity CA bundle. Legitimate agents already use certificates signed by this CA. Add a regression test that verifies the embedded kube-apiserver manifest retains the authentication setting.

Which issue(s) this PR fixes:

NO-JIRA

Special notes for your reviewer:

Focused tests passed:
GO111MODULE=on GOWORK=off GOFLAGS=-mod=vendor go test -race ./control-plane-operator/controllers/hostedcontrolplane/v2/assets ./control-plane-operator/controllers/hostedcontrolplane/v2/kas

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Summary by CodeRabbit

  • Bug Fixes
    • Updated the Konnectivity server configuration to authenticate agents using the cluster CA certificate.
    • Ensured the kube-apiserver deployment passes the --cluster-ca-cert argument pointing to the mounted Konnectivity CA file.
  • Tests
    • Added a unit test that verifies the Konnectivity server container exists in the kube-apiserver manifest and includes the correct --cluster-ca-cert setting.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 17, 2026
@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 17, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@hlipsig: This pull request explicitly references no jira issue.

Details

In response to this:

What this PR does / why we need it:

The Konnectivity proxy-server currently accepts agent connections without validating client certificates because its agent listener is not configured with a cluster CA. A reachable client can therefore connect without a certificate signed by the hosted cluster's Konnectivity CA.

Configure --cluster-ca-cert with the existing mounted Konnectivity CA bundle. Legitimate agents already use certificates signed by this CA. Add a regression test that verifies the embedded kube-apiserver manifest retains the authentication setting.

Which issue(s) this PR fixes:

NO-JIRA

Special notes for your reviewer:

Focused tests passed:
GO111MODULE=on GOWORK=off GOFLAGS=-mod=vendor go test -race ./control-plane-operator/controllers/hostedcontrolplane/v2/assets ./control-plane-operator/controllers/hostedcontrolplane/v2/kas

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci

openshift-ci Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci openshift-ci Bot added do-not-merge/needs-area area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release and removed do-not-merge/needs-area labels Jul 17, 2026
@coderabbitai

coderabbitai Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Review was skipped due to path filters

⛔ Files ignored due to path filters (5)
  • control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/AROSwift/zz_fixture_TestControlPlaneComponents_kube_apiserver_deployment.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/GCP/zz_fixture_TestControlPlaneComponents_kube_apiserver_deployment.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/IBMCloud/zz_fixture_TestControlPlaneComponents_kube_apiserver_deployment.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/TechPreviewNoUpgrade/zz_fixture_TestControlPlaneComponents_kube_apiserver_deployment.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/zz_fixture_TestControlPlaneComponents_kube_apiserver_deployment.yaml is excluded by !**/testdata/**

CodeRabbit blocks several paths by default. You can override this behavior by explicitly including those paths in the path filters. For example, including **/dist/** will override the default block on the dist directory, by removing the pattern from both the lists.

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 22d7a82a-0634-4911-b4c5-29937d5fecb4

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The konnectivity-server container in the kube-apiserver Deployment now receives --cluster-ca-cert=/etc/konnectivity/ca/ca.crt. A unit test loads the deployment manifest, locates the container, and verifies the flag and certificate path.

🚥 Pre-merge checks | ✅ 11
✅ Passed checks (11 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed Added test names are static and deterministic; no dynamic Ginkgo titles or generated identifiers were introduced.
Test Structure And Quality ✅ Passed PASS: The new unit test checks one manifest behavior, has explicit fatal messages for preconditions, and no cluster ops, waits, or cleanup concerns; it matches package style.
Topology-Aware Scheduling Compatibility ✅ Passed PASS: The manifest change only adds --cluster-ca-cert to konnectivity-server; no nodeSelector, affinity, spread, replica, or taint assumptions were introduced.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed Added test is a unit test over embedded manifests, with no Ginkgo/e2e code, IP assumptions, or external connectivity.
No-Weak-Crypto ✅ Passed Changed files only add a manifest flag and a test; no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or secret comparisons found.
Container-Privileges ✅ Passed Changed manifest only adds --cluster-ca-cert; searches found no privileged, hostPID/Network/IPC, capabilities, runAsUser, or allowPrivilegeEscalation fields.
No-Sensitive-Data-In-Logs ✅ Passed Changed code adds no logging of sensitive data; only manifest args and plain test failures were introduced.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: adding cluster CA authentication for Konnectivity agents.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
control-plane-operator/controllers/hostedcontrolplane/v2/assets/assets_test.go (1)

55-55: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Use the required “When ... it should ...” test description format.

Rename or wrap this test so its case description follows the repository convention, for example When configured with the cluster CA, it should authenticate agents.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@control-plane-operator/controllers/hostedcontrolplane/v2/assets/assets_test.go`
at line 55, Rename TestKonnectivityServerAuthenticatesAgents to use the required
“When ..., it should ...” test description convention, such as describing the
cluster CA configuration and expected agent authentication behavior.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@control-plane-operator/controllers/hostedcontrolplane/v2/assets/assets_test.go`:
- Around line 59-60: Update the test around
LoadDeploymentManifest("kube-apiserver") to stop execution immediately when the
lookup returns an error or nil deployment, before any deployment.Spec
dereference; use the test’s fatal/early-return mechanism and apply the same
guard before dereferencing konnectivityServer.Args.
- Around line 71-74: Update the konnectivityServer.Args assertion in the
relevant test to verify that “--cluster-ca-cert” is immediately followed by
“/etc/konnectivity/ca/ca.crt”, using an ordered-pair or exact-subsequence
assertion instead of ContainElements.

---

Nitpick comments:
In
`@control-plane-operator/controllers/hostedcontrolplane/v2/assets/assets_test.go`:
- Line 55: Rename TestKonnectivityServerAuthenticatesAgents to use the required
“When ..., it should ...” test description convention, such as describing the
cluster CA configuration and expected agent authentication behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: fa825fbc-fa44-44fa-aec7-729246c79fbe

📥 Commits

Reviewing files that changed from the base of the PR and between e4576ff and 71820bb.

📒 Files selected for processing (2)
  • control-plane-operator/controllers/hostedcontrolplane/v2/assets/assets_test.go
  • control-plane-operator/controllers/hostedcontrolplane/v2/assets/kube-apiserver/deployment.yaml

Comment thread control-plane-operator/controllers/hostedcontrolplane/v2/assets/assets_test.go Outdated
Comment thread control-plane-operator/controllers/hostedcontrolplane/v2/assets/assets_test.go Outdated
@codecov

codecov Bot commented Jul 17, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 44.17%. Comparing base (e4576ff) to head (e95cd3a).
⚠️ Report is 6 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9031      +/-   ##
==========================================
- Coverage   44.17%   44.17%   -0.01%     
==========================================
  Files         772      772              
  Lines       96334    96352      +18     
==========================================
+ Hits        42559    42565       +6     
- Misses      50831    50843      +12     
  Partials     2944     2944              

see 2 files with indirect coverage changes

Flag Coverage Δ
cmd-support 38.23% <ø> (-0.02%) ⬇️
cpo-hostedcontrolplane 46.27% <ø> (+0.01%) ⬆️
cpo-other 45.22% <ø> (ø)
hypershift-operator 54.14% <ø> (ø)
other 32.34% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Hilliary Lipsig (RED HAT INC) and others added 2 commits July 16, 2026 19:30
Require Konnectivity agents to present certificates signed by the hosted
cluster's existing Konnectivity CA. Add a manifest regression test so agent
authentication cannot be removed unnoticed.

Signed-off-by: Hilliary Lipsig (RED HAT INC) <b-hlipsig@microsoft.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Stop immediately when manifest lookup prerequisites fail. Verify the cluster
CA path is the value directly following its proxy-server flag.

Signed-off-by: Hilliary Lipsig (RED HAT INC) <b-hlipsig@microsoft.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@hlipsig
hlipsig force-pushed the fix-konnectivity-agent-auth branch from 7af17d9 to 69928e6 Compare July 17, 2026 02:30
@bryan-cox

Copy link
Copy Markdown
Member

/retitle OCPBUGS-99014: Authenticate Konnectivity agents with cluster CA

@openshift-ci openshift-ci Bot changed the title NO-JIRA: Authenticate Konnectivity agents with cluster CA OCPBUGS-99014: Authenticate Konnectivity agents with cluster CA Jul 17, 2026
@openshift-ci-robot openshift-ci-robot added the jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. label Jul 17, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@hlipsig: This pull request references Jira Issue OCPBUGS-99014, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

What this PR does / why we need it:

The Konnectivity proxy-server currently accepts agent connections without validating client certificates because its agent listener is not configured with a cluster CA. A reachable client can therefore connect without a certificate signed by the hosted cluster's Konnectivity CA.

Configure --cluster-ca-cert with the existing mounted Konnectivity CA bundle. Legitimate agents already use certificates signed by this CA. Add a regression test that verifies the embedded kube-apiserver manifest retains the authentication setting.

Which issue(s) this PR fixes:

NO-JIRA

Special notes for your reviewer:

Focused tests passed:
GO111MODULE=on GOWORK=off GOFLAGS=-mod=vendor go test -race ./control-plane-operator/controllers/hostedcontrolplane/v2/assets ./control-plane-operator/controllers/hostedcontrolplane/v2/kas

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Summary by CodeRabbit

  • Bug Fixes
  • Updated the Konnectivity server configuration to authenticate agents using the cluster CA certificate.
  • Ensured the kube-apiserver deployment passes the --cluster-ca-cert argument pointing to the mounted Konnectivity CA file.
  • Tests
  • Added a unit test that verifies the Konnectivity server container exists in the kube-apiserver manifest and includes the correct --cluster-ca-cert setting.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@bryan-cox

Copy link
Copy Markdown
Member

/approve

@openshift-ci

openshift-ci Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bryan-cox, hlipsig

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 17, 2026
@bryan-cox

bryan-cox commented Jul 17, 2026

Copy link
Copy Markdown
Member

The TestControlPlaneComponents fixture test is failing because the golden fixture files need to be regenerated after adding --cluster-ca-cert to the deployment YAML. You can fix it by running:

UPDATE=true go test ./control-plane-operator/controllers/hostedcontrolplane/...

This will update all the zz_fixture_* YAML files under testdata/kube-apiserver/ (and its subdirectories) to include the new flag.

@csrwng

csrwng commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

The PR lgtm, pls fix the unit tests and I can tag.

Regenerate the kube-apiserver deployment fixtures so each platform includes the Konnectivity agent client CA configuration.

Signed-off-by: Hilliary Lipsig (RED HAT INC) <b-hlipsig@microsoft.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@hlipsig
hlipsig marked this pull request as ready for review July 17, 2026 16:11
@openshift-ci openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 17, 2026
@openshift-ci
openshift-ci Bot requested review from clebs and muraee July 17, 2026 16:13
@hlipsig

hlipsig commented Jul 17, 2026

Copy link
Copy Markdown
Author

The fixture updates are pushed in e95cd3a. The current head now passes Unit Tests (including cpo-hostedcontrolplane), Lint, and Verify. @csrwng, assuming nothing shows up in the e2e jobs this is ready for the /lgtm tag when you have a chance.

@csrwng

csrwng commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jul 17, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-v2-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-gke

@csrwng

csrwng commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Manual Verification Report

Tested on a live HostedCluster with the PR's CPO image deployed.

Environment:

  • CPO image: quay.io/csrwng/hypershift:e95cd3a947 (commit e95cd3a from this PR)
  • Release image: registry.ci.openshift.org/ocp/release-5:5.0.0-0.ci-2026-07-16-024429
  • Platform: AWS, Public, SingleReplica
  • Region: us-east-2

Results: 9/9 passed

# Scenario Result
1 --cluster-ca-cert flag present in konnectivity-server container args ✅ PASS
2 Flag value points to /etc/konnectivity/ca/ca.crt ✅ PASS
3 CA file exists inside the container and contains a valid certificate ✅ PASS
4 konnectivity-server container is ready ✅ PASS
5 All konnectivity-agent pods healthy on guest cluster (2/2 ready) ✅ PASS
6 Konnectivity tunnel connectivity (oc get nodes via guest kubeconfig) ✅ PASS
7 Unauthenticated TLS connection rejected on agent port 8091 ✅ PASS
8 Fake (non-CA-signed) client certificate rejected on agent port 8091 ✅ PASS
9 No TLS errors in konnectivity-server logs for legitimate agents ✅ PASS

Key observations:

  • Scenarios 7-8 confirm the security fix is effective: the proxy-server now requires a valid Konnectivity CA-signed client certificate on the agent listener (port 8091). Both unauthenticated connections and connections with self-signed certificates are rejected.
  • Scenario 5-6 confirm no regression: legitimate konnectivity-agents connect successfully and the tunnel is fully functional for oc exec, oc logs, and node access.

@csrwng

csrwng commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

/verified by @csrwng

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Jul 17, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@csrwng: This PR has been marked as verified by @csrwng.

Details

In response to this:

/verified by @csrwng

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@csrwng

csrwng commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Test plan and script here: https://gist.github.com/csrwng/94e9e6d68a61f346823c29d61f4bd9d6

@csrwng csrwng added the acknowledge-critical-fixes-only Indicates if the issuer of the label is OK with the policy. label Jul 17, 2026
@cwbotbot

cwbotbot commented Jul 17, 2026

Copy link
Copy Markdown

Test Results

e2e-aws

Failed Tests

Total failed tests: 8

  • TestCreateCluster
  • TestCreateCluster/Main
  • TestCreateCluster/Main/EnsureGlobalPullSecret
  • TestCreateCluster/Main/EnsureGlobalPullSecret/When_management-cluster_hostedCluster.Spec.PullSecret_is_updated_in-place_it_should_propagate_to_guest_without_rollout
  • TestNodePool

... and 3 more failed tests

e2e-aks

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

/retest-required

Remaining retests: 0 against base HEAD 6142e46 and 2 for PR HEAD e95cd3a in total

@openshift-ci

openshift-ci Bot commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

@hlipsig: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws e95cd3a link true /test e2e-aws

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

acknowledge-critical-fixes-only Indicates if the issuer of the label is OK with the policy. approved Indicates a PR has been approved by an approver from all required OWNERS files. area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants